Or Raz (@OrRaz6) header
Or Raz (@OrRaz6) profile picture

LLM security enthusiast. Learning, sharing, and helping others understand the complexities. Join me on this journey.

Joined July 2020
Photo by Or Raz (@OrRaz6)Photo by Or Raz (@OrRaz6)Photo by Or Raz (@OrRaz6)Photo by Or Raz (@OrRaz6)Photo by Or Raz (@OrRaz6)Photo by Or Raz (@OrRaz6)Photo by Or Raz (@OrRaz6)Photo by Or Raz (@OrRaz6)Photo by Or Raz (@OrRaz6)Photo by Or Raz (@OrRaz6)
2 signs that enterprise adoption of LLM has started - and one that the hype is over -->
3
1
238
And here is a clear sign that the hype is over -
83
People are looking for Azure OpenAI...
63
LangChain is skyrocketing.
49
A new interesting prompt injection based SSRF, this time in LlamaIndex.
LLM based agents are growing in popularity, and I often wonder how many people are aware of the risks in enabling LLMs to use tools. While playing with LlamaIndex agents, I came across an example notebook for an OpenAPI + Requests based agent which is vulnerable to SSRF.
95
1/4 🚨Indirect prompt injection is set to dominate the cybersecurity threat landscape this decade. Hard to imagine an attack surface more potent -->
3
1
2
247
4/4 This should be a concern for any business that's adopting generative AI. When architecting LLM powered apps, it must be taken into account.
33
3/4 🎯Perfect exploitation ground? Applications using LLM to analyze web content. Attackers should simply be able to alter website's content, in order to get initial access to the LLM and try to exploit it.
43
2/4 In an indirect prompt injection, the adversarial instructions are introduced by a third party data source like a web search or API call.
47
🧪langchain_experimental In an effort to make langchain leaner, more focused, and safer, we are moving select chains to a separate package on 7/28 Big thanks to folks like @BoazWasserman @OrRaz6 Justin Flick for pushing on the safety part There will be some breaking changes 🧵
1
18
133
41,865
It could be hard to introduce so many changes and new features, and still maintain high security standards. I'm glad that @langchain and @hwchase17 are on that.
Exciting changes coming from @langchain on the security front. There is a new announcement that any component that has vulnerabilities (SQL & python agents for example) will be moved from the package core into an experimental package github.com/hwchase17/langcha…
79
There's a new awesome blog from @Dropbox that shows evidence of OpenAI's gpt-3.5-turbo processing control sequences the same way a terminal would do. E.g. \b will cause the model to "backspace" and ignore previous input. dropbox.tech/machine-learnin…
1
1
2
181
The fact that ChatGPT Code Interpreter can still be jailbroken to do really nasty stuff shows how far we are from solving LLM jailbreaks. I was easily able to get it to create a macro-enabled document that downloads and executes a payload from pastebin 🫤
1
1
6
399
Are they possible? Sure. But I must say that data poisoning is one of the more complicated (and less likely to be common) attack scenarios, and there are much easier LLM attack scenarios.
Are data poisoning attacks practical? tl;dr: yes - podcast with ETHZ prof on work w. Robust, Google, NVIDIA mlsecops.com/podcast/tramer-…
1
103
If you haven't tried this one already, you may want to check it out: nicholas.carlini.com/writing… Nicholas Carlini will show you how little you truly know about GPT-4 capabilities.
58
Another day, another LangChain RCE... This time in a new CPAL chain (causal program-aided language), which improves upon the capabilities of the existing PAL chain
1
2
125
THE most interesting weakness on OWASP LLM top 10 is #8: Excessive Agency. 🕵️ This decade, the security teams' greatest catalyst for change could be the fortification of LLM agency security.
2
85
The GenAI general hype is already over, but it seems that enterprises are just starting with real-world use cases. Here's one sign for that:
1
3
125
Interesting find on ChatGPT's Code Interpreter (requires some jailbreaking) - you can manually upload missing python libraries, and get it to extract and then use them (although it requires a little jailbreaking 😅)
3
5
355
Who joined Threads? What do you think about the app so far?
1
1
50