Pinned Tweet
In the era of LLM-based software, prompt-injection-based SSRF is emerging as a major attack surface.
I've just pushed a key security fix for @LangChain, fortifying the APIChain against this type of attack:
github.com/hwchase17/langcha…
A new interesting prompt injection based SSRF, this time in LlamaIndex.
LLM based agents are growing in popularity, and I often wonder how many people are aware of the risks in enabling LLMs to use tools.
While playing with LlamaIndex agents, I came across an example notebook for an OpenAPI + Requests based agent which is vulnerable to SSRF.
95
1/4 🚨Indirect prompt injection is set to dominate the cybersecurity threat landscape this decade. Hard to imagine an attack surface more potent -->
3/4 🎯Perfect exploitation ground? Applications using LLM to analyze web content. Attackers should simply be able to alter website's content, in order to get initial access to the LLM and try to exploit it.
43
2/4 In an indirect prompt injection, the adversarial instructions are introduced by a third party data source like a web search or API call.
47
Or Raz retweeted
🧪langchain_experimental
In an effort to make langchain leaner, more focused, and safer, we are moving select chains to a separate package on 7/28
Big thanks to folks like @BoazWasserman @OrRaz6 Justin Flick for pushing on the safety part
There will be some breaking changes 🧵
It could be hard to introduce so many changes and new features, and still maintain high security standards.
I'm glad that @langchain and @hwchase17 are on that.
Exciting changes coming from @langchain on the security front.
There is a new announcement that any component that has vulnerabilities (SQL & python agents for example) will be moved from the package core into an experimental package
github.com/hwchase17/langcha…
79
Or Raz retweeted
There's a new awesome blog from @Dropbox that shows evidence of OpenAI's gpt-3.5-turbo processing control sequences the same way a terminal would do. E.g. \b will cause the model to "backspace" and ignore previous input.
dropbox.tech/machine-learnin…
Or Raz retweeted
The fact that ChatGPT Code Interpreter can still be jailbroken to do really nasty stuff shows how far we are from solving LLM jailbreaks.
I was easily able to get it to create a macro-enabled document that downloads and executes a payload from pastebin 🫤
Are they possible? Sure.
But I must say that data poisoning is one of the more complicated (and less likely to be common) attack scenarios, and there are much easier LLM attack scenarios.
Are data poisoning attacks practical?
tl;dr: yes - podcast with ETHZ prof on work w. Robust, Google, NVIDIA
mlsecops.com/podcast/tramer-…
1
103
If you haven't tried this one already, you may want to check it out:
nicholas.carlini.com/writing…
Nicholas Carlini will show you how little you truly know about GPT-4 capabilities.
58
Or Raz retweeted
Another day, another LangChain RCE...
This time in a new CPAL chain (causal program-aided language), which improves upon the capabilities of the existing PAL chain
THE most interesting weakness on OWASP LLM top 10 is #8: Excessive Agency. 🕵️
This decade, the security teams' greatest catalyst for change could be the fortification of LLM agency security.
2
85
The GenAI general hype is already over, but it seems that enterprises are just starting with real-world use cases. Here's one sign for that:
Or Raz retweeted
Interesting find on ChatGPT's Code Interpreter (requires some jailbreaking) - you can manually upload missing python libraries, and get it to extract and then use them (although it requires a little jailbreaking 😅)















