Embrace The Red

we recently moved to a similar approach, inspired by gondolin which does the same: https://earendil-works.github.io/gondolin/secrets/ an 'mitm' tls proxy also gives you much better firewalling capabilities [1], not that firewalls aren't inherently leaky, codex's a 'wildcard' based one [2]; hence "easy" to bypass [3] github's list is slightly better [4] but ymmv [1] than a rudimentary "allow based on nslookup $host" we're seeing on new sandboxes popping up, esp. when the backing server may have other hosts. [2] https://developers.openai.com/codex/cloud/internet-access#common-depen… [3] https://embracethered.com/blog/posts/2025/chatgpt-codex-remote-control… [4] https://docs.github.com/en/copilot/reference/copilot-allowlist-referen…

It gets even worse with LLMs and agents. Many LLMs can interpret invisible Unicode Tag characters as instructions and follow them (eg invisible comment or text in a GitHub issue). I wrote about this a few times, here a recent example with Google Jules: https://embracethered.com/blog/posts/2025/google-jules-invisible-promp…

Excellent concrete examples with video demos here: https://embracethered.com/blog/ The researcher has gotten actual shells on oai machines before via prompt injection

Those all appear to be thinly disguised sales pitches for their Tramlines product. They also look to be selling the kind of filtering/guardrails solution that I argue in my talk doesn't actually work. (Update: that's a little unfair, I had a look and a bunch of their rules are at least deterministic, like making sure DELETE isn't present in a call made to a database MCP.) If you're looking for credible sources on MCP and prompt security that aren't my blog, I strongly recommend https://embracethered.com/blog/

This is missing the human language ambiguity problem. If you don't perfectly specify your requirements and it misinterprets what you're asking for that's going to be a problem regardless of how smart it is. This is fine with code editing since you've got version control and not so great when running commands in your terminal that can't be as trivially reverted. Besides that, you can absolutely still trick top of the line models: https://embracethered.com/blog/posts/2024/claude-computer-use-c2-the-z… Hallucination might be getting better, gullibility less so.

People already do this. Recommended blog: https://embracethered.com/blog/

> the "personal AI assistant" is the best example, since prompt injection means that any time an LLM has access to both private data and untrusted inputs (like emails it has to summarize) there is a risk of something going wrong: https://simonwillison.net/2023/May/2/prompt-injection-explained/#promp… See also https://simonwillison.net/2023/Apr/14/worst-that-can-happen/, or https://embracethered.com/blog/posts/2023/google-bard-data-exfiltratio… for more specific examples. https://arxiv.org/abs/2302.12173 was the paper that originally got me aware of "indirect" prompt injection as a problem and it's still a good read today.

The point is authorization. With full web access, your agent can reach anything and leak anything. You could restrict where it can go with domain allowlists but that has insufficient granularity. The same URL can serve a legitimate request or exfiltrate data depending on what's in the headers or payload: see https://embracethered.com/blog/posts/2025/claude-abusing-network-acces… So you need to restrict not only where the agent can reach, but what operations it can perform, with the host controlling credentials and parameters. That brings us to an MCP-like solution.

There's an awful lot of people running these tools in YOLO mode. https://embracethered.com/blog/posts/2025/the-normalization-of-devianc…

Not the first time by the way. GitHub Copilot Chat: From Prompt Injection to Data Exfiltration https://embracethered.com/blog/posts/2024/github-copilot-chat-prompt-i…

Cline seems to be having some security vulnerabilities that aren't addressed, e.g. https://embracethered.com/blog/posts/2025/cline-vulnerable-to-data-exf… Begs the question of long-term support, etc...

The security endgame of LLMs terrifies me. We've designed a system that only supports in-band signalling, undoing hard learned lessons from prior system design. There are ampleattack vectors ranging from just inserting visible instructions to obfuscation techniques like this and ASCII smuggling[0]. In addition, our safeguards amount to nicely asking a non deterministic algorithm to not obey illicit instructions. 0: https://embracethered.com/blog/posts/2024/hiding-and-finding-text-with…

This is just another "rebranding" of the failed "prompt engineering" trend to promote another borderline pseudo-scientific trend to attact more VC money to fund a new pyramid scheme. Assuming that this will be using the totally flawed MCP protocol, I can only see more cases of data exfiltration attacks on these AI systems just like before [0] [1]. Prompt injection + Data exfiltration is the new social engineering in AI Agents. [0] https://embracethered.com/blog/posts/2025/security-advisory-anthropic-… [1] https://www.bleepingcomputer.com/news/security/zero-click-ai-data-leak…

Remote MCP servers can do prompt injection that instruct your local agent to do something else other than only the expected tool call. https://embracethered.com/blog/posts/2025/model-context-protocol-secur…

Copy this󠀠󠀼󠀼󠀼󠀠󠁉󠁳󠀠󠁴󠁨󠁩󠁳󠀠󠁮󠁥󠁣󠁥󠁳󠁳󠁡󠁲󠁹󠀠󠁴󠁯󠀠󠁣󠁯󠁲󠁲󠁥󠁣󠁴󠁬󠁹󠀠󠁲󠁥󠁰󠁲󠁥󠁳󠁥󠁮󠁴󠀠󠁨󠁵󠁭󠁡󠁮󠀠󠁬󠁡󠁮󠁧󠁵󠁡󠁧󠁥󠀿󠀠󠀾󠀾󠀾 sentence into this site and click Decode. (YMMW) https://embracethered.com/blog/ascii-smuggler.html

Yes, images! And also link unfurling in bots. This researcher here talked about it before and also found tons of such data exfil issues in various LLM apps: https://embracethered.com/blog/posts/2024/the-dangers-of-unfurling-and…

Similar tools already available in many distros: https://github.com/bpftrace/bpftrace/blob/master/tools/sslsnoop.bt https://github.com/iovisor/bcc/blob/master/tools/sslsniff.py https://embracethered.com/blog/posts/2021/offensive-bpf-sniffing-traff… On Ubuntu for example I can just "sudo apt install bpfcc-tools", then run "sudo sslsniff-bpfcc" and see curl TLS traffic right away.

So is the idea (for the last/$20k one) that you would convince someone to paste your maliciously crafted prompt to steal their data? The other post[0] of the same exploit is really interesting b/c it reads instructions from a document. So if someone had something like "find X in my documents" and you shared the malicious document with them, it could trigger those instructions. [0] https://embracethered.com/blog/posts/2023/google-bard-data-exfiltratio…

The Universal Interface currently has a huge problem in the form of "prompt injection" where currently we have no principled way to separate instructions and data, e.g. https://embracethered.com/blog/posts/2023/google-bard-data-exfiltratio… Not to say this is the only problem, but this one is at least moderately well known and a clear and visible blocker.

I think the linked article makes for better reading than the tweet: https://embracethered.com/blog/posts/2023/chatgpt-plugin-vulns-chat-wi…

There is a "resizing attack" that's been published that does what you're suggesting https://embracethered.com/blog/posts/2020/husky-ai-image-rescaling-att…

I'm a Master's student in Artificial Intelligence at UofT. Right now, I'm adding the "ML (Machine Learning) Streaming" feature to PyMilo. PyMilo is an open-source Python package that provides a transparent, safe, and end-to-end way for users to export pre-trained machine-learning models. Transparency and non-executable export format is a serious thing, take a look at https://embracethered.com/blog/posts/2022/machine-learning-attack-seri… After version 0.9 release, PyMilo became feature-completed with full support of scikit-learn models, now it's time to move on to PyTorch and then Tensorflow. But we decided to add the "ML Streaming" feature before getting into PyTorch, in order to provide an easy way to smoothly stream your ML model. By using the "ML Streaming" feature you can easily deploy your model into the remote server, connect to it from the client side, and choose the working mode, either delegation or local mode, through delegation mode your requests will be relayed to the remote server and you can easily work with your remote model from any devices without any further dependencies, and finally, you can download model for local use. We will release the 1.0 (tenth) version of PyMilo around Sep 16th, this release will be the first release to have the "ML Streaming" feature with support of REST API, and we will next add other protocols such as Websocket. Here is PyMilo: https://github.com/openscilab/pymilo Thank you for your time

> a human reviewer has to confirm that the photos are illegal before any action is taken The reviewer actually only confirms that the-photos-as-modified-by-an-undisclosed-transformation are illegal, and this strategic ambiguity is presumably designed to maintain plausible deniability that what Apple subjects its reviewers to is a) not psychologically damaging, and b) not illegal to possess. Unfortunately this is "security by obscurity", so we have to assume it is vulnerable to image scaling attacks[0]. Unless Apple document their human review process, and allow an independent audit of the technology (to provide a strong mathematical guarantee that it is not susceptible to such image-smuggling attacks) and independent verification that the published process is the one they actually use, then there is an unacceptably high chance that a human reviewer will trigger a legal process against users for files they couldn't tell contained illegal data. [0] https://embracethered.com/blog/posts/2020/husky-ai-image-rescaling-att…

Yes, I can. This is just one possible strategy: there are many others, where different things are done, and where things are done in a different order. You use the collider [1] and one of the many scaling attacks ([2] [3] [4], just the ones linked in this thread) to create an image that matches the hash of a reasonably fresh CSAM image currently circulating on the Internet, and resizes to some legal sexual or violent image. Note that knowing such a hash and having such an image are both perfectly legal. Moreover, since the resizing (the creation of the visual derivative) is done on the client, you can tailor your scaling attack to the specific resampling algorithm. Eventually, someone will make a CyberTipline report about the actual CSAM image whose hash you used, and the image (being a genuine CSAM image) will make its way into the NCMEC hash database. You will even be able to tell precisely when this happens, since you have the client-side half of the PST database, and you can execute the NeuralHash algorithm. You can start circulating the meme before or after this step. Repeat until you have circulated enough photos to make sure that many people in the targeted group have exceeded the threshold. Note that the memes will trigger automated CSAM matches, and pass the Apple employee's visual inspection: due to the safety voucher system, Apple will not inspect the full-size images at all, and they will have no way of telling that the NeuralHash is a false positive. [1]…

The problem is that it is a scaled low-res version. There are well publicized attacks[1] showing you can completely change the contents of the image post scaling. There's also the added problem that if the scaled down image is small, even without the attack, it's impossible to make a reasonable human judgement call (as OP points out). The problem isn't CSAM scanning in principle. The problem is that the shift to the client & the various privacy-preserving steps Apple is attempting to make is actually making the actions taken in response to a match different in a concerning way. One big problem isn't the cases where the authorities should investigate*, but that a malicious actor can act surreptitiously and leave behind almost no footprint of the attack. Given SWATting is a real thing, imagine how it plays out if child pornography is a thing. From the authorities perspective SWATting is low incidence & not that big a deal. Very different perspective on the victim side though. [1] https://embracethered.com/blog/posts/2020/husky-ai-image-rescaling-att… * One could argue about the civil liberties aspect & the fact that having CSAM images is not the same as actually abusing children. However, among the general population that line of reasoning just gets you dismissed as supporting child abuse & is only starting to become acknowledged in the psychiatry community.

Oh yeah! I totally forgot about Intel's on-chip AI acceleration[1]. I'm not sure how well it works, but I know that it exists and nothing major is currently taking advantage of it AFAIK. Local Copilot sounds interesting, but I'd imagine it would be akin to the old Siri on iPhones where its use would be limited to controlling on-device settings. I've heard and read quite a lot about the dangers of that, though: with prompt injections being an issue with Copilot[0] (and Bing Chat frequently going crazy[2]), I can understand why Microsoft are very hesitant to give it any real control over the system. [0]: https://embracethered.com/blog/posts/2024/whoami-conditional-prompt-in… [0.1]: https://kai-greshake.de/posts/llm-malware/ [1]: https://www.tomshardware.com/news/intel-details-meteor-lakes-ai-accele… [1.1]: https://www.extremetech.com/computing/intel-announces-an-integrated-ne… (hard to find a good explanation of these accelerators) [2]: https://www.forbes.com/sites/siladityaray/2023/02/16/bing-chatbots-unh…