NCC Group Research

Quite an interesting article. The Vice example is hilarious. But for all doom and gloom you haven't addressed the most obvious mitigation - Preflight Prompt Check [1]. It would be trivial to detect toxic prompts and halt further injection. Surely there will be other mitigations to follow. [1] https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-at…

Never did a security audit, but I did regulatory and financial audits of banks. Most of our work involved looking at last year's work, re-performing it and changing dates. Writing reports + financial statement notes followed a similar process. Exceptions are immediately escalated to the audit committee and sometimes end up as a small footnote in the public reports. Most of the time it says "we were unable to collect sufficient evidence" to provide assurance. Almost never "this was done wrong". It's interesting to see how the second report differed from their first assessment earlier in the year: https://research.nccgroup.com/2021/04/08/public-report-vpn-by-google-o… Most of the findings in the first report were "Fixed".

Get your AuthZ design right early on, or you’ll forever be plagued with APIs that fail open or aren’t protected, and it’s so very hard to change later on. The best advice I’ve seen on this: https://research.nccgroup.com/2020/04/21/code-patterns-for-api-authori…

I'm not a RISC-V expert, primarily relying on open-source firmware community knowledge and the opinion of such figures as Ron Minnich, but AFAIK, most firmware for production RISC-V deployments is closed-source, so we can't say if vendors use OpenSBI implementation or some modified version for their malicious purposes. If I need to be corrected, please point me to products that state how they transparently use SBI. There is evidence of transparent use of SMM in coreboot. SBI is not the only RISC-V TEE. There are also other concepts, like MultiZone [1] and PMP-based Keystone [2], advertised as trusted execution environments The openness of specification only matters a little here - UEFI specification about management mode is also open. But IBVs screwed us many times [3] by implementing low-quality SMIs that could elevate privileges, and the spec does not guarantee that implementers follow it. Lack of tooling makes compliance difficult - it slowly changes with various startups seeing that as an opportunity (e.g., Binarly, Eclypsium?). My point is that SMM as TEE is not a unique x86 feature, and other significant architectures have similar mechanisms. OS has a means of figuring out that it was in SMI, so it is not entirely hidden but still has superpowers. In the CNC world SMI latency is measured from OS [4]. Trusted execution environments are hammers and can be used for a good purpose in a transparent way and for malicious purposes. Further typically depends on the…

Here are additional resources about specific defense techniques for prompt attacks: NCC Group: Exploring Prompt Injection Attacks https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-at… Preamble: Ideas for an Intrinsically Safe Prompt-based LLM Architecture https://www.preamble.com/prompt-injection-a-critical-vulnerability-in-… @Riley, hello, I wanted to say hi and I would love to connect with you if you have time, as I also work in the prompt safety space and would be honored to brainstorm with you someday. Would you like to start a message thread on a platform that supports it? I think the research you are doing is amazing and would love to bounce some ideas back & forth. I was the one who discovered some version of prompt injection in May 2022 while researching AGI safety and using LLM as a stand-in for the hypothetical AGI. You could email me at [email protected] to reach me if you would like! Sincerely, another prompt safety researcher

HN discussion: https://news.ycombinator.com/item?id=41184219 But nothing new: https://research.nccgroup.com/2023/04/27/state-of-dns-rebinding-in-202… https://github.com/nccgroup/singularity/wiki/Protection-Bypasses

Good to be skeptical. I'm very suspicious of most VPNs, but Google did get an audit: https://research.nccgroup.com/2021/04/08/public-report-vpn-by-google-o…

Randomness from a CSPRNG (cryptographically-secure random number generator) never really gets "depleted," since as long as the seed contains enough entropy and isn't compromised, then it's computationally infeasible to learn anything about the internal state of the CSPRNG from it's outputs. See https://research.nccgroup.com/2019/12/19/on-linuxs-random-number-gener… for a nice overview. The Linux random number generator did used to have a notion of entropy depletion, but that is no longer the case (at least for x86-64 systems: https://wiki.archlinux.org/title/Random_number_generation). On older systems that have a notion of entropy depletion, you would eventually deplete the entropy counter and /dev/random would start blocking if you aren't feeding new entropy into the system.

Recently, I found a privilege escalation vulnerability in a shell script as a result of arithmetic expansion (similar to the one described at https://research.nccgroup.com/2020/05/12/shell-arithmetic-expansion-an… For example, $((1 + ENV_VAR)) allows you to inject code if you can control $ENV_VAR. Unfortunately, shellcheck did not catch that. At least not with the default settings. But if you are implementing anything remotely security-critical, you should not be using shell anyway.

"Unfortunately, all Nvidia GPUs since recent years have VBIOS digital signature checks, making VBIOS modification impossible" This is not necessarily true. As seen with android devices you can force digital signature checking mechanisms by varying voltage levels in order to get the device to completely skip the checks as if they were never there. https://research.nccgroup.com/2020/10/15/theres-a-hole-in-your-soc-gli… I'm sure a similar strategy could be developed here.

There’s more info on this here: https://research.nccgroup.com/wp-content/uploads/2022/04/NCC_Group_Goo… In short we can only verify PINs, and only a very limited number of guesses, enforced by hardware.

Some articles on this topic: https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-at… https://simonwillison.net/2022/Sep/16/prompt-injection-solutions/

And in case anybody is interested in the gory technical details, Google let a third party review their implementation of encrypted backups that included visiting their data center because it used their custom Titan hardware. Fascinating read https://research.nccgroup.com/wp-content/uploads/2022/04/NCC_Group_Goo…

There was an external audit in 2021 – https://research.nccgroup.com/2021/10/27/public-report-whatsapp-end-to…

Decompilation. Reverse engineering. Network monitoring. Third-party attestations like https://research.nccgroup.com/wp-content/uploads/2021/10/NCC_Group_Wha… The lack of whistleblowers from within Meta itself. There are are hundreds of employees working on this product.

You are probably talking about this: https://research.nccgroup.com/wp-content/uploads/2020/07/ncc_group-ass… In short - not a fundamental limitation - just that kernels (even if they are small) have a ton of work that goes into them. Nanos for instance has page protections, ASLR, virtio-rng (if on GCP), etc.

This is what I found at the top of an internet search: Under voltage fault injection - "In general, single instruction skips are easily achievable, though skips of multiple nearby instructions are more difficult to induce and control." https://research.nccgroup.com/2021/07/07/an-introduction-to-fault-inje…

A few links which provide some good insight for your idea. https://research.nccgroup.com/2021/10/21/cracking-rdp-nla-supplied-cre… https://github.com/nccgroup/nlahoney

Here is a nice discussion of what happens when you don't validate your elliptic curve parameters properly: * https://research.nccgroup.com/2021/11/18/an-illustrated-guide-to-ellip… The highlight here is that in some cases, failure to properly validate gets an attacker the secret key material. Note all the conditional bits. Different curves have different properties and different issues. There are a bunch of different curves in common use while RSA pretty much always uses the same value for the parameter these days (RSA literally has just one parameter. The exponent.).

Here’s a good way to learn more about cryptography: https://cryptopals.com/ https://research.nccgroup.com/2021/12/10/announcing-ncc-groups-cryptop…

Can't find it, however you might be interested in this article from HN earlier in the year. Written/done by an intern. https://research.nccgroup.com/2020/10/15/theres-a-hole-in-your-soc-gli…

Hi HN! Principal Mapper is a tool + library for analyzing and securing your AWS IAM configuration. It generates a model of your account and/or organization and uses it to give you a better idea of the effective permissions of your IAM Users and Roles. It has privilege escalation detection built-in and is hopefully written in a way that will let you extend it for your use-cases. This v1.1.0 update covers more types of policies (resource policies, permission boundaries, session policies, SCPs), supports AWS Organizations, enables cross-account checks, and more! https://github.com/nccgroup/PMapper https://research.nccgroup.com/2021/03/29/tool-release-principal-mapper… Happy to answer any questions you have here!

That this happens right after Thomas Pornin ridiculing the blocking pool (https://research.nccgroup.com/2019/12/19/on-linuxs-random-number-gener… HN discussion https://news.ycombinator.com/item?id=21843081) is obviously purely coincidental, right? Especially as it was read and commented upon by Theodore Tso that at last changed his mind...

> Any evidence in this area? See danShumway's post below. People are regularly posting exploits on twitter, including getting the system to dump it's prompt. May I ask politely, are you a programmer, and have you secured system's previously? It will change the way I approach trying to carry my message across. For background, a finished LLM is a blackbox. You can't program the LLM in the box in the traditional sense, because we don't fully understand what happens in the box at a level where we can "code" it. Judging the security of a filter by the cases where it works is a very bad way to judge security. Blocklists ARE NOT SAFE because it is impossible to account for the infinite variety of things that can be tried. Here's a whitepaper on the difficulties. There's been lots of writing about this: https://research.nccgroup.com/wp-content/uploads/2020/07/ncc-group-whi… Now, this has been shown to be difficult for really constrained scenarios, like SQL and so forth, but English has a million words, for starters.

When the input is first selected here's will be a hot plug detect, EDID request and an EDID response. That response can be up to 32kB so anywhere up to ~2 seconds depending on how many extension blocks are in use by the display. Here's a nice breakdown on more of what happens there: https://research.nccgroup.com/wp-content/uploads/2020/07/hdmi_-_hackin… When a suitable set of image parameters have been agreed HDCP (may) then step in to ruin your day. Again a nice walkthrough of that is available here: https://blog.cryptographyengineering.com/2012/08/27/reposted-cryptanal… As other commenters have correctly noted, there's definite room for improvement. One thing to remember though is this involves hardware that can be in operation for the order of decades so there's a lot of cruft that accretes in order to maintain compatibility. Frankly it's mildly impressive that for the most part all the above happens in silence and continues to (mostly) work.

Allocator efficiency was the primary concern in the past (both speed and packing), which is why there are several things in malloc which is obviously bad for allocation safety. The allocators in the past have tried to use the space it is not currently using for user code for its own book-keeping, which is very efficient when it comes to usage of space. So the free-lists are interleaved with the actual allocations to be as cheap as they can be (not in a different page). Oddly enough, packing all the metadata in a single area might actually be faster though at lower scales might be worse for the data itself. I've used talloc[1] in the past, which traded some of the efficiencies for safety (I have my own sins trying to copy, but simplify pool based allocators for PHP apc with some tricks[3] borrowed from xcache). If the author is reading, the missing magic I've always missed with jemalloc are the .gdb macros for walking the allocator and do things like print the allocations by size/content [2]. You can write them in python now and that can let you print them out neatly in a graphviz dot structure with linkages, which is the sort of project which you hand out to an intern who just joined. [1] - https://talloc.samba.org/talloc/doc/html/libtalloc__dts.html [2] - https://research.nccgroup.com/2015/08/11/libtalloc-a-gdb-plugin-for-an… [3] - http://notmysock.org/blog/2009/Jan/08/

Do you mean Ethernet over HDMI? Even though there was a standard and special HDMI cables that claimed compatibility, I don’t think any devices actually implemented it. Well perhaps one device according to this security paper[1] which basically means it’s useless. I control access by connecting a Samsung TV to my Ethernet network but keep it configured to use WiFi most of the time. I’ve never entered a WiFi password so the network is effectively disconnected until I configure the TV to use the wired connection. The TV prompts me to use the wired connection when I reboot my router and the link drops and comes back up but I can cancel switching to wired at that point. [1]: What the HEC? Security implications of HDMI Ethernet Channel and other related protocols Andy Davis, Research Director NCC Group https://research.nccgroup.com/wp-content/uploads/2020/07/44con_hdmi_et…