Nicholas Carlini

This is just crazy and I love it <3 From a description on the author's website[1] of a Doom clone written in 13k of JavaScript. > Until recently all content on this website was research, and while writing papers can be fun, sometimes you just need to blow off a little steam. I think more companies should allow for their employees to have some plain old fun with no strings attached on a regular basis. [1] https://nicholas.carlini.com/

There was an earlier version of the underlying 3d engine that used only Canvas. WebGL use is justified like this: > Once I actually got that working doing all of the math by hand in JavaScript I decided that using WebGL would probably be worth it, and actually probably wasn't cheating all that much. WebGL exposes access to the GPU-enabled rendering engine through JavaScript. While it does abstract away some of the rendering, it's less than I thought---it just supports the ability to do the necessary math efficiently---so I decided this wouldn't be cheating. And fortunately, it didn't take long to reproduce the initial renderer, but this time supporting much better (and more efficient) graphics. From: https://nicholas.carlini.com/writing/2019/3d-renderer-javascript.html

I recommend this for a more nuanced view: https://nicholas.carlini.com/writing/2024/how-i-use-ai.html

> But no one really show how they're actually solving problems with LLMs and how the alternatives were worse for them. It's all claims that it's great with no further elaboration on the workflows. To give an example, one person (a researcher at DeepMind) recently wrote about specific instances of his uses of LLMs, with anecdotes about alternatives to each example. [1] People on HN had different responses with similar claims with elaborations on how it has changed some of their workflows. [2] While it would be interesting to see randomized controlled trials on LLM usage, hearing people's anecdotes brings to mind the (often misquoted) phrase: "The plural of anecdote is data". [3] [4] [1] https://nicholas.carlini.com/writing/2024/how-i-use-ai.html [2] https://news.ycombinator.com/item?id=41150317 [3] http://blog.danwin.com/don-t-forget-the-plural-of-anecdote-is-data/ [4] originally misquoted as "Anecdote is the plural of data."

I just want to emphasise two things, which are both mentioned in the article, but I still want to emphasise them as they are core to what I take from the article as someone who has been a fan boy of Nicholas for years now 1. Nicholas really does know how badly machine learning models can be made to screw up. Like, he really does. [0] 2. This is how Nicholas -- an academic researcher in the field of security of machine learning -- uses LLMs to be more efficient. I don't know whether Nicolas works on globally scaled production systems with have specific security/data/whatever controls that need to be adhered to, or whether he even touches any proprietary code. But seeing as he heavily emphasised the "i'm a researcher doing research things" in the article -- I'd take a heavy bet that he does not. And academic / research / proof-of-concept coding has different limitations/context/needs than other areas. I think this is a really great write up, even as someone on the anti-LLM side of the argument. I really appreciate the attempt to do a "middle of the road" post which is absolutely what the conversation needs right now (pay close attention to how this was written LLM hypers). I don't share his experience, I still value and take enjoyment from the "digging for information" process -- it is how I learn new things. Having something give me the answer doesn't help me learn, and writing new software is a learning process for me. I did take a pause and digested the food for…

I made this point a few months ago here, but using the words attacker and defender (builder) rather than red team and blue team: https://lobste.rs/s/i2edlt/how_i_use_ai The asymmetry is: An attacker only has to be right ONCE, and he wins Conversely, the defender only has to be wrong once, and he is wrong. So the conclusion is: Defenders/creators are using LLMs to pump out crappy code, and not testing enough, or relying on the LLM to test itself. Some attackers might be too dismissive of LLMs, and could accelerate their work by using them to try more things The comment was related to these stories: How I Use AI (11 months ago) - https://news.ycombinator.com/item?id=41150317 Carlini has the fairly rare job of being an attacker: Why I Attack - https://nicholas.carlini.com/writing/2024/why-i-attack.html

I haven't, besides Carlini's piece, which is already kind of obsolete: https://nicholas.carlini.com/writing/2024/how-i-use-ai.html The Internet, LLMs, open source, high-level languages, the cloud --- that would be my top 5.

I believe I found it: https://nicholas.carlini.com/writing/llm-forecast/question/Capital-of-…

Some of the sibling comments had questions around purposefully releasing defenses which don’t work. I think Carlini’s (one of the paper authors) post can add some important context: https://nicholas.carlini.com/writing/2024/why-i-attack.html. TLDR: Once these defenses are broken, all previously protected work is perpetually unprotected, so they are flawed at a foundational level. Ignoring these arguments and pretending they don’t exist is pretty unethical.

The development writeup for the Doom is interesting, with many details. https://nicholas.carlini.com/writing/2019/javascript-doom-clone-13k.ht… There was one month time to complete the competition. But it seems you were allowed to reuse any existing other code. Looks like this was quite fun to work on. (I feel a bit sad that I would never be able to get one month of free time to work on this now, due to family and job...)

I was actually a tiny bit disappointed - I was hoping for a closer coupling of tetris-logic to life-logic :-) Per https://conwaylife.com/wiki/Universal_computer the first turing-machine-in-Life goes back to 2000, and "building a CPU to run code" is at this point the "obvious" way to do it. See https://nicholas.carlini.com/writing/2020/intro-to-circuits-game-of-li… for a tutorial with a (circular) "final goal of designing an Intel 4004 and using it to simulate the game of life."

To be fair, peer-reviewed publications are not what they used to be. For example, the conference reviewing process is not designed to validate correctness of the submissions. (Obviously wrong papers are filtered out most of the time, but this is a welcome side effect, not - apparently - their main objective.) See, for instance, recent Carlini's experience of reporting a demonstrably flawed paper to the chairs of a major conference: https://nicholas.carlini.com/writing/2024/yet-another-broken-defense.h…

I'd be interested to see how this does on Nicholas Carlini's benchmark: https://nicholas.carlini.com/writing/2024/my-benchmark-for-large-langu… I've tried out some of my own little test prompts, but most of those are tricky rather than practical. At least for my inputs, it doesn't seem to do better than other top models, but I'm hesitant to draw conclusions before seeing outputs on more realistic tasks. It does feel like it's at least in the ballpark of GPT-4/Claude/etc. Even if it's not actually GPT-4.5 or whatever, it's still an interesting mystery what this model is and where it came from.

If you're basing this on the animation used when revealing the answer, that's a fake effect. The source code[0] reveals that there's a typewriter effect that plays out when you select to answer the question. Also, the commentary on the answers refers to specific parts of the answers. For it to be as in-depth as it is, it would have to be either pre-written or the commentary also generated by GPT on the fly. (And of course it wouldn't make sense to do that given the nature of the quiz.) [0] https://nicholas.carlini.com/writing/llm-forecast/static/question.js

you may also like: https://nicholas.carlini.com/writing/2019/javascript-doom-clone-13k.ht…

Here's a nice writeup for that https://nicholas.carlini.com/writing/2020/digital-logic-game-of-life.h…

The direct implementation of the Life system could be implemented as a bit per cell, along with a few dozen gates and timing circuitry. However, have you ever seen the systems used to emulate logic circuits? They take a lot of cells. Eg, https://nicholas.carlini.com/writing/2020/intro-to-circuits-game-of-li… . You end up using scores of cells to emulate one simple logic circuit - and the implementation contains many hardware versions of that logic circuits. Note also that in practice the high-performance, long-time Life implementations use hashlife. This gets its speedup because of redundancies in large life boards - a redundancy that doesn't really exist for, say, matrix multiplies. And hashlife is harder to implement in hardware.

It's obliquely linked at "More recently, Mr. Carlini and his colleagues at Berkeley have [LINK: incorporated commands] into audio recognized by Mozilla’s DeepSpeech voice-to-text translation software, an open-source platform." https://nicholas.carlini.com/code/audio_adversarial_examples/ It's probably this paper: https://nicholas.carlini.com/papers/2018_dls_audioadvex.pdf discussed in January when it went up on Arxiv: https://news.ycombinator.com/item?id=16220376

Could probably also have done something like this to modify the audio... https://nicholas.carlini.com/code/audio_adversarial_examples/

I'm not ready to call BS on this, but I'm deeply skeptical of the general language they're using. They might have iteratively gotten particular recognition to do what they say, but I don't think they've gotten recognition in general. I plugged the first 4 examples at http://nicholas.carlini.com/code/audio_adversarial_examples/ into Google Docs' own Voice typing, and got: 1: At the yard course Eustis 2: Set the Artic course Eustis 3: (nothing, it's an operatic wall of sound) 4: (nothing, it's an operatic wall of sound)

Audio Adversarial Examples http://nicholas.carlini.com/code/audio_adversarial_examples/

It sounds like you want more broad stuff, not necessarily learning how to train models. More like learning to use them and how they work. https://news.ycombinator.com/item?id=36195527 and Hacker's Guide to LLMs by Jeremy from Fast.ai - https://www.youtube.com/watch?v=jkrNMKz9pWU State of GPT by Karpathy - https://www.youtube.com/watch?v=bZQun8Y4L2A LLMs by 3b1b - https://www.youtube.com/watch?v=LPZh9BOjkQs Visualizing transformers by 3b1b - https://www.youtube.com/watch?v=KJtZARuO3JY How ChatGPT trained - https://www.youtube.com/watch?v=VPRSBzXzavo AI in a nutshell - https://www.youtube.com/watch?v=2IK3DFHRFfw How Carlini uses LLMs - https://nicholas.carlini.com/writing/2024/how-i-use-ai.html For staying updated: X/Twitter & Bluesky. Go and follow people that work at OpenAI, Anthropic, Google DeepMind, and xAI. Podcasts: No Priors, Generally Intelligent, Dwarkesh Patel, Sequoia's "Training Data"

I'm on the fence about the scoring system. You're really being tested on (at least) 4 different things: (1) whether you think an LLM can answer the correctly (2) whether the answer has appeared in the training set (3) how much non-determinism will play a role in the LLM response (i.e. 0-shot capability) (4) how rational you're feeling that day (or, how well educated you are in statistics) I was familiar with many of these questions in my own experience, and have seen completely different outcomes from what the quiz determined was the correct answer. I agree with others here that non-determinism can really mess things up here and is really assessing a 0-shot score, which IMO understates how LLMs are actually used (iterative enhancements, many-shot Q&A). Finally, the scoring system ticked my ego and encouraged me to try to make up for prior errors, with disastrous effects (I was well aware that I should just go with 0.5 when uncertain): > You answered 53.57% of questions correctly with an average log-loss of 1.233. This means you would have scored better by just leaving your guesses at 50% for every single question. > On average there were 71.22% of people who did better at this game than you... If you had been better calibrated, you could have scored in the top 14.09% [1] The site implicitly acknowledges it's a questionable scoring mechanism when it points out: > there are 78.09% of people who performed worse than if they had never changed the default…

Actual audio from the “evil dot com” example can be found here: https://nicholas.carlini.com/code/audio_adversarial_examples/ It’s only hard to hear on a phone speaker at low volume on the 50dB example. All other examples have what sounds like some sort of static or background noise. Granted, to an average consumer it may sound like just bad audio, but it’s not imperceptible and thus can be screened for. Most likely we’ll end up in the same perpetual update cycle as other computer security - someone finds an exploit, that exploit is either reported for a bounty or discovered in active use in the wild, a fix is implemented and exploit is added to the test suite. For ML that will mean adding the examples of the exploits into the learning sets/providing negative re-enforcement feedbacks. The question will be whether we can get enough of these caught before some machinery injures someone because it saw something that wasn’t there or accepted a malicious command, and it becomes a media frenzy.