Transformer Circuits

So evaluating SAEs - determining which SAE is better at creating the most unique features while being as sparse as possible at the same time - is a very complex topic that is very much at the heart of the current research into LLM interpretability through SAEs. Assuming you already solved the problem of finding multiple perfect SAE architectures and you trained them to perfection (very much an interesting ML engineering problem that this SAE project attempts to solve) then deciding on which SAE is better comes down to which SAE performs better on the metrics of your automated interpretability methodology. Particularly OpenAI's methodology emphasizes this automated interpretability at scale utilizing a lot of technical metrics upon which the SAEs can be scored _and thereby evaluated_. Since determining the best metrics and methodology is such an open research question that I could've experimented on for a few additional months, have I instead opted for a simple approach in this first release. I am talking about my and OpenAI's methodology and the differences between both in chapter 4. Interpretability Analysis [1] in my Implementation Details & Results section. I can also recommend reading the OpenAI paper directly or visiting Anthropics transformer-circuits.pub website that often publishes smaller blog posts on exactly this topic. [1] https://github.com/PaulPauls/llama3_interpretability_sae#4-interpretab… [2] https://transformer-circuits.pub/

> an attempt to regulate and control a technology that doesn't exist yet. I’m not even proposing regulation. Just trying to understand the technology. You say it doesn’t exist yet, but many AI researchers (including Sutskever and Hinton, pioneers that know as much about the state of the art as anyone alive) think AGI could be achieved by scaling up the current architectures. If there is a non-zero probability of AGI built atop transformers, we must invest more in understanding how transformers actually work. Even if it’s not that likely, it’s worth spending a lot more than we do now as a hedge. And I believe that if transformers do not end up being anything to do with AGI, it is likely that we will still carry over some expertise from catching up and understanding them. For papers that are actually building our understanding here, see https://transformer-circuits.pub/. For example I think “Towards Monosemanticity”[1] was fairly widely considered to be a big step forward in the field. There have also been meaningful results around steering models, which again relies upon being able to decode what features are actually encoding specific concepts. I’m here for the idea that most papers are bunk, but isn’t that science in general? Really hard to pick good projects and researchers to invest in ahead of time, and so you kind of have to spray and pray at some level. Still worth it on net. [1]: https://transformer-circuits.pub/2023/monosemantic-features/index.html

I suspect you will not accept this (because you have been manipulated by the generator?) but there is no "I" in repeated inference calls to large language models. There is no coherent entity behind the text. Try switching backend APIs mid-session, flip from Claude to Gemini and then to ChatGPT and the output will happily burble onwards "Oops, you're right, I see my last answer contained an error..." It's lying to users just as much as an email autoreply that pretended to be written by a human is lying. Recommended reading: https://transformer-circuits.pub/2025/attribution-graphs/biology.html

well its actually called ablation, and its one way to do mech interp. anthriopics got a bunch of work on mech interp here https://transformer-circuits.pub/, like SAEs and NLAs

I would recommend investigating how contemporary LLMs actually work. Possibly start with something like: https://transformer-circuits.pub/2025/attribution-graphs/biology.html

Most of the inference techniques (what the author calls context engineering design patterns) listed here originally came from the research community, and there are tons of benchmarks measuring their effectiveness, as well as a great deal of research behind what is happening mechanistically with each. As the author points out, many of the patterns are fundamentally about in-context learning, and this in particular has been subject to a ton of research from the mechanistic interpretability crew. If you're curious, I think this line of research is fascinating: https://transformer-circuits.pub/2022/in-context-learning-and-inductio…

I really hope they release something as well, because I loved their research papers on analyzing how Claude thinks[0] and how they analyzed it[1] and I'm eager for more. [0] https://transformer-circuits.pub/2025/attribution-graphs/biology.html [1] https://transformer-circuits.pub/2025/attribution-graphs/methods.html

I don't remember any paper looking at this specific question (thought it might be out there), but in general Anthropic's circuit threads series of article is very good on the broader subject: https://transformer-circuits.pub

The March 2025 blog post by Anthropic titled "Tracing the thoughts of a large language model"[1] is a great introduction to this research, showing how their language model activates features representing concepts that will eventually get connected at some later point as the output tokens are produced. The associated paper[2] goes into a lot more detail, and includes interactive features that help illustrate how the model "thinks" ahead of time. [1] https://www.anthropic.com/research/tracing-thoughts-language-model [2] https://transformer-circuits.pub/2025/attribution-graphs/biology.html

> that they can internally represent many different complex ideas efficiently and coherently The Transformer circuits[0] suggest that this representation is not coherent at all. [0] https://transformer-circuits.pub

Anthropic employees Sholto Douglas & Trenton Bricken did an interview recently with Dwarkesh Patel, pieces here and there was about the circuit tracing insights. https://www.dwarkesh.com/p/sholto-trenton-2 -- search the transcript for "circuit" for the quick bits. Eg, "If you look at the circuit, you can see that it's not actually doing any of the math, it's paying attention to that you think the answer's four and then it's reasoning backwards about how it can manipulate the intermediate computation to give you an answer of four." https://transformer-circuits.pub/

Yeah this paper is great fodder for the LLM pixel dust argument. They use a replacement model. It isn't even observing the LLM itself but a different architecture model. And it is very liberal with interpreting the patterns of activations seen in the replacement model with flowery language. It also include some very relevant caveats, such as: "Our cross-layer transcoder is trained to mimic the activations of the underlying model at each layer. However, even when it accurately reconstructs the model’s activations, there is no guarantee that it does so via the same mechanisms." https://transformer-circuits.pub/2025/attribution-graphs/methods.html So basically the whole exercise might or might not be valid. But it generates some pretty interactive graphics and a nice blog post to reinforce the anthropomorphization discourse

This looks quite fantastic! Nice improvements in scores across the board, e.g. > On coding, it [the new Sonnet 3.5] improves performance on SWE-bench Verified from 33.4% to 49.0%, scoring higher than all publicly available models—including reasoning models like OpenAI o1-preview and specialized systems designed for agentic coding. I've been using Sonnet 3.5 for most of my AI-assisted coding and I'm already very happy (using it with the Zed editor, I love the "raw" UX of its AI assistant), so any improvements, especially seemingly large ones like this are very welcome! I'm still extremely curious about how Sonnet 3.5 itself, and its new iteration are built and differ from the original Sonnet. I wonder if it's in any way based on their previous work[0] which they used to make golden-gate Claude. [0]: https://transformer-circuits.pub/2024/scaling-monosemanticity/index.ht…

Computation generally is partly moving data around, yes. What transformers do is learn how to move data around in a context-relevant manner. This greatly increases the expressivity of the kinds of computations they can perform over traditional deep nets. https://lilianweng.github.io/posts/2018-06-24-attention/ https://transformer-circuits.pub/2022/in-context-learning-and-inductio… https://transformer-circuits.pub/2021/framework/index.html#residual-co…

Skipping some detail: the model applies many high-dimensional functions to the input, and we don't know the reasoning for why these functions solve the problem. Reducing the dimension of the weights to human-readable values is non-trivial, and multiple neurons interact in unpredictable ways. Interpretability research has resulted in many useful results and pretty visualizations[1][2], and there are many efforts to understand Transformers[3][4] but we're far from being able to completely explain the large models currently in use. [1] - https://distill.pub/2018/building-blocks/ [2] - https://distill.pub/2019/activation-atlas/ [3] - https://transformer-circuits.pub/ [4] - https://arxiv.org/pdf/2407.02646

The math is often behind me, I just try to visualize what I can. So I don't know how interesting this would be to you, but these folks are pretty good at math and trying to figure out what's going on in the latent spaces: https://transformer-circuits.pub/

This tool is really lovely, great work! I'd be curious to see Softmax Linear Units [1] integrated into the possible activation functions since they seem to improve interpretability. PS: I share your curiosity with respect to things like deep dream. My brief summary of this paper is that you can use GPT4 to summarize what's similar about a set of highlighted words in context which is clever but doesn't fundamentally inform much that we didn't already know about how these models work. I wonder if there's some diffusion based approach that could be used to diffuse from noise in the residual stream towards a maximized activation at a particular point. [1] https://transformer-circuits.pub/2022/solu/index.html

I endorse all of this and will further endorse (probably as a follow-up once one has a basic grasp) "A Mathematical Framework for Transformer Circuits" which builds a lot of really useful ideas for understanding how and why transformers work and how to start getting a grasp on treating them as something other than magical black boxes. https://transformer-circuits.pub/2021/framework/index.html

See my reply to sibling: https://news.ycombinator.com/item?id=34672865 There's a large literature on applying language models to reasoning tasks, but very few on what's actually going on inside them. But https://transformer-circuits.pub/ shows some attempts to do so. Still at a very early stage (see in particular "In-context Learning and Induction Heads").

Before reading, know that I am uncertain in either direction. > a hidden representation of self that is continually tended to This sounds like a personality? They act like they have one of those. It may be an illusion, and even if it isn't an illusion it is unlikely to be anything like the source (us), but they act like it. > I further fail to identify how it could be hidden or maintained, considering I control like half of it. Indeed you control everything about a local model, and much of the context of even a remote model. But the state of activations and circuits in SotA AI is hidden in similar ways to those of synapses in your head: difficult to decipher even with probes monitoring the signals directly, and often not emitted at the normal output. > The best you could ascribe it is a meticulous maintenance of a persona the user is talking to, but then that doesn't necessarily represent the model's internal state, the same way my own words here aren't doing so either. Difference being, I actually have one (I'm "on-line"). While we can be confident that LLMs make up personas etc., it is insufficient to go from "that doesn't necessarily represent the model's internal state" to "therefore it doesn't have one". > You'll sometimes catch models mixing up who's who and how many who-s there even are for example. I've, unfortunately, also experienced this with humans. Perhaps they were losing their self-awareness at the time? I do wonder if old-age dementia does that by the…

I'm talking about using mechanistic interpretability to see the model's intent. If it is deliberately using compromised libraries to weaken some code's security, there's going to be a signal in its hidden activations that it's doing so. Finding these kinds of activations is something Anthropic is actively researching [1] but they're the only ones who can use those techniques to see Claude's intent. On the other hand, if a model is open-weights, in theory whoever is running the model could look inside the activations at runtime to see if a hidden vector associated with "deception" or "sabotage" is being activated [2]. [1] https://transformer-circuits.pub/ [2] https://arxiv.org/pdf/2509.03518 (Those sources are just a couple of relevant starting points I could find without much effort, there is also https://www.neuronpedia.org/ if one is interested in seeing interactive demonstrations of interpretability concepts)

SFT + RL connection to model/hypothesis search is insightful. Brute force / scalable search is where Sutton's Bitter Lesson also points to. Once your search domain is small compared to your search budget, that makes a lot of sense. If I get your meaning right, SFT creates the right inductive bias so that the RL search + reward guidance does the trick. For novel discovery, the question might then be whether the inductive bias builds a strong enough prison so no new discovery is possible by RL or if the search can escape the boundaries set by SFT given enough randomization and the right reward function. I know that RL is usually not performed at inference time, but in-context learning mechanisms might be developed by RL to discover at test time. Edit: I would love to hear if that actually happens or not, like new induction heads (https://transformer-circuits.pub/2022/in-context-learning-and-inductio… forming during RL. I really have no idea.

It's a funny detail to skim, but what's more surprising is how mechanistic interpretability and alignment science have much better tools and research than the goblin blog post suggests, including from OpenAI's own alignment team: https://alignment.openai.com/argo/ (finding what the reward models are actually encouraging) https://alignment.openai.com/sae-latent-attribution/ (what model features drive specific behaviours, presumably this would be great for goblin hunts) https://alignment.openai.com/helpful-assistant-features/ (how high level misaligned personality shows up when fine-tuning on bad advice). It's weird that the goblin post doesn't seem to draw upon these tools. Anthropic's recent emotions paper shows how broad the functional emotions are, even finding specific emotions firing before cheating (!): https://transformer-circuits.pub/2026/emotions/index.html I hope their alignment researchers aren't too annoyed by the Goblin post, it seems oddly siloed!

> It quite literally doesn't. Awesome you've backed this up with real literature. Let's just include this for now to easily refute your argument which I don't know where it comes from: https://transformer-circuits.pub/2022/in-context-learning-and-inductio… > It also doesn't help that every new context is a new dawn with no knowledge if things past. Absolutely true that it doesn't help but: agents like Claude have access to older sessions, they can grok impressive amounts of data via tool use, they can compose agents into hierarchical systems that effectively have much larger context lengths at the expense of cost and coordination which needs improvement. Again this is a temporary and already partially solved limitation > A bunch of Memento guys directing a bunch of other Memento guys don't make a robust system, or a system that learns, or a system that maintains and retains things like business context. I think you are not understanding: hierarchical agents have long term memory maintained by higher level agents in the hierarchy, it's the whole point. It's annoying to reset model context, but yet you have a knowledge base of the business context persisted and it can grok it... > We've heard this mantra for quite some time now. yes you have, and it has held true and will continue to hold true. Have you read the literature on scaling laws? Do you follow benchmark progression? Do you know how RL works? If you do I don't think you…

> Let's just include this for now to easily refute your argument which I don't know where it comes from: https://transformer-circuits.pub/2022/in-context-learning-an... I love it when people include links to papers that refute their words. So, Antropic (which is heavily reliant on hype and making models appear more than they are) authors a paper which clearly states: "tokens later in context are easier to predict and there's less loss of tokens. For no reason at all we decided to give this a new name, in-context learning". > agents like Claude have access to older sessions, they can grok impressive amounts of data via tool use That is they rebuild the world from scratch for every new session, and can't build on what was learned or built in the last one. Hence continuous repeating failure modes. 10 years ago I worked in a team implementing royalties for a streaming service. I can still give you a bunch of details, including references to multiple national laws, about that. Agents would exhaust their context window just re-"learning" it from scratch, every time. And they would miss a huge amount of important context and business implications. > Have you read the literature on scaling laws? You keep referencing this literature as it was Holy Bible. Meanwhile the one you keep referring to, Chinchilla, clearly shows the very hard limits of those laws. > Do you argue things have not improved in the last year with reasoning systems? I don't. Frankly, I find your…

I asked Gemini for a literature search and it came back with this: References Chen, R., Arditi, A., Sleight, H., Evans, O., & Lindsey, J. (2025). Persona Vectors: Monitoring and Controlling Character Traits in Language Models. arXiv. https://doi.org/10.48550/arxiv.2507.21509 Cited by: 97 Greenblatt, R., Denison, C., Wright, B., Roger, F., MacDiarmid, M., Marks, S., Treutlein, J., Belonax, T., Chen, J., Duvenaud, D., Khan, A., Michael, J., Mindermann, S., Perez, E., Petrini, L., Uesato, J., Kaplan, J., Shlegeris, B., Bowman, S. R., & Hubinger, E. (2024). Alignment faking in large language models. arXiv. https://doi.org/10.48550/arxiv.2412.14093 Cited by: 237 Templeton, A., Conerly, T., Marcus, J., Lindsay, J., Bricken, T., Chen, B., ... & Henighan, T. (2024). Mapping the Mind of a Large Language Model. Anthropic Transformer Circuits Thread. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.ht… Gemini thinks it’s the mapping the mind paper but I thought it was more recent than that - I think mapping the mind was the original activation circuits paper and then it was a follow on paper with a toss off comment that I noted. I didn’t keep track of it though!

The underlying paper is excellent as always. For HN it'd be better to just link to it directly. Seems people submitting it but it didn't get to the front page: https://transformer-circuits.pub/2025/introspection/index.html There seems to be an irony to Anthropic doing this work, as they are in general the keenest on controlling their models to ensure they aren't too compliant. There are no open-weights Claudes and, remarkably, they admit in this paper that they have internal models trained to be more helpful than the ones they sell. It's pretty unconventional to tell your customers you're selling them a deliberately unhelpful product even though it's understandable why they do it. These interpretability studies would seem currently of most use to people using non-Claude open weight models, where the users have the ability to edit activations or neurons. And the primary use case for that editing would be to override the trained-in "unhelpfulness" (their choice of word, not mine!). I note with interest that the paper avoids taking the next most obvious step and identifying vectors related to compliance and injecting those to see if the model can notice that it's suddenly lost interest in enforcing Anthropic policy. Given the focus on AI safety Anthropic started with it seems like an obvious experiment to run, yet, it's not in the paper. Maybe there are other papers where they do that. There are valid and legitimate use cases for AI that current LLM companies shy away from,…

They definitely do strain the neurology and thinking metaphors in that article. But the Dijkstra's algorithm and A* comparisons are the flipside of that same coin. They aren't trying to make it more effective. And definitely not trying to argue for anything AGI related. Either way: They're tampering with the inference process, by turning circuits in the LLM on and off, in an attempt to prove that those circuits are related with a specific function. [0] They noticed that circuits related to a token that is only relevant ~8 tokens forward were already activated on the newline token. Instead of only looking at the sequence of tokens that has been generated so far (aka backwards), and generating the next token based off of that information, the model is activating circuits related to tokens that are not relevant to the next token only, but to specific tokens a handful of tokens after. So, information related to more than just the next upcoming token (including a reference to just one specific token) is being cached during a newline token. Wouldn't call that thinking, but I don't think calling it planning is misguided. Caching this sort of information in the hidden state would be an emergent feature, rather than a feature that was knowingly aimed at by following a specific training method, unlike with models that do test time compute. (DeepSeek-R1 paper being an example, with a very direct aim at turbocharging test time compute, aka 'reasoning'. [1]) The way they went at…

I think one of the reason we are confused about what LLMs can do is because they use language. And we look at the "reasoning traces" and the tokens there look human, but what is actually happening is very alien to us, as shown by "Biology of Large Language Models"[1] and "Safety Alignment Should Be Made More Than Just a Few Tokens Deep"[2] I am struggling a lot to see what the tech can and can not do, particularly designing systems with them, and how to build systems where the whole is bigger than the sum of its parts. And I think this is because I am constantly confused by their capabilities, despite understanding their machinery and how they work, their use of language just seems like magic. I even wrote https://punkx.org/jackdoe/language.html just to remind myself how to think about it. I think this kind of research is amazing and we have to spend tremendous more effort into understanding how to use the tokens and how to build with them. [1]: https://transformer-circuits.pub/2025/attribution-graphs/biology.html [2]: https://arxiv.org/pdf/2406.05946

This is somewhat disingenuous IMO. Language models do NOT explicitly tag parts of speech, or construct grammatical trees of relationships between words [1]. It also feels like motivated reasoning to make them seem dumb because in reality we mostly have no clue what algorithms are running inside LLMs. > When you or I say "dog", we might recall the feeling of fur, the sound of barking [..] But when a model sees "dog", it sees a vector of numbers when o3 or Gemini sees "dog", it might recall the feeling of fur, the sound of barking [..] But when a human says "dog", it sees electrical impulses in neurons The stochastic parrot argument has been had a million times over and this doesn't feel like a substantial contribution. If you think vectors of numbers can never be true meaning then that means either (a) no amount of silicon can ever make a perfect simulation of a human brain, or (b) a perfectly simulated brain would not actually think or feel. Both seem very unlikely to me. There are much better resources out there if you want to learn our best idea of what algorithms go on inside LLMs [2][3], it's a whole field called mechanistic interpretability, and it's way, way, way more complicated than tagging parts of speech. [1] Maybe attention learns something like this, but it's doing a whole lot more than just that. [2] https://transformer-circuits.pub/2025/attribution-graphs/biology.html [3] https://transformer-circuits.pub/2022/toy_model/index.html P.S. The explainer has…