Neel Nanda

Recommending Gwern on any technical topic is practically cheating; he always has in-depth, impeccably referenced overviews, complete with experiments he has done. For deep learning in particular, I will add Neel Nanda's interpretability work: https://www.neelnanda.io/mechanistic-interpretability

I was not familiar with it, and that does look fascinating, thank you. If anyone else is interested, this guide "Concrete Steps to Get Started in Transformer Mechanistic Interpretability" on his site looks like a great place to start: https://www.neelnanda.io/mechanistic-interpretability/getting-started

Some interesting follow-ups on OthelloGPT https://www.neelnanda.io/mechanistic-interpretability/othello

I procrastinate as well. What I find has been working for me lately is not making it about the thing itself, but about who I'm becoming by doing it (and doing it early), as described here: https://www.neelnanda.io/blog/become-a-person-who-actually-does-things (the rest of the blog is fantastic if you want to check it out, there's many posts on personal improvement) I was also pointed to this resource by Neel, you could use to it to re-frame the goal of the task (e.g. instead of "do well on exam" -> "pass exam with as little effort as possible"): https://mindingourway.com/half-assing-it-with-everything-youve-got/

Hey! Author of the piece here. I somewhat disagree, I think experiments are a great way to overcome wasted motion. Sure, only 20% will succeed. But there's a major asymmetry with trying new things - you can drop the failures after trying it once, but can keep doing the successes again and again. For example, if I want to find a new hobby, I should try lots of things. 80% will be boring and I can drop it after one session, but 20% could be incredibly fun, and worth doing for the rest of my life. I flesh this point out in this post: https://www.neelnanda.io/blog/33-upside-risk

> This is speculation at best. Models are black boxes, even to those who make them. We can't discern a "meaningful internal representation" in a model, anymore than a human brain. They are not pure black boxes. They are too complex to decipher, but it doesn't mean we can't look at activations and get some very high level idea of what is going on. For world models specifically, the paper that first demonstrated that LLM has some kind of a world model corresponding to the task it is trained on came out in 2023: https://www.neelnanda.io/mechanistic-interpretability/othello. Now you might argue that this doesn't prove anything about generic LLMs, and that is true. But I would argue that, given this result, and given what LLMs are capable of doing, assuming that they have some kind of world model (even if it's drastically simplified and even outright wrong around the edges) should be the default at this point, and people arguing that they definitely don't have anything like that should present concrete evidence ot that effect. > We're just highly advanced "stochastic parrots" with a game loop. If that is your assertion, then what's the point of even talking about "stochastic parrots" at all? By this definition, _everything_ is that, so it ceases to be a meaningful distinction.

LLM based AIs have lots of "features" which are kind of synonymous with "concepts" - these can be anything from `the concept of an apostrophe in the word don't`, to `"George Wash" is usually followed by "ington" in the context of early American History`. Inside of the LLMs neural network, these are mapped to some circuitry-in-software-esque paths. We don't really have a good way of understanding how these features are generated inside of the LLMs or how their circuitry is activated when outputting them, or why the LLMs are following those circuits. Because of this, we don't have any way to debug this component of an LLM - which makes them harder to improve. Similarly, if LLMs/AIs ever get advanced enough, we'll want to be able to identify if they're being wilfully deceptive towards us, which we can't currently do. For these reasons, we'd like to understand what is actually happening in the neural network to produce & output concepts. This domain of research is usually referred to as "interpretability". OpenAI (and also DeepMind and Anthropic) have found a few ways to inspect the inner circuitry of the LLMs, and reveal a handful of these features. They do this by asking questions of the model, and then inspecting which parts of the LLM's inner circuitry "lights up". They then ablate (turn off) circuitry to see if those features become less frequently used in the AIs response as a verification step. The graphs and highlighted words are visual representations of concepts…

Have a look at this research paper[1] and follow-up post[2]. The upshot is that the researchers trained a model of the same shape that the main language models are to play Othello. The model started with no knowledge of the game, and was fed a bunch of move sequences (e.g. "c4 c3 d3 e3"). The researchers were then able to take that model, and figure out, by looking only at the activations, what the board state was. Changing the activations so that they modeled a different board state caused the model to play moves that made sense with the new board state, and did not make sense with the original board state. And then the follow-up research[2] established that, not only can you determine the board state by looking at the activations, but that you can trivially do so. Specifically, if you look at the residual stream after layer 4, you can build a linear classifier for whether each of the 64 squares is empty, and whether each of the 64 squares contains a token owned by the player whose turn it is. If you bump the values in the direction implied by each of those classifiers, you can make the model output moves in the same way as if the tokens on those squares had the opposite color. There's even a colab notebook[3] you can play with. That research was on Othello, not chess, but I'm pretty sure that "LLMs are able to develop models of the world if doing so helps them predict the next token better" generalizes to chess too. [1] https://arxiv.org/pdf/2210.13382.pdf [2]…

I am building https://linklonk.com and I think it adds a social proof to content discovery while preserving anonymity. Here is how it works: - When you upvote an item, you connect stronger to users that also posted that item and to RSS feeds that posted this. For example, if you upvote "Post 43: Intentionally Making Close Friends — Neel Nanda"(https://linklonk.com/item/827619236936941568) then you will get connected to 6 users that upvoted that article and the RSS feed https://www.neelnanda.io/blog?format=rss - The recommendation algorithm shows you other items upvoted by users and feeds you are connected to. - At the top of you recommendations you see content from users and feeds that you are most strongly connected to (ie, those who have posted more useful content for you). With this mechanism you discover users and feeds that post great content just by rating content, without having to know the users personally. Yet, when you see your recommendations you know that they are coming not from random people, but from people who found useful for you content before. I think, that is a more meaningful version of social proof than the aggregate counts of likes in the traditional social media.

Neel Nanda is also very active in the field and writes some potentially more approachable articles, if you're interested: https://www.neelnanda.io/mechanistic-interpretability Much of their work is focused on discovering "circuits" that occur between layer activations as they process data, which correspond to dynamics the model has learned. So, as a simple hypothetical example, instead of embedding the answer to 1 million arbitrary addition problems in the weights, models might learn a circuit that approximates the operation of addition.

You seem to believe "concept" is a concept in English language. It is not. Concept is an abstraction layer above human languages. Here's a good article that touched on this topic: https://www.neelnanda.io/mechanistic-interpretability/glossary

> I'd argue that the biggest reason machines are black boxes are because no one is bothering to look inside of them. People do look, but it's extremely hard. Take a look at how hard the mechanistic interpretability people have to work for even small insights. Neel Nanda[1] has some very nice writeups if you haven't already seen them. [1]: https://www.neelnanda.io/mechanistic-interpretability

Related work: Interpreting Modular Addition in MLPs https://www.lesswrong.com/posts/cbDEjnRheYn38Dpc5/interpreting-modular… Paper Replication Walkthrough: Reverse-Engineering Modular Addition https://www.neelnanda.io/mechanistic-interpretability/modular-addition…

I find this article a nice exploration of slack concept in personal lives https://www.neelnanda.io/blog/38-slack

Neel Nanda did it better: https://www.neelnanda.io/blog/34-learning

Part 2: Prioritisation, Part 2 - Achieving Goals — https://www.neelnanda.io/blog/mini-blog-post-17-prioritisation-part-2-…

Not going to comment on the thinking part, because who knows what that means, but there's evidence that transformers do in fact learn predictive models of their input space. There's a cool blog post on this here: https://www.neelnanda.io/mechanistic-interpretability/othello

>We don't build our world model fresh with every sentence someone says to us Neither do LLMs. The state for the current text perhaps. Definitely not the entire world model(s) which is learnt from the training process and stored in its weights. >They quite literally have no ability to have a "world model" You keep repeating this so let's get one thing straight. You're wrong. You're just wrong. I'm not trying to convince you of my opinion. This has been empirically observed and tested multiple times. https://www.neelnanda.io/mechanistic-interpretability/othello https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world… You're literally saying absolute nonsense with a high level of confidence. When an LLM does this, somehow it's a "hallucination". Why are you different ?

This post we're commenting on. Also https://www.neelnanda.io/mechanistic-interpretability/othello Also, there's a paper I can't find now which shows you can find the name of the operation in the middle layers if you provide example of what to do. That is, if you prompt: "9 2 5 -> 5 2 9, 0 7 3 ->", you'll find the internals generate "reverse" even though it doesn't appear in the output.

But the human mental model is purely internal. For that matter, there is strong evidence that LLMs generate mental models internally. [1] Our interface to motor actions is not dissimilar to a token predictor. > The mental model we build and update is not just based on a linear stream, but many parallel and even contradictory sensory inputs So just like multimodal language models, for instance GPT-4? > as experiences in a world of which we are part of. > The simple fact that we don't just complete streams, but do so with goals, both immediate and long term, and fit our actions into these goals Unfalsifiable! GPT-4 can talk about its experiences all day long. What's more, GPT-4 can act agentic if prompted correctly. [2] How do you qualify a "real goal"? [1] https://www.neelnanda.io/mechanistic-interpretability/othello [2] https://github.com/hwchase17/langchain

Before we had proper GPUs everyone said the same thing about Neural Networks. Current model architectures are optimized to get the most out of GPUs, which is why we have transformers dominating as they're mostly large dense matrix multiplies. There's plenty of work showing transformers improve with inner dimension size but it's not feasible to scale them up further because it blows up parameter and activation sizes (including KV caches) so people to turn to low rank ("sparse") decompositions like MLA. Lottery ticket hypothesis shows that most of the weights in current models are redundant and that we could get away with much smaller sparse models, but currently there's no advantage to doing so because on GPUs you still end up doing dense multiplies. Plenty of mech interp work shows that models are forced to commingle different concepts to fit them into the "low" dimensional vector space. (https://www.neelnanda.io/mechanistic-interpretability/glossary) https://arxiv.org/abs/2210.06313 https://arxiv.org/abs/2305.01610

> LLMs can't think. They are generating tokens one at a time Huh? They are generating tokens one at a time - sure that's true. But who's shown that predicting tokens one at a time precludes thinking? It's been shown that the models plan ahead, i.e. think more than just one token forward. [1] How do you explain the world models that have been detected in LLMs? E.g. OthelloGPT [2] is just given sequences of games to train on, but it has been shown that the model learns to have an internal representation of the game. Same with ChessGPT [3]. For tasks like this, (and with words), real thought is required to predict the next token well; e.g. if you don't understand chess to the level of Magnus Carlsen, how are you going to predict Magnus Carlsen's next move... ...You wouldn't be able to, even just from looking at his previous games; you'd have to actually understand chess, and think about what would be a good move, (and in his style). [1] https://www.anthropic.com/research/tracing-thoughts-language-model [2] https://www.neelnanda.io/mechanistic-interpretability/othello [3] https://adamkarvonen.github.io/machine_learning/2024/01/03/chess-world…

I want to highlight this blog article which in my opinion gives more actionable advice than all of the books listed (I read 6/8 of those) on the topic of having good social life https://www.neelnanda.io/blog/mini-blog-post-23-taking-social-initiati… tldr; taking social initiative is like a cheat code. The easiest way to make friends is to start organizing. I started applying this approach to my social life and I created two new friends groups from scratch. 1. Me and my brother started organizing lasertag matches. Basically anyone is invited. We even post open invitation to all our friends. Right now we have 20 people in our group chat and we do lasertags + beers once a month. 2. I created a group of friends from highschool - like a reunion shit but once every month. When I reached out to people everyone was hyped. I had only one of these meetings atm but everyone was happy and said they want more. We will be having a second one soon. If I needed more friends that's what I would be focusing on. Organizing a cyclic get together of people. It's like a cheat code for having a lot of friends. Right now I am a bit of time constrained but I look forward to organizing something different in the future.

>What GPT3.5 does is not quite playing chess but creating realistic chess moves that a human might make. No it's playing games. And if you're not at about the level I spoke of, you will lose repeatedly over whatever number or stretch of games you imagine. https://github.com/adamkarvonen/chess_gpt_eval 3.5 Instruct (different model from regular 3.5 that can't play) can play chess. There's no trick. Any other framing seems like a meaningless distinction. The goal is to model the chess games and there's no better way to do that than to learn to play the game. >all it needs is grounding (by feeding it the current board state) The Model is already constructing a board state to play the game. https://www.neelnanda.io/mechanistic-interpretability/othello >agency (RF to reward the model for winning games) Predict the next token loss is already rewarding models for winning when the side they are predicting wins. And when the preceeding text says x side wins then the loss is rewarding for knowing how to make a side win. Nothing else is really necessary.

I know this discussion is a bit old at this point, but I came across this[1] essay for the first time today, and this shows more of what I was trying to get across earlier in the thread. Hopefully you'll find it interesting. Essentially, they trained a GPT on predicting the next move in a game of Othello, and by analyzing the weights of the network, found that the weights encode an understanding of the game state. Specifically, given an input list of moves, it calculates the positions of its own pieces and that of the opponent (a tricky task for a NN given that Othello pieces can swap sides based on moves made on the other side of the board). Doing this allowed it to minimize loss. By analogy, it formed a theory about what makes moves legal in Othello (in this case, the positions of each player's pieces), and found out how to calculate those in order to better predict the next move. [1] https://www.neelnanda.io/mechanistic-interpretability/othello

You mentioned that you're not sure about how to improve the one-sided empathy situation with your SO. So I thought it might be helpful to share some pertinent info that I've come across on HN in the past. I feel that the article and comment discussion on the "Intentionally making close friends" post (https://news.ycombinator.com/item?id=33774353) may offer some actionable advice. Also, see the authors other post specifically on "debugging others" (https://www.neelnanda.io/blog/30-debugging-others). My own advice is to be direct with your SO about your needs / expectations not being met. I went through the same thing with my SO. We went to couples therapy for many years with no improvement, until eventually they realized it was serious enough of a problem to lead to the end of the relationship, prompting them to put in the necessary effort to change. Another poster mentioned Gottman "bids for attention" which is spot on as well. Both partners in the relationship need to show up for each other to provide emotional support. Some people are simply more used to receiving support and don't know how / aren't used to providing it. It should be made clear that this is unacceptable and needs to change, but also that you are willing to give it time and work with your SO to help them gradually improve their empathetic capacity so that they can be a better partner to you (and better person in general) and keep your relationship healthy. I think it's also important to note that balance…