David Noel Ng

Found it: https://news.ycombinator.com/item?id=47500709 Part 3 might be the best introduction: https://dnhkng.github.io/posts/sapir-whorf/ tl;dr: Based on experiments with similar prompts translated to different languages LLM layers group into three phases: the first decodes from the source language into an abstract space, the middle does something, then there's a last part where the abstract result gets transformed back to the target language. And you can repeat the middle to get a stronger model. Which neatly fits Anthropic's findings here that something similar to CoT is happening in those middle layers Three months ago. I wonder if Anthropic's J-Space research was actually inspired by those blog posts

Would recommend reading https://dnhkng.github.io/posts/rys/ and check his github code to reproduce the findings. Easy to way to start hacking LLMs; there's much of value there and a fun way to get into it before tackling heavy math / CS topics.

Source for those interested https://dnhkng.github.io/posts/rys/

I keep thinking of the RYS (Repeat Yourself) experiment of simply looping some of the inner layers of LLMs for better results and wonder if any progress was made on it. https://dnhkng.github.io/posts/rys/ Feels it should be straightforward to integrate in LLMs a network to control the looping. Or just duplicate entire blocks of layers after the initial training.

Yes you can! Well, mostly, depends on how pedantic you are with definitions: you can transplant layers but not weights, which in common parlance are conceptually similar. But usually it isn’t a good idea for a few reasons. There’s a really fascinating example[1] where a guy identifies a particular set of layers and transplants them. Overgeneralizing, early layers are encoders and the later layers are decoders and in the middle some blocks seem to do specific things or tasks related “reasoning”. So you can actually create a FrankenLLM and it sometimes works. This needs architectures to be roughly similar however and internal representations to be consistent-ish so for “stealing” it’s not really a thing (other practical concerns aside) [1] https://dnhkng.github.io/posts/rys/

You can also edit binary distributions of models with means besides changing their weights. See "LLM Neuroanatomy: How I Topped the LLM Leaderboard Without Changing a Single Weight." On the spectrum of: careful engineering--hacking--mad science This kind of thing falls far towards the mad science end of the scale, but has proven effective. https://dnhkng.github.io/posts/rys/

This reminds me of https://dnhkng.github.io/posts/rys/ David looks into the LLM finds the thinking layers and cut duplicates then and put them back to back. This increases the LLM scores with basically no over head. Very interesting read.

There is an interesting on-going research https://dnhkng.github.io/posts/sapir-whorf/ that shows LLMs think in a language-agnostic way. (It will probably get posted to HN after it is finished.)

apparently you can straight up duplicate/add/rearrange layers without changing any of the weights and get better results as well - https://dnhkng.github.io/posts/rys/

You're right that my quote was misleading, I overlooked "the weird part" in the post because it didn't seem new to me either. Here's the section in the original post that covers it: https://dnhkng.github.io/posts/rys/#the-brain-scanner All heatmaps are split by tasks and show an optimal point for each. The resulting routing he chose is a trade-off for both tasks, there isn't much else to do unless you intend to train a router anyway. > So the ‘math organ’ has boundaries on both sides. Too few layers and you get nothing — you’ve cut into the circuit and it can’t complete its operation. Too many layers and you also get nothing — you’ve included tissue from a neighbouring circuit that doesn’t belong. Pre-training carved these structures out of the layer stack, and they only work whole. It also doesn’t translate to other tasks, as the heatmap for EQ scores doesn’t have this patch.

Conceptually, this seems a good direction. The other piece that has always struck me as a huge inefficiency with current usage of LLMs is the hoops they have to jump through to make sense of existing file formats - especially making sense of (or writing) complicated semi-proprietary formats like PDF, DOC(X), PPT(X), etc. Long-term prediction: for text, we'll move away from these formats and towards alternatives that are designed to be optimal for LLMs to interact with. (This could look like variants of markdown or JSON, but could also be Base64 [0] or something we've not even imagined yet.) [0] https://dnhkng.github.io/posts/rys/

Yes! I tried that pretty early on, the its basically never good. Its described in the the section: https://dnhkng.github.io/posts/rys/#the-beginning-of-llm-neuroanatomy