Interconnects

https://www.interconnects.ai/ has great writing on this; the author is currently working on https://rlhfbook.com/.

To quote Elon Musk in court: > OpenAI’s counsel asked Musk whether xAI has ever “distilled” technology from OpenAI. > Musk: “Generally AI companies distill other AI companies.” > “Is that a yes?” Savitt asked. > Musk: “Partly.” From https://www.interconnects.ai/p/the-distillation-panic which is worth reading in full.

"we have information" says a US Government official who almost certainly has had Anthropic and/or OpenAI on the phone spinning him stories. See also, don't trust anyone in Trump's government who says "we have information". "they distilled us" is fast becoming standard US FUD. The same as people telling me with a serious face that the Chinese models are distilled just because it says "I am Claude". I am not the only one, look at this post on interconnects about Kimi K3 for example:[1] It should be clear looking at this model that if adversarial distillation from the closed frontier models in the U.S. contributed, it is at most to a relatively small degree. AI observers who followed the distillation panic and came away with the wrong conclusion that Chinese AI labs are only producing good models due to IP theft are in for an awakening – that Chinese companies are extremely good at building models in the same way the leading American companies are. [1] https://www.interconnects.ai/p/kimi-k3-the-open-weights-escalation

What makes more sense is to do something like deepseek at one of the major universities put those bright computer science young minds to work, in the good old days almost every major university would have done that has any major US university done that? Stanford Harvard Berkeley if the Chinese can put together a team like deep seek why can’t that be done at a major university in the United States? https://www.interconnects.ai/p/the-american-deepseek-project

Whether or not it's "noise" might depend on if you think Chains of Thought are causally relevant? I liked these pieces if you want to read more about CoT / O1: O1 Technical Primer: https://www.lesswrong.com/posts/byNYzsfFmb2TpYFPW/o1-a-technical-prime… Using Search Was a Psyop: https://www.interconnects.ai/p/openais-o1-using-search-was-a-psyop Value Attribution: https://www.lesswrong.com/posts/FX5JmftqL2j6K8dn4/shapley-value-attrib…

Just search for "System 1 / System 2" and add word "researcher" there, filter by time <1y< and you'll find plenty. OP is right that a bunch of people made far-going predictions, & now doing mental gymnastics to justify they were right all along. Most notably, LeCun https://www.threads.net/@yannlecun/post/DD0ac1_v7Ij and Chollet https://www.youtube.com/watch?v=w9WE1aOPjHc Some argue they were kind of tricked into thinking that; see https://www.interconnects.ai/p/openais-o1-using-search-was-a-psyop and some other writing by Lambert which has turned out pretty much on-point as far as RL and verifiers are concerned.

I did appreciate the discussion on the actual cost. I've been hearing a lot of that $5M number being thrown around, and while it definitely felt unlikely, I didn't have the context. In particular [1] linked from the post was really helpful. [1] https://www.interconnects.ai/p/deepseek-v3-and-the-actual-cost-of

For context: R1 is a reasoning model based on V3. DeepSeek has claimed that GPU costs to train V3 (given prevailing rents) were about $5M. The true costs and implications of V3 are discussed here: https://www.interconnects.ai/p/deepseek-v3-and-the-actual-cost-of

I’m as skeptical of OpenAI as just about anyone, but Marcus just seems to engage in the kind of contrarian ranting that happens to people when they spend too much time on social media. I would say a better indication of o3’s performance is François Chollet seeming kind of bummed that LLMs have reached this point, contrary to his expectations. EDIT: In the interest of not being overly negative myself, I found Nathan Lambert’s more measured and detailed breakdown far more instructive: https://www.interconnects.ai/p/openais-o3-the-2024-finale-of-ai

There's some more commentary on their open-ness in this blog too https://www.interconnects.ai/p/olmo

There are multiple stages of training, and the data/compute mix at each are quite different and produce different "layers" of intelligence. The pretraining stage is the first stage which consists of "next token prediction" on the entire internet, PB of tokens, etc. This is what most people think of when they think of training LLMs, however it produces a "base model" which is not really "intelligent", but rather much like a blurry JPEG of all human language and knowledge. You cannot really talk to such a model; it will simply complete your prompt by producing both sides of the conversation. Note however at some level the training has encoded enough structure through compression that it is able to simulate all sorts of phenomena, from human conversations to code. The next few stages are collectively called post-training, and typically consist of supervised fine-tuning, then reinforcement learning. In supervised fine-tuning, the model is further trained to predict the next token, but on a much more focused data set of natural language conversations where the "assistant" and "user" turns are explicitly delineated with special tokens. The output of this stage is a model which is capable of carrying on proper conversations, but typically with no ability to creatively problem-solve, and less of a personality. The data and compute are many orders of magnitude smaller than in pretraining. The reinforcement learning stage used to be a small part of model training, but ever since…

This is one area where reinforcement learning can help. The way you should think of RL (both RLVR and RLHF) is the "elicitation hypothesis[1]." In pretraining, models learn their capabilities by consuming large amounts of web text. Those capabilities include producing both low and high quality outputs (as both low and high quality outputs are present in their pretraining corpora). In post training, RL doesn't teach them new skills (see E.G. the "Limits of RLVR"[2] paper). Instead, it "teaches" the models to produce the more desirable, higher-quality outputs, while suppressing the undesirable, low-quality ones. I'm pretty sure you could design an RL task that specifically teaches models to use modern idioms, either as an explicit dataset of chosen/rejected completions (where the chosen is the new way and the rejected is the old), or as a verifiable task where the reward goes down as the number of linter errors goes up. I wouldn't be surprised if frontier labs have datasets for this for some of the major languages and packages. [1] https://www.interconnects.ai/p/elicitation-theory-of-post-training [2] https://limit-of-rlvr.github.io

Something that's really hitting me is something brought up in this piece: https://www.interconnects.ai/p/gpt-5-and-bending-the-arc-of-progress When a model comes out, I usually think about it in terms of my own use. This is largely agentic tooling, and I mostly us Claude Code. All the hallucination and eval talk doesn't really catch me because I feel like I'm getting value of these tools today. However, this model is not _for_ me in the same way models normally are. This is for the 800m or whatever people that open up chatgpt every day and type stuff in. All of them have been stuck on GPT-4o unbeknwst to them. They had no idea SOTA was far beyond that. They probably dont even know that there is a "model" at all. But for all these people, they just got a MAJOR upgrade. It will probably feel like turning the lights on for these people, who have been using a subpar model for the past year. That said I'm also giving GPT-5 a run in Codex and it's doing a pretty good job!

That quote is straight up wrong to claim Deepseek has "loathed to reveal their secret sauce". The source of all the excitement is exactly how much they revealed, and I feel like that thread as a whole emphasizes why people who aren't deeply familiar with the pipeline should not get to define these things. "Nick" claims the secret sauce is in something not provided, then posts an article that demonstrates the exact "secret sauce", seemingly not making the connection: https://www.interconnects.ai/p/deepseek-r1-recipe-for-o1 - There is a lot of detail about the nature of the data used and the exact steps needed to reproduce their findings with your own data. They even provide R1-Zero to demonstrate things that might be dead ends just in case someone can continue them. That should be enough to satisfy any useful definition of open source. Even in the same thread you linked: > Just a curiosity, according to the Model Openness Framework from the Linux Foundation, DeepSeek-R1 classifies as an Open Model: > https://mot.isitopen.ai/model/1143 At the end of the day this is as good as it needs to be for LLMs: By their nature a lot of data being used to train them cannot or should not be openly shared, but the shape and motivations behind the data used are able to push others very far along the way to reproduction and iteration.

I am fairly optimistic about LLMs as a human math -> theorem-prover translator, and as a fan of Idris I am glad that the AI community is investing in Lean. As the author shows, the answer to "Can AI be useful for automated mathematical work?" is clearly "yes." But I am confident the answer to the question in the headline is "no, not for several decades." It's not just the underwhelming benchmark results discussed in the post, or the general concern about hard undergraduate math using different skillsets than ordinary research math. IMO the deeper problem still seems to be a basic gap where LLMs can seemingly do formal math at the level of a smart graduate student but fail at quantitative/geometric reasoning problems designed for fish. I suspect this holds for O3, based on one of the ARC problems it wasn't able to solve: https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:… (via https://www.interconnects.ai/p/openais-o3-the-2024-finale-of-ai) ANNs are simply not able to form abstractions, they can only imitate them via enormous amounts of data and compute. I would say there has been zero progress on "common sense" math in computers since the invention of Lisp: we are still faking it with expert systems, even if LLM expert systems are easier to build at scale with raw data. It is the same old problem where an ANN can attain…

The big question is whether or not o3 is using any type of “meta-generation” algorithm at inference time, I.e are there multiple invocations of the LLM generation at all, or does it generate an insanely long reasoning trace in a single autoregressive stream that some somehow implicitly has search-like behavior? In other words, is the search-like behavior learned entirely in post-training and only implicitly exhibited at inference time, or is it explicitly done at inference time? Given the enormous compute costs of o3, my speculation has been that search is explicit, but I’ve seen this post from Nathan Lambert for example that speculates (in the context of o1) that it’s possible for search to be entirely “baked-into” a single single stream roll-out (which would depend on significant long-context innovations): https://www.interconnects.ai/p/openais-o1-using-search-was-a-psyop If true this would be extremely interesting.