Latent Space

Simon's blog is fragmented because it's, well, a blog. It would be hard to find a better source to "keep updated on things AI" though. He does do longer summary articles sometimes, but mostly he's keeping up with things in real time. The search and tagging systems on his blog work well, too. I suggest you stick his RSS feed in your feed reader, and follow along that way. Swyx also has a lot of stuff keeping up to date at https://www.latent.space/, including the Latent Space podcast, although tbh I haven't listened to more than one or two episodes.

oh wow. surprised to see this old piece here. it has certainly far outlasted my own expectations in terms of people quoting it and citing me on it on a weekly basis, even today. an updated chapter was rewritten for my book https://learninpublic.org/ (which yes is a few years late in the promised V2... IT WILL HAPPEN) questions welcome. it has continued to reap rewards for me and others who follow the practice long after publication (see: https://latent.space/, my current implementation), which is IMO the best validation I can have.

Thank you. This reminds me of a paragraph from the LatentSpace newsletter [0] > The excellent on device capabilities makes one wonder if these are the basis for the models that will be deployed in New Siri under the deal with Apple…. https://www.latent.space/p/ainews-gemma-4-the-best-small-multimodal

Pretty fantastic follow-up to https://www.latent.space/p/clippy-v-anton

I think it's worth carefully pulling apart _what_ DeepSeek is cheaper at. It's somewhat cheaper at inference (0.3 OOM), and about 1-1.5 OOM cheaper for training (Inference costs: https://www.latent.space/p/reasoning-price-war) It's also worth keeping in mind that depending on benchmark, these values change (and can shrink quite a bit) And it's also worth keeping in mind that the drastic drop in training cost(if reproducible) will mean that training is suddenly affordable for a much larger number of organizations. I'm not sure the impact on GPU demand will be as big as people assume.

I am extremely interested in your spam. Will you post it to https://www.latent.space/ ?

The term AI engineer is now pretty well recognised in the field (https://www.latent.space/p/ai-engineer), and is very much not the same as an AI researcher (which would be involved in training and building new models). I'd expect an AI engineer to be primarily a software developer, but with an excellent understanding of how to implement, use and evaluate LLMs in a production environment, including skills like evaluation and fine-tuning. This is not some dataset you can just bundle in software developer.

Do you want to train models from scratch, or do you want to build cool things on top of AI models? If the former, I suggest digging into things like the excellent Fast AI course: https://course.fast.ai/ If the latter, the (relatively new) keyword you are looking for is likely "AI Engineer" - https://www.latent.space/p/ai-engineer There's an argument that deep knowledge of how to train models isn't actually that useful when working with generative AI (LLMs etc) - knowing how to train or fine-tune a new model is less useful that developing knowledge of the other weird things you have to figure out about prompting, evals and using these models to build production-quality apps.

This was discussed on the Latent Space podcast a few days ago: https://www.latent.space/p/llm-training-2024 That was a good episode, worth listening to for hearing justifications behind some of these decisions.

This is a great recap of the present role. I have been following a conversation around the AI Engineer as a label (although I like Applied AI better) from the Latent Space podcast team so a fair amount of meme hypecycle, but also active and actionable. They are setting up a conference in October and you can see the discussion unfold around the blog post below on X-itter. https://www.latent.space/p/ai-engineer

Training smaller models can be really compute intensive (a 7B models should get trained on 1.4T tokens to follow the "LLaMA laws"). So that would be C = 6 * 1.4T * 7B = 58.8T FLOP-seconds. That's 1/5th the compute of GPT3 for example, but it's still a lot. We asked Quentin to do a similar post but for fine tuning math; that's still a very underexplored space. (Not to self plug too much, but this is exactly what last episode's with Tianqi Chen was about if you're interested :) https://www.latent.space/p/llms-everywhere#details

We already know ChatGPT is not good enough to do all that without being able to run code. That's why ChatGPT Code Interpreter is so powerful. It was even branded GPT4.5 in this podcast[0] which I think is apt. However, bringing that auto-coding loop into a persistent executable environment that you can revisit, adjust, and follow up on is a big win I think. [0] https://www.latent.space/p/code-interpreter

This is a bit more product focused, but I've found it useful: https://www.latent.space/ It's a newsletter/podcast.

I had access to GitHub Copilot as a student in early 2022 while learning Haskell and immediately realised that it would hinder my learning if I didn't turn it off and implicitly follow this understand, plan, execute, reflect loop. AI products like Cursor have the notion of an 'autonomy slider' [1] that can fortunately be turned all the way down (disable Cursor Tab) but relying on this discipline seems fickle when with the right agentic loops [2] and context engineering, thousands of lines of code can be churned out with minimal supervision. I've considered always working on two projects over a long timespan, one with no AI assistance, possibly in a separate IDE like Zed, and one in Vibe Kanban (my current daily driver) but this feels like an inefficient proxy to accelerating this four step learning loop with a tool like solveit. Since the solveit product isn't released and seemingly isn't competing with solutions, is there an opportunity to convey how AI product developers should be thinking about amplifying their users and keeping them in the learning loop? So far, I've seen Claude Code's Learning output style [3], and also ChatGPT's study mode but in these cases, the only product change is a prompt and solveit is more than that. [1] https://www.latent.space/i/166191505/part-a-autonomy-sliders [2] https://simonwillison.net/2025/Sep/30/designing-agentic-loops/ [3] https://docs.claude.com/en/docs/claude-code/output-styles#built-in-out…

Yes, there's a podcast with the post-training lead for L3 where he mentions this. Lemme try and find it. edit: found it. The money quote is here, but I really recommend the entire podcast since it's full of great tidbits and insights. > Thomas [00:33:44]: You mean between supervised fine-tuning like supervised fine-tuning annotation and preference annotation? Yeah. So 100% to RLHF. In fact, that's quite interesting. You start for Llama 2 with a pre-trained model and you have to have an instruction model to chat model. Otherwise, like the model is just like continue finishing sentences. So you need that to start RLHF. So we had to annotate like 10,000 examples. What did we do for Llama 3? You start with a new pre-trained model and then you want, before starting the RLHF, to have now a chat model, which is not too bad. The option one was, let's do human annotation again, like SFT stage. But in fact, by the principle I said before, the annotation would be actually worse than Llama 2. So what we did is that we generated all the data on the prompts with Llama 2 and we applied like basically the last round of Llama 2 we had to kick off and start Llama 3 post-training. So Llama 3 post-training doesn't have any like human written answers there basically, almost. It's just leveraging pure synthetic data from Llama 2. https://www.latent.space/p/llama-3

appreciate your constructive feedback! > i wonder if there are new ops solutions for the realtime apis popping up This is something we have spent quite some time on already, both on designs internally and talking to teams using Langfuse with realtime applications. IMO the usage patterns are still developing and the data capturing/visualization needs across teams is not aligned. What matters: (1) capture streams, (2) for non-text provide timestamped transcript/labels, (3) capture the difference between user-time and api-level-time (e.g. when catching up on a stream after having categorized the input first). We are excited to build support for this, if you or others have ideas or a wishlist, please add them to this thread: https://github.com/orgs/langfuse/discussions/4757 > retries for instructor like structured outputs mess up the traces, i wonder if they can be tracked and collapsible Great feedback. Being able to retroactively downrank llm calls to be `debug` level in order to collapse/hide them by default would be interesting. Added thread for this here: https://github.com/orgs/langfuse/discussions/4758 > chatgpt canvas like "drafting" workflows are on the rise (https://www.latent.space/p/inference-fast-and-slow) and again its noisy to see in a chat flow Can you share an example trace for this or open a thread on github? Would love to understand this in more detail as I have seen different trace-representations of it -- the best yet was a _git diff_ on a wrapper…

There was an interesting interview with David Luan about this recently. For context, he was a co-lead at Google Brain, early hire at OpenAI, and is now a founder at Adept: https://www.latent.space/p/adept The TL;DR on his take is that there are organizational and cultural issues that prevent Google from focusing their research efforts in the way that is necessary for what he calls "big swings," like training GPT-3. In regards to your second question, Google's reputation in ML is definitely not hype. Purely on the research side, Google has been behind some of the most important papers in modern ML, particularly around language model. The original Transformers paper, BERT, lots of work around neural machine translation, all of the work that DeepMind has done post-acquisition, and the list goes on. On the applied side, they also have some of the most successful/widely-adopted ML-powered products on the market (think RankBrain/anything involving a recommendation engine, Translate, Maps, a ton of functionality in Gmail, etc).

You might be disappointed.. It seems to have been weakly confirmed by Geohot that this sort of mixed system is already what GPT-4's 'secret sauce' is. [1] It's something I've also been speculating for months on. Ctrl+F for "220 billion in each". His phrasing, numbers, and details are suggestive of a leak unless he's just completely blowing smoke - and I don't think there's any real reason to think he is. It looks like they were pumping the model size up, started getting diminishing returns, and so turned to a mixed model of 8 expert systems meet LLMs - to try to keep squeezing out just a bit more juice. I think Geohot offered an incredibly insightful quote, "...whenever a company is secretive, it's because they're hiding something that's not that cool. And people have this wrong idea over and over again that they think they're hiding it because it's really cool." It also goes a long way to explain their recent, ultimately unsuccessful, gambit in congress. If they don't see any way of overcoming the diminishing returns, then they're like the guy who got a headstart in a race where distance = time ^ (1/2). [1] - https://www.latent.space/p/geohot#details

The latest Latent.Space episode https://www.latent.space/p/inference-eng

Yeah it was very disappointing that so few details were provided. One of the reasons I think it's going to be an open source project or effort that is going to show, sooner or later, how effective these things actually are. In their podcast interview, they mention that it's an Electron app that users download, and so they periodically create a new build. See section "Autonomous Merging Flow" here: https://www.latent.space/p/harness-eng

Maybe. Humans just can't thoroughly review the amounts of code AI can produce. But the answer is not YOLO and just ship blindly whatever AI created. The answer is to move the human review/judgment upstream. You review specs. You review the steps. You review the verification rules. You review the contract the code must fulfill. https://www.latent.space/p/reviews-dead

It's a retread of another (also baffling) "Why OpenAI Should Build Slack" post from a popular AI Substack. Just more empty grist for the AI adjacent content mill. "Slack sucks" doesn't let you draft off the current hype zeitgest, so we get "content" like this. https://www.latent.space/p/ainews-why-openai-should-build-slack

A similar argument to OpenAI: https://www.latent.space/p/ainews-why-openai-should-build-slack

Because HFT is about adversarial play with a lot of hidden information. Relevant article from two days ago https://www.latent.space/p/adversarial-reasoning

I've often found it helpful in search. Specifically, when the topic is well-documented, you can provide a clear description, but you're lacking the right words or terminology. Then it can help in finding the right question to ask, if not answering it. Recall when we used to laugh at people typing in literal questions into the Google search bar? Those are the exact types of queries that the LLM is equipped to answer. As for the "improvements" in GPT 5.1, seems to me like another case of pushing Clippy on people who want Anton. https://www.latent.space/p/clippy-v-anton

Antony, you’d be right to call me out on providing a source. So in case it’s helpful, this is the last place I recall the subject being discussed: RAG is Dead, Context Engineering is King https://www.latent.space/p/chroma

Latent Space. "Claude Code is also the most direct way to consume Sonnet for coding, rather than going through all the hidden prompting and optimization than the other products do. You will feel that right away, as the average spend per user is $6/day on Claude Code compared to $20/mo for Cursor, for example. Apparently, there are some engineers inside of Anthropic that have spent >$1,000 in one day!" Link: [https://www.latent.space/p/claude-code](https://www.latent.space/p/cla…

https://www.latent.space/p/o3-pro Have completed around a dozen chats with o3-pro so far. Can't say I'm impressed, output feels qualitatively very similar to regular o3. Tried feeding in loads of context as suggested in the article but generally feels like a miss.

Yes the authors openly acknowledge [0] to be inspired by LSP. [0]: https://www.latent.space/p/mcp

There's several famous examples of people expecting a problem to be AGI-hard — that is, solving it is equivalent to solving general intelligence[0] — only for someone to make an AI which can do that without being able to do everything else: • Fluent natural conversation • Translation • Go • Chess So perhaps we'll get an AI that makes the profession of "programmer" go the same way as the profession of "computer" before, after, or simultaneously with, an AI that does this to the profession of "therapist". [0] https://www.latent.space/p/agi-hard