Epoch AI

If one wants to follow AI development mostly in the sense of LLMs and associated frontier models, that's an excellent list with over half of the names familiar, to whom I have converged independently. I have a list in X for AI; it's the best source of information overall on the subject, although some podcasts or RSS feeds directly from the long-form writers would be quite close. (If one is a researcher themselves, then of course it's a must to follow the paper feeds, not commentary or secondary references.) I'd add https://epoch.ai to the list, on podcasts at least Dwarkesh Patel; on blogs Peter Wildeford (a superforecaster), @omarsar0 aka elvis from DAIR in X, also many researchers directly although some of them like roon or @tszzl are more entertaining than informative. The point about polluted information environment resonates on me; in general but especially with AI. You get a very incomplete and strange understanding by following something like NYT who seem to concentrate more on politics than technology itself. Of course there are adjacent areas of ML or AI where the sources would be completely different, say protein or genomics models, or weather models, or research on diffusion, image generation etc. The field is nowadays so large and active that it's hard to grasp everything that is happening on the surface level. Do you _have_ to follow? Of course not, people over here are just typically curious and willing to follow groundbreaking technological advancements.…

Good thing we have reams of data on this, holding performance constant the cost goes down 10-40x per year: https://epoch.ai (like the first box) Also, frontier token prices have remained roughly constant: 3.5 sonnet: $3/$15 3.7 sonnet: $3/$15 Opus 4: $15/$75 (opus tier) opus 4.1: same Opus 4.5: $5/$25 Opus 4.6 (same) 4.7 (same) 4.8 (Same) Fable: $10/$50 So Fable is cheaper than Opus 4 was at launch. One thing that has increased quite significantly? Spending and adoption.

Are we plotting against cost? How is the capability advancement vs dollars paid for development? By my read of the (very sparse) data, we're getting linear improvements in capability for super-linear increases in costs. [1] Indicates that by 2027 models will cost $1 billon to train. Dario estimates that model runs will cost $10 billion in 2026 [2]. That to me indicates costs are potentially growing faster than capability. Maybe by quite a bit. If the value prop of LLMs doesn't prove out, that won't last. I'm of the opinion there is no data that shows actual economic value being delivered by models. The best data shows that LLM use might be destroying value [3]. [1] https://epoch.ai/publications/how-much-does-it-cost-to-train-frontier-… [2] https://lexfridman.com/dario-amodei-transcript/ [3] https://unessays.substack.com/p/talk-is-cheap

Mathematician Ken Ono (https://en.wikipedia.org/wiki/Ken_Ono) gives a well nuanced viewpoint on AI in Mathematics (and more) - https://www.youtube.com/watch?v=jGZOi-7haCw He states that he struggled to come up with problems which would be challenging for AI to solve (at the below site) and thus forced to accept that mathematicians have to rethink their profession. FrontierMath: Benchmarking AI against advanced mathematical research by Epoch AI - https://epoch.ai/frontiermath As a follow up to the above, see "First Proof: Mathematicians Putting AI to the Test" featuring eminent mathematicians - https://www.youtube.com/watch?v=AaICCTpkI7Q

The technology already exists now for the next 10x drop between everyone adopting DeepSeek's MLA, MoE (mostly already done), Medusa (a better version of Google's speculative decoding), Kimi's Attn Residuals, and Mimo's Sliding Window Attention, and (possibly) Microsoft's 1.58b (this may be a nothing burger). Historic trends -> every 18 months, performance for the same level of quality has gone down 90%. https://www.reddit.com/r/LocalLLaMA/comments/1gpr2p4/llms_co... See Chart 13 here: https://www.rdworldonline.com/ais-great-compression-20-chart... See here: https://epoch.ai/data-insights/llm-inference-price-trends Additionally, GRAM (from ~10 days ago) is likely to be a 5-10x on its own.

And yet: https://epoch.ai/benchmarks/eci GPT-4 was a world-shaking release. The best "AIs" (LLMs) of 2026 are only marginally better. The gap between 2023 and 2026 isn't nearly so large as you imply, especially insofar as "general" intelligence (not coding specifically) is concerned.

That would be insightful if the cost of inference weren’t declining at roughly 90% per year. Source: https://epoch.ai/data-insights/llm-inference-price-trends

https://epoch.ai/benchmarks/simplebench Just do it. I use a service where I have access to all SOTA models and many open sourced models, so I change models within chats, using MCPs eg start a chat with opus making a search with perplexity and grok deepsearch MCPs and google search, next query is with gpt 5 thinking Xhigh, next one with gemini 3 pro, all in the same conversation. It's fantastic! I can't imagine what it would be like again to be locked into using one (or two) companies. I have nothing to do with the guys who run it (the hosts from the podcast This day in AI, though if you're interested have a look in the simtheory.ai discord. I don't know how people use one service can manage...

The source here is "CAISI Evaluation of DeepSeek V4 Pro" [1]; the US NIST ran their own benchmarks (including several internal ones) and reported the following table: | Domain | Benchmark | Model (reasoning level) | | | | |--:-------------------|------------------------|-------------------------|-----------------------------|--------------------------|-----------------------| | | | OpenAI GPT-5.5 (xhigh) | OpenAI GPT-5.4 mini (xhigh) | Anthropic Opus 4.6 (max) | DeepSeek V4 Pro (max) | | Cyber | CTF-Archive-Diamond | **71%** | 32% | 46% | 32% | | Software Engineering | SWE-Bench Verified* | **81%** | 73% | 79% | 74% | | | PortBench | **78%** | 41% | 60% | 44% | | Natural Sciences | FrontierScience | **79%** | 74% | 72% | 74% | | | GPQA-Diamond | **96%** | 87% | 91% | 90% | | Abstract Reasoning | ARC-AGI-2…

Wait hold on, let's put some numbers on this. Please correct my calculations if I'm wrong. 1. The human brain draws 12 - 20 watts [1, 2]. So, taking the lower end, a task taking one hour of our time costs 12 Wh. 2. An average ChatGPT query is between 0.34 Wh - 3 Wh. A long input query (10K tokens) can go up to 10 Wh. [3] I get the best results by carefully curating the context to be very tight, so optimal usage would be in the average range. 3. I have had cases where a single prompt has saved me at least an hour of work (e.g. https://news.ycombinator.com/item?id=44892576). Let's be pessimistic and say it takes 3 prompts at 3 Wh (9 Wh) and 10 minutes (2 Wh) of my time prompting and reviewing to complete a task. That is 11 Wh for the same task, which still beats out the human brain unassisted! And that's leaving aside the recent case where I vibecoded and deployed a fully-tested endpoint on a cloud platform I had no prior experience in, over the course of 2 - 3 hours. I estimate it would have taken me a whole day just to catch up on the documentation and another 2 days tinkering with the tools, commands and code. That's at least an 8x power savings assuming an 8-hour workday!! 4. But let's talk data instead of anecdotes. If you do a wide search, there is a ton of empirical evidence that improves programmer productivity by 5 - 30% (with a lot of nuance). I've cited some here: https://news.ycombinator.com/item?id=45379452 -- there is no measure of the amount of prompt usage…

A million people like you can do a million times more things that require intelligence†, and would consume a million times more Taco Bell double deckers. Ergo, in that limited sense, more energy == more intelligence. "Over time" clearly means that he's talking about the far future, not the next few years when they're cleaning up the low-hanging fruit. Rather than "absurd ... fantastic and impossible" I would describe the Dyson-sphere outcome as inevitable, unless humanity goes extinct within a few centuries. Maybe you thought he meant next March? In https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use, Josh You, Alex Erben, and Ege Erdil estimate that a typical GPT-4o query uses 0.3 watt hours, based on the estimates that it's a 400-billion-parameter MoE model in which ¼ of the parameters are activated for a given query, so each token requires 200 gigaflops (100 billion multiplies and 100 billion adds I guess), a typical query produces 500 output tokens (about a page of text), and they're running on 1-petaflop/s H100 GPUs, with the overall cluster consuming at peak 1500 watts per GPU, with a utilization rate of 10%, and that the cluster on average consumes 70% of peak power. This works out to 1.05 kilojoules, or about 0.25 kilocalories, the amount of food energy in one gram of carbohydrates. So, a 320-kcal Double DeckerⓇ Taco™ works out to 1280 GPT-4o queries answered, and a standard 2000kcal/day diet works out to 8000 GPT-4o queries per day, if we…

I find this unconvincing. The actual discussion of LLM generation is very lacking. The original link [1] cites a discussion of the cost per query of GPT-4o at 0.3whr [2]. When you read the document [2] itself you see 0.3whr is a lower bound & 40whr is the upper bound. The paper [2] is actually pretty solid, I recommend it. It uses the public metrics from other LLM APIs to derive a likely distribution of the context size of the average query for GPT-4o which is a reasonable approach given that data isn't public. Then factoring in GPU power per FLOP, average utilization during, and cloud/renting overhead. It admits this likely has non-trivial error bars, concluding the average is between 1-4whr per query. This is disappointing to me as the original link [1] attempts to bring in this source [2] to disprove the 3whr "myth" created by another paper [3], yet this 3whr figure lies directly in the error bars their new source [2] arrives at. Links: 1. https://simonwillison.net/2025/Apr/29/chatgpt-is-not-bad-for-the-envir… 2. https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use 3. https://www.sciencedirect.com/science/article/pii/S2542435123003653?dg… Edit: whr not w/hr

It's hard for me to imagine how you could define "killer product" to exclude something that takes you from $9B to $65B ARR in 8 months. https://epoch.ai/data/ai-companies?view=graph&tab=revenue

You can look at the benchmark and the GPT-5 failures like answering "April 22, 2019" instead of the correct "Oct 23, 2018" for the question: What day, month, and year was Carrie Underwood's album "Cry Pretty" certified Gold by the RIAA? If your idea of the smartest person in the world is the guy who always wins tuesday night pub trivia, this blog post is for you. It also gets it's foundational factual claim wrong (as seen via epoch.ai). Very on brand. https://epoch.ai/benchmarks/simple-qa-verified?view=graph&tab=rele… https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?log_file=htt…

https://epoch.ai/data-insights/google-custom-tpus-ai-compute appears to be that link.

There’s SimpleQA Verified https://epoch.ai/benchmarks/simple-qa-verified

> The fact that tokens aren't free will limit the number of unserious requests. LLM costs are already pretty cheap (regardless of whether someone believes they'll continue falling). For a recent example, see a relatively powerful LLM like DeepSeek Flash, where you can get a million tokens for $0.20. And if someone is OK with their task being batched (instead of executing it right now), that can lower prices too. Inference price trends in previous years: https://epoch.ai/data-insights/llm-inference-price-trends

I think they dumb down their public models to be only slightly better than the competition. And the real competition is China, so the current state of the Chinese models would define the baseline. I think one evidence is that the US has more than 5x the compute of China. With that difference in training speed, it should be impossible for Chinese models to close the gap that easily. It's also very unlikely that they sell the same public models to their private customers (military etc). We also know they talk about "unpublished internal models" for things like the last HuggingFace hacking incident. So it's not a bad theory. https://epoch.ai/publications/trends-in-ai-supercomputers

>with their daily active users count, OpenAI and Anthropic would have been profitable for years if that was true. No they wouldn't have. Low cost isn't no cost, R$D is expensive and there are a class of tokenmaxxing users well outside the median (e.g Agentic Coding). > I don't know where you get that from https://cloud.google.com/blog/products/infrastructure/measuring-the-en… https://epoch.ai/gradient-updates/how-much-energy-does-chatgpt-use

> No they wouldn't have. Low cost isn't no cost, R$D is expensive I'm not talking about R&D. > there are a class of tokenmaxxing users well outside the median (e.g Agentic Coding). It wasn't a thing before last year and OpenAI wasn't making profit before that either. > https://cloud.google.com/blog/products/infrastructure/measur... > https://epoch.ai/gradient-updates/how-much-energy-does-chatg... None of these support your claim. Again, when you're here saying that the CEOs of the two biggest AI companies are basically idiots, nobody will take you seriously.

In research we are still seeing massive jumps. Subjects that LLMs were completely useless for half a year ago are now definitely in scope. And there are benchmarks that cleanly separate the SOTA models: https://epoch.ai/MirrorCode Saturation of benchmarks is a property of benchmarks just as much as of the models.

Yep! I found this interesting article after banging my head against a wall for a long time: https://epoch.ai/gradient-updates/gpqa-diamond-whats-left

"Anthropic and OpenAI generate a lot of revenue with relatively few employees – an estimated $9M and $5.5M in revenue per employee (RPE), respectively. If either company were to go public, it would have a higher RPE than any public tech company on Forbes’ Global 2000 list." https://epoch.ai/data-insights/revenue-per-employee-ai-companies

> The only argument they have here is that they use GB300 GPU's which for some reason should not be available to chinese citizens Note that Chinese companies are free to rent from GB300 clouds internationally. There are large datacenter hubs in Singapore and Malaysia serving chinese and other customers. Though there is also reported [1] significant smuggling of Nvidia chips into China as well. [1] https://epoch.ai/publications/chip-smuggling

Epoch AI estimates that LLM costs are falling by over 10x per year at fixed intelligence https://epoch.ai/data-insights/llm-inference-price-trends, so I expect prices to fall at this level of quality. I also think there's a good division of labor here where you can use the smartest model to write the script and prompts, and then use a cheaper model for all the glue work and minor revisions that trigger the tool calls.

I would say it's not just the VCs but the various other entities that will be left holding the bag of debt if the AI-fueled datacenter construction boom/bubble pops. For a list of large and well known projects and their scales: https://epoch.ai/data/ai-data-centers

Nobody is "missing" anything. There are migrations to the cloud, but AI growth is on top of that. AI computing capacity is doubling every seven months. https://epoch.ai/data-insights/ai-chip-production.

> Yes. And since LLMs can not improve knowledge 1. GPT proved Erdős Unit Distance Conjecture entirely on its own 2. GPT-5.4 Pro Solves Erdős Problem #1196 (April 2026) In fact there's a whole benchmark that's measuring this: https://epoch.ai/frontiermath

I'm not sure what you mean, we have already scaled up AI exponentially. The amount of AI compute in the world has been doubling every 7 to 8 months [1], so it is already exponential despite not being able to do human or super human research. The % of all AI compute going towards academic style knowledge research is quite low as well. So it stands to reason AI Compute used to do research would in fact scale up exponentially if we did figure out how use weights in a datacenter to do frontier research. That doesn't mean were going to go right to Dyson spheres so that every possible molecule is going towards scaling. [1] https://epoch.ai/trends

I obviously don't know the full economics of the Chinese-hosted models, but estimates[1] put the cost of hardware (servers + networking) at 70-80% of the total cost. Those things aren't meaningfully cheaper in China, so serving DeepSeek at 1/3 the cost of the cheapest US provider doesn't really compute unless it's heavily subsidized or we believe that Chinese engineers are just that much better at optimization. Edge models, yes, they can be convenient to run batch jobs locally. I still would argue there's no economic benefit over paying for models. Haiku has a bad price/perf but others in that class are significantly cheaper in hosted APIs. Doesn't matter what I think, the reality is that the majority of enterprises (where the real $ comes from) will not consider sending their data to China. 1. https://epoch.ai/data-insights/ai-datacenter-cost-breakdown