METR

It's a solid question - and to some degree what https://programbench.com/ tries to measure. Some of the issues (off the top of my head): - Note - that my "product" was about 3,000 lines of code - so tiny. But https://metr.org/ should give you some insight into the complexity the models are capable of. - you have to be able to imagine the product. If I have the time, and energy, to imagine what I want - the model will build it. Here is an example of a much better programmer than I and something he wanted built - https://www.boatbomber.com/blog/claude-fable-5 - These are the first drafts. On average - any complex system needs about 10 years and at least 1000 active and enthusiastic about reporting users to really get robust code. Writing if via LLM doesn't (at least so far in my experience) help that much in reducing bugs if you were previously following any semblance of TDD. Lots of bugs in the code - the products you listed above have literally tens of millions of years of user experiences and bug reports that got them to where they are today. No silver bullet yet - just faster, less effort - and it enables non-technical people to create (still buggy) products.

> What exactly does the evaluation entail? I believe the US AISI has published less on their specific approach, but they’re largely expected to follow the general approach implemented by the UK AISI [1] and METR [2]. This is mostly focused on evaluating models on potentially dangerous capabilities. Some major areas of work include: - Misuse risks: For example, determining whether models have (dual-use) expert-level knowledge in biology and chemistry, or the capacity to substantially facilitate large scale cyber attacks. A good example of this is the work by Soice et al on bioweapon uplift [5] or Meta's work on CYBERSECEVAL [6], respectively. - Autonomy: Whether models are capable of agent-like behavior, like the kind that would be hard for humans to control. A big sub-area is Autonomous Replication and Adaptation (ARA), like the ability of the model to escape simulated environments and exfiltrate its own weights. A good example is METR's original set of evaluations on ARA capabilities [3]. - Safeguards: How vulnerable these models are to say, prompt injection attacks or jailbreaks, especially if they're also in principle capable of other dangerous capabilities (like the ones above). Good examples here are the UK AISI's work developing in-house attacks on frontier LLMs [4]. Labs like OAI, Anthropic and GDM already perform these internally as they're part of their respective responsible scaling policies, which determine which safety measures they should have implemented…

> Meanwhile, a study (late 2025) seems to report that although participating developers felt they completed tasks faster using AI, they where around 19% slower. METR reran the study early this year and, while they caveat it, this time they found a speedup, which is consistent with subjective estimates of productivity also having increased -- the simplest explanation is that subjective estimates exaggerate, but there's still a speedup with current models: https://metr.org/blog/2026-02-24-uplift-update/#wider-adoption-of-ai-h… (Nobody seems to cite the followup since it's not such a fun counterintuitive finding.)

https://metr.org/blog/2026-02-24-uplift-update/ This is probably the best one for coding? The two main findings are that developers didn't want to do tasks without AI (implication being that they would find it too tedious) and for the tasks that were measured, there was a speedup (and more of a speedup if you had more experience with AI tools) Unfortunately "productivity" is very hard to measure directly. I prefer looking at how much money companies are paying Applied AI companies (a lot) because in aggregate, that meant these companies justified ROI vs. OpenAI/Anthropic directly, and sufficiently enough that large enterprises are willing to go through the time and money to spend on a vendor. It's not foolproof but it dampens the effect of companies tokenmaxxing their Codex/Claude Code to look productive.

Yup, spot on. There's a capability-reliability gap that the industry does not like to talk about too much. It often feels like the AI industry is continually glossing over the fact that capability and reliability are fundamentally different qualities. We tend to use "accurate" and "reliable" interchangeably, but they describe different things. A model can ace a benchmark (capability/accuracy) and still be a liability in production (reliability). Just look at recent reactions to yet another release from METR showing improved capabilities. But the less talked about part is how their measure is for an 80% success rate (and the even lesser talked about secondary measure they have at 50% success rate). https://metr.org/ I implement AI systems for enterprises and I don't know any that would ever be okay with 80% reliability.

> The current tech is a sigmoid What makes you say that? I'm only asking because the data I've seen looks pretty cleanly exponential still, e.g. https://metr.org.

> I'm significantly faster while guiding an LLM using this methodology. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud… >When developers are allowed to use AI tools, they take 19% longer to complete issues—a significant slowdown that goes against developer beliefs and expert forecasts. This gap between perception and reality is striking: developers expected AI to speed them up by 24%, and even after experiencing the slowdown, they still believed AI had sped them up by 20%. If we're being honest with ourselves, it's not making devs work faster. It at best frees their time up so they feel more productive.

> we now think it's +9% - +38% If you are referring to the following quote [0], you are off by a sign: > we now estimate a speedup of -18% with a confidence interval between -38% and +9%. [0] https://metr.org/blog/2026-02-24-uplift-update/

This March 2026 blog post is citing a 2025 study based on Sonnet 3.5 and 3.7 usage. Given that organization who ran the study [1] has a terrifying exponential as their landing page, I think they'd prefer that it's results are interpreted as a snapshot of something moving rather than a constant. [1] - https://metr.org/

There's been a huge amount of improvement in coding agent effectiveness since they ran that experiment. In a more recent follow up experiment, METR found 20% speed up from AI assistance and says they believe that is likely an underestimate of the impact. https://metr.org/blog/2026-02-24-uplift-update/ They are working on making a new measurement approach that will be more accurate.

Classic https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud…

Yes a strange comment. Opus 4.5 is significantly better than before and Opus 4.6 is even better. Same with the 5.2 and 5.3 Codex models. If anything, the pace has increased. This may be one of the most important graphs to keep an eye on: https://metr.org/ and it tracks well to my anecdotal experience. You can see the industry did hit a bit of a wall in 2024 where the improvements drop below the log trend. However, in 2025 the industry is significantly _above_ the trend line.

How are we still citing the (excellent) METR study in support of conclusions about productivity that its authors rightly insist[0] it does not support? My paraphrase of their caveats: - experts on their own open source proj are not representative of most software dev - measuring time undervalues trading time for effort - tools are noticeably better than they were a year ago when the study was conducted - it really does take months of use to get the hang of it (or did then, less so now) Before you respond to these points, please look at the full study’s treatment of the caveats! It’s fantastic, and it’s clear almost no one citing the study actually read it. [0]: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud…

Note that self-reporting productivity gains is a completely unreliable and unscientific metric. One study[1], small in scope but a noteworthy data point, found that over the course of the study that LLMs reduced productivity by ~20% but even after the fact the participants felt that on average their productivity had increased by ~20%. This study is surely not the end-all be-all and you could find ways to criticise it or say it doesn't apply or they were doing it wrong or whatever reason you think the developers should have had increased productivity, but the point is that people cannot accurately judge their own productivity by vibes alone. [1] https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud…

I don't get it? Yes you should require a valid reason before believing something The only objective measures I've seen people attempt to take have at best shown no productivity loss: https://substack.com/home/post/p-172538377 https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud… This matches my own experience using agents, although I'm actually secretly optimistic about learning to use it well

Developers who use AI think they're quicker and better, but they're actually slower and worse. Not surprised to hear it's similar in other fields. AI in general, just like drugs, makes people feel good but without any actual substance behind the feeling. (If feeling without substance is all you need, then it's okay to use AI. AI Dungeon, for example, was pretty cool. Or slide backgrounds that would have otherwise been solid colours because they're worth $0 and you wouldn't have paid a designer.) This first chart should be absolutely damning: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud…

> The only bottleneck is mdoel (sic) speed and quality. But with billions of dollars pouring into generative AI every year, we will see instant voice-to-code capabilities + bug-free quality in 2-5 years. Anytime I think the AI bubble can't go any higher I'm reminded of the fact there are people who genuinely believe this. These are the people boardrooms are listening to despite the evidence for actual developer productivity savings being dubious at best: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud…

>The actual benchmark improvements are marginal at best GPT-5 demonstrates exponential growth in task completion times: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long…

https://aicodinghorrors.com/ - the database deletion one was how I found out this site exists. https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud… - somewhat controversial because measuring different tasks. But nobody can measure dev tasks anyway. https://venturebeat.com/ai/stack-overflow-data-reveals-the-hidden-prod… - Saw this very recently, haven't dug in to the sources. https://www.wheresyoured.at/ Talks a LOT about the AI hype bubble in general. The landscape here in the US sucks too. Layoffs and offshoring are still going strong. Everything new is some stupid AI startup. I kinda thought I was done too until I got laid off and started working on my own projects to stay sharp. Realized I still love making stuff. Just hate the corporate games.

Check out section AI increasing issue scope (C.2.3) in the paper -- https://metr.org/Early_2025_AI_Experienced_OS_Devs_Study.pdf We speak (the best we can) to changes in amount of code -- I'll note that this metric is quite messy and hard to reason about!

One thing law explicitly says is full shutdown capability. So it should be tested whether it can autonomously hack computers on the internet and propagate itself. In fact Anthropic tested this. See https://metr.org/ for more.

Yes, but their newer write-up discusses that (and shows that the self-reported numbers have gone up radically, in a way that suggests that even if there is some inflation, the numbers are almost certainly positive if you deflate). They also have an update -- linked from the original study! -- explaining that it's out of date and no longer reliable, and explaining why they had to cancel a follow-up study because it was understating productivity gains (but also was showing wins for the people who carried over from their previous study): https://metr.org/blog/2026-02-24-uplift-update/ The authors of this paper decided to ignore all of METR's follow-up data and discussion, and to report only the ancient number from early 2025 (a time when Windsurf was state of the art). And then, rather than apologizing for it, and caveating it as a number not to be taken seriously, they described it as a study done "recently." That's either shockingly dishonest or incredibly out-of-touch.

I agree that not having to read the code is the future but I struggle to get there because the AI, like humans, can make mistakes (not follow instructions). For example, I might ask the AI to render a scene using a physical model, and we might come to an agreement at the plan stage, but then it will go off and cut some corner in implementation. So the question is how to reliably verify compliance. If you merely tell another AI to check it properly implemented the plan it too can miss bugs, so you need to put all your effort in making the acceptance tests foolproof, and this is easier said than done. You need a defense in depth approach, using the type system, unit tests, formal verification (if you can), linting, etc. It is an open question how to optimally allocate your correctness efforts for a given budget but, as models get smarter, it is clear to see that the amount of human diligence needed to achieve likely correctness for a given task (say, in LOC) is going to tend to zero (https://metr.org/time-horizons/)

> But! Hold my beer… in February 2026 METR effectively walked it back : their follow-up estimates flipped to a speedup (with error bars wide enough to ride a Moto Guzzi, with panniers, through!), and they abandoned the study design entirely - because developers now refuse to work without AI, and can’t reliably self-report time on agentic work. Their latest position: AI probably speeds developers up in 2026, and we can no longer cleanly measure by how much. This may be true, but they followed in May with this [0]: > Importantly, survey results are not necessarily grounded in reality. There are reasons to be skeptical of people’s responses to counterfactual questions such as about AI’s effect on productivity — for instance, our study in early 2025 found that people overestimated AI’s effect on their time spent on tasks by 40 percentage points on average. [0] https://metr.org/blog/2026-05-11-ai-usage-survey/#productivity-gains-o…

>I see the same level of quality and excellence from people who use it like it’s another tool in their toolbelt It's certainly currently an edge if you know what to tell the mystery box of magic in precise terms. I don't think this distinction is going to endure though - every level of "it can't do it" has fallen and generally faster & more decisively than predicted. We started with it printing hello world, to autocomplete where you still needed to be able to know what the line should do, to autocompleting functions, to writing entire units, to working out architecture tradeoffs, to doing research planning architecture execution and testing all autonomously. That trajectory plus people retreating to nebulous "I'm adding taste" tells me this is going to sail straight past "tool in toolbelt" territory at Mach 10. Everyone has their own perspective but to me "show me the receipt" at a specific point in time is a completely wrong lens for a tech that shows clear signs of exponential improvement (i.e. https://metr.org/ ).

Probably the biggest thing that serious predictions are relying on is the METR graph: https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long… It shows a remarkably consistent curve for AI completing increasingly difficult coding tasks over time. In fact, the curve is exponential, where the X axis is time and the Y axis is task difficulty as measured by how long a human would take to perform the task. The current value for 80% success rate is only 45 minutes, but if it continues to follow the exponential curve, it will only take 3 years and change to get to a full 40 hour human work week's worth of work. The 50% success rate graph is also interesting, as it's similarly exponential and is currently at 6 hours. Of course, progress could fall off as LLMs hit various scaling limits or as the nature of the difficulty changes. But I for one predicted that progress would fall off before, and was wrong. (And there is nothing saying that progress can't speed up.) On the other hand, I do find it a little suspicious that so many eggs are in the one basket of METR, prediction-wise.

It's true that some people will just continually move the goalposts because they are invested in their beliefs. But that doesn't mean that the skepticism around certain claims aren't relevant. Nobody serious is disputing that LLM's can generate working code. They dispute claims like "Agentic workflows will replace software developers in the short to medium term", or "Agentic workflows lead to 2-100x improvements in productivity across the board". This is what people are looking for in terms of evidence and there just isn't any. Thus far, we do have evidence that AI (at least in OSS) produces a 19% decrease in productivity [0]. We also have evidence that it harms our cognitive abilities [1]. Anecdotally, I have found myself lazily reaching for LLM assistance when encountering a difficult problem instead of thinking deeply about the problem. Anecdotally I also struggle to be more productive using AI-centric agents workflows in areas of expertise. We want evidence that "vibe engineering" is actually more productive across the entire lifespan of a software project. We want evidence that it produces better outcomes. Nobody has yet shown that. It's just people claiming that because they vibe coded some trivial project, all of software development can benefit from this approach. Recently a principle engineer at Google claimed that Claude Code wrote their team's entire year's worth of work in a single afternoon. They later walked that claim back, but most do not. I'm more than…

I see it brought up almost every week! It's a firm favorite of the "LLMs don't actually help write code" contingent, probably because there are very few other credible studies they can point to in support of their position. Most people who cite it clearly didn't read as far as the table where METR themselves say: > We do not provide evidence that: > 1) AI systems do not currently speed up many or most software developers. Clarification: We do not claim that our developers or repositories represent a majority or plurality of software development work > 2) AI systems do not speed up individuals or groups in domains other than software development. Clarification: We only study software development > 3) AI systems in the near future will not speed up developers in our exact setting. Clarification: Progress is difficult to predict, and there has been substantial AI progress over the past five years [3] > 4) There are not ways of using existing AI systems more effectively to achieve positive speedup in our exact setting. Clarification: Cursor does not sample many tokens from LLMs, it may not use optimal prompting/scaffolding, and domain/repository-specific training/finetuning/few-shot learning could yield positive speedup https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud…

Well, for example, consider this recent study that claimed developers using AI tools take 19% longer to finish tasks [1]. This was their methodology: > we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors that would normally be part of their regular work. Then, we randomly assign each issue to either allow or disallow use of AI while working on the issue. Now consider the question of whether you expect this research to generalize. Do you expect that if you / your friends / your coworkers started using AI tools (or stopped using AI tools) that the difference in productivity would also be 19%? Of course not! They didn't look at enough people or contexts to get two sig figs of precision on that average, nor enough to expect the conclusion to generalize. Plus the AI tools are constantly changing, so even if the study was nailing the average productivity change it would be wrong a few months later. Plus the time period wasn't long enough for the people to build expertise, and "if I spend time getting good at this will it be worth it" is probably the real question we want answered. The study is so weak that I don't even feel compelled to trust the sign of their result to be predictive. And I would be saying the same thing if it reported…

"it's better at coding" is not useful information, sorry. i'd love to hear tangible ways it's actually better. does it still succumb to coding itself in circles, taking multiple dependencies to accomplish the same task, applying inconsistent, outdated, or non-idiomatic patterns for your codebase? has compliance with claude.md files and the like actually improved? what is the round trip time like on these improvements - do you have to have a long conversation to arrive at a simple result? does it still talk itself into loops where it keeps solving and unsolving the same problems? when you ask it to work through a complex refactor, does it still just randomly give up somewhere in the middle and decide there's nothing left to do? does it still sometimes attempt to run processes that aren't self-terminating to monitor their output and hang for upwards of ten minutes? my experience with claude and its ilk are that they are insanely impressive in greenfield projects and collapse in legacy codebases quickly. they can be a force multiplier in the hands of someone who actually knows what they're doing, i think, but the evidence of that even is pretty shaky: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-stud… the pitch that "if i describe the task perfectly in absolute detail it will accomplish it correctly 80% of the time" doesn't appeal to me as a particularly compelling justification for the level of investment we're seeing. actually writing the code is the…