DeepSWE Blog

For what it's worth the source of the data[0] does have 3.7 flash with all 3 reasoning levels. 3.5/3.6 are in fact just the single points though (high reasoning). The datapoint in the announcement screenshot is either med or high, but they're pretty much exactly the same so can't say for certain. [0]: https://deepswe.datacurve.ai/

The only benchmarks that matters is your actual task. I've had models that benched poorly but performed great. And I constantly see models at near the top of AA, which are terrible. There doesn't necessarily seem to be a lot of overlap between benchmarks and real world usage. (Let alone common sense!) As far as they go, though, these harder benchmarks match my experience more closely: https://deepswe.datacurve.ai/ and https://cognition.ai/blog/frontier-code Where we see "top" models drop way down in score when given longer tasks. That being said, I've had a reasonably pleasant time with GLM-5.2 so far. (And have had an OK time with DeepSeek as well.) By the time I'm done testing all the Chinese models, they'll be obsolete :)

Why does an open weights model cost nearly the same as GPT5.6? $1.14 vs $1.23 on the cost index. What cost the most in API. Input, Cached Input, or Output. There you have your answer. Unfortunately, we have moved so much of the actual intelligence of models towards reasoning, what results in some models getting good scores, but this is because they are dumping a insane amount of reasoning tokens at the problem. So a mid priced model, with heavy reasoning output, cost the same as a expensive model, with medium reasoning output. Before the GPT Luna price drop of 80%, you actually had the same price if you used Luna High and Sol Low. With the difference that Sol Low was insane fast, and often way better code. https://deepswe.datacurve.ai/ Do not look at the top score but more what is on the horizontal axis as you go down. Sol Medium is frankly, was the best performance for dollar, until that Luna price drop. I will even argue that despite the higher price, Sol Medium is still way better despite Luna Max being cheaper. Or Opus Low, one of the better values also. What do you notice? Is that those models all have a high intelligence start point for their low setting. So that means they do not rely as much on output tokens aka thinking.

>equally "frontier" models A key point I want to make is that the notion of "frontier" is somewhat fictive in the sense that a model which dominates all others on a given eval is not guaranteed to be number one on another eval, even if both evals are ostensibly for the same task. For example, the best publicly-available model (i.e. excluding Claude Mythos and Fable) on DeepSWE[0] is gpt-5.5-xhigh at 67%, which is soundly better than claude-opus-4.8-max at 59%. I would say an 8pp gap on a benchmark is quite large. But on FrontierCode[1], claude-opus-4.8-xhigh is the best, at a score of 13.4% compared to gpt-5.5-medium at 6.3%. That's quite a significant reversal! Now, one might wish to claim that either DeepSWE or FrontierCode is poorly constructed and that the other is more accurate. But I think you'll find that the degree to which eval-design considerations in this case affect measurement is probably of no less magnitude than user-specific considerations affect measurement in general. [0] https://deepswe.datacurve.ai/ [1] https://cognition.com/blog/frontier-code

Ever since the insane discount with GPT-5.6 Luna, not much excites me anymore. I mean just look at the benchmarks, even though Gemini 3.7 Flash performs well on the DeepSWE 1.1, Luna (Max) still performs way better. I personally have stuck to Luna (Xhigh) because its been more than enough and does not bloat up the context window too fast with reasoning tokens. https://deepswe.datacurve.ai > Starting January 1, 2027, $1.50/1M input tokens and $7.50/1M output tokens will apply. Compare this to Luna which is at $0.2/1M input ($0.02 cached) and $1.2/1M output. https://developers.openai.com/api/docs/models/gpt-5.6-luna

You can see the Pareto Frontier well in DeepSWE's chart here - https://deepswe.datacurve.ai/ ChatGPT 5.6 Luna on the right (cheaper) cover most of the frontier, with a point for Deepseek flash, and higher performance overlapping heavily between 5.6 Sol and Fable. That DeepSeek point will probably move back towards Luna as deepseek announced a "significant" price increase coming to their API [1], which kind of demonstrates that beating the Pareto frontier is where the difficulty actually is). [1] https://www.bloomberg.com/news/articles/2026-08-06/deepseek-plans-sign…

Depends on the reasoning effort, see https://deepswe.datacurve.ai (add Luna via model selection drop down, if it's not shown by default)

According to https://deepswe.datacurve.ai/, Luna at Max at it's previous cost was comparable in both performance and cost to Sol at High. With an 80% reduction in cost that becomes a ridiculous outlier in efficiency.

https://deepswe.datacurve.ai/ https://artificialanalysis.ai/

eager to see how it benchmarks on https://deepswe.datacurve.ai/

OP link seems to be article about DeepSWE (with a link to it as well). For those interested in a direct link to DeepSWE: https://deepswe.datacurve.ai DeepSWE has info on their goals below the data on their homepage, or more details at https://deepswe.datacurve.ai/blog

It’s just inherently a difficult problem to solve, I think (just like with human individual contributors, where there famously also doesn’t exist one universal, automated process). “Good” benchmarks to gauge development skills at the moment seem to be: DeepSWE [0] by Datacurve FrontierCode [1] by Cognition And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore. But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again. [0] https://deepswe.datacurve.ai [1] https://cognition.com/blog/frontier-code [2] https://xcancel.com/i/article/2064210146558136827

DeepSWE is the one I generally trust: https://deepswe.datacurve.ai/

I've generally found DeepSWE[0] to be pretty true to reality. [0]: https://deepswe.datacurve.ai/

Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code. [0] https://deepswe.datacurve.ai/ [1] https://cognition.com/blog/frontier-code-1.1

You've got it backwards. Opus is the token/money burning one https://deepswe.datacurve.ai/ Gpt 5.5 uses a third of the opus 4.8 tokens for the same task and scores higher. Glm 5.2 was worse in quality but used half the tokens - 5.3 is not tested yet but will be higher.

DeepSWE “feels” like the right benchmark in comparison to Artificial Analysis indices and other coding benchmarks. And by their metrics, GPT-5.5 is still king in token efficiency, speed, and overall intelligence per dollar. https://deepswe.datacurve.ai/ Fable 5 is cool and all, but we have not yet seen GPT-5.6.

That remains to be seen. It's notable that Anthropic are still using SWEBench as a coding benchmark rather than the newer more difficult DeepSWE which shows them well behind GPT 5.5 https://deepswe.datacurve.ai/ Bear in mind that all the marketing efforts such as solving Erdos problem are the result of concerted RL training to impart those narrow capabilities, and how much of any benchmark results, or "early access" paid shill vibe reports, reflect improved performance for more general real-world use cases remains to be seen.

This benchmark draws a very different picture having GPT5.5 on the very top with 70% and DeepSeek at 8% https://deepswe.datacurve.ai

> Performance on benchmarks has practically leveled off Ehm, no? DeepSWE[1] for example shows that new models like gpt-5.5 continue to show big improvements compared to older models. > Also prices are going up. Prices for frontier intelligence have gone up, but prices for the same level of intelligence have gone way down (what you can get for pennies now was SOTA just a couple of years ago). The pareto frontier is still expanding. [1] https://deepswe.datacurve.ai/

Source: https://deepswe.datacurve.ai Just select the two models from the drop down.

DeepSWE has been making the rounds and at least seems to making an honest effort https://deepswe.datacurve.ai/

DeepSWE is closer to that https://deepswe.datacurve.ai/

Have you seen https://deepswe.datacurve.ai/blog? This is the closest to a vibe check i’ve felt even with the open models.

Out of curiosity, I examined the worst task: https://deepswe.datacurve.ai/data/trials/quill-shared-toolbar-focus__v… It seems like GPT here is failing due to an environment issue of connecting to chromium, even though its local unit tests passed. All the models failed 4/4 and checking Opus it ran into the same problem I checked some other tasks and they seemed legit, although in general the prompts seem somewhat contrived vs. what a typical user would ask their coding agent (such is the difficulty of benchmark construction)