Cognition Blog

The design of the website looks very close to: https://cognition.com

The only benchmarks that matters is your actual task. I've had models that benched poorly but performed great. And I constantly see models at near the top of AA, which are terrible. There doesn't necessarily seem to be a lot of overlap between benchmarks and real world usage. (Let alone common sense!) As far as they go, though, these harder benchmarks match my experience more closely: https://deepswe.datacurve.ai/ and https://cognition.ai/blog/frontier-code Where we see "top" models drop way down in score when given longer tasks. That being said, I've had a reasonably pleasant time with GLM-5.2 so far. (And have had an OK time with DeepSeek as well.) By the time I'm done testing all the Chinese models, they'll be obsolete :)

>equally "frontier" models A key point I want to make is that the notion of "frontier" is somewhat fictive in the sense that a model which dominates all others on a given eval is not guaranteed to be number one on another eval, even if both evals are ostensibly for the same task. For example, the best publicly-available model (i.e. excluding Claude Mythos and Fable) on DeepSWE[0] is gpt-5.5-xhigh at 67%, which is soundly better than claude-opus-4.8-max at 59%. I would say an 8pp gap on a benchmark is quite large. But on FrontierCode[1], claude-opus-4.8-xhigh is the best, at a score of 13.4% compared to gpt-5.5-medium at 6.3%. That's quite a significant reversal! Now, one might wish to claim that either DeepSWE or FrontierCode is poorly constructed and that the other is more accurate. But I think you'll find that the degree to which eval-design considerations in this case affect measurement is probably of no less magnitude than user-specific considerations affect measurement in general. [0] https://deepswe.datacurve.ai/ [1] https://cognition.com/blog/frontier-code

On the contrary; it seems Luna (max) is 2x as capable at 1/5 the cost: https://cognition.com/frontiercode

They compare it to 5.6 Terra, however https://cognition.com/frontiercode puts Terra at about 1/2 the price Also have to compare to the recent Grok 4.6 release, which appears to straight up be better AND cheaper Hard to understand why anyone would choose 3.7 Flash under these conditions.. is Deepmind still a frontier lab?

https://cognition.com/frontiercode Quality vs cost - medium is the sweet (perhaps better too!) spot.

It’s just inherently a difficult problem to solve, I think (just like with human individual contributors, where there famously also doesn’t exist one universal, automated process). “Good” benchmarks to gauge development skills at the moment seem to be: DeepSWE [0] by Datacurve FrontierCode [1] by Cognition And then there’s TerminalBench, which I’m certain has been saturated in post-training to no end, so I wouldn’t think of it as a gold standard anymore. But yeah, in general, it’s not going to get easier knowing which benchmarks are actually measuring “frontier” capability, and which are just getting results inflated by way of time/token budget [2], ever again. [0] https://deepswe.datacurve.ai [1] https://cognition.com/blog/frontier-code [2] https://xcancel.com/i/article/2064210146558136827

https://cognition.ai/blog/frontier-code (disclaimer - was on the team - but also we covered swebench pro/deepswe issues in here as well.)

Either DeepSWE [0] or FrontierCode [1], depending on personal goals and requirements. The later is more interesting for me personally, due to the design of the benchmark heavily grading "mergability", i.e. how the provided output is to review and whether a serious developer can easily parse it and'd be willing to merge the result. In my mind and with my private evals, for quite some time I've held firm that a model can have a higher ceiling but that has limited value if I do not feel truly confident in signing off on the code. [0] https://deepswe.datacurve.ai/ [1] https://cognition.com/blog/frontier-code-1.1

This is the goal behind Devin Fusion, pretty good results so far I think. https://cognition.com/blog/devin-fusion

On the new FrontierCode [1] benchmark (ie graded from an OSS maintainer's perspective of "would I merge this code?") - Opus 4.7 xhigh: 5.2% - Opus 4.8 xhigh: 13.4% - Fable 5 xhigh: 29.3% Seems like a huge jump. [1] https://cognition.ai/blog/frontier-code

Also on the HN homepage: https://cognition.ai/blog/frontier-code Not quite ASCII but pretty close!

COBOL migration is one of Devin's advertised capabilities: https://docs.devin.ai/use-cases/examples/cobol-modernization https://cognition.ai/blog/infosys-cognition

Ah. Thx. Blogpost for others: https://cognition.ai/blog/swe-1-5 Takeaway is that this is sonnet-ish model at 10x the speed.

I think both Cursor and Cognition and going in the same direction of SWE-grep[0]. SWE-grep was able to hit ~700tokens/s and Cursor ~300token/s, hard to compare the precision/recall and cost effectiveness though, considering SWE-grep also adopted a "hack" of running it on Cerebras. I'm trying to kickstart a RL-based code search project called "op-grep" here[1], still pretty early, but looking for collaborators! [0]: https://cognition.ai/blog/swe-grep [1]: https://github.com/aperoc/op-grep

This link redirects to https://cognition.ai/blog/swe-grep now?

Yes, but at the same time having the 1 million context enabled is nice because the model is aware that they have more context left and actually perform better. [0] [0] https://cognition.ai/blog/devin-sonnet-4-5-lessons-and-challenges#:~:t…

It is, but there's a direct tension here between security and capabilities. It's hard to do useful things with private data without opening up prompt injection holes. And there's a huge demand for this kind of product. Agents also typically work better when you combine all the relevant context as much as possible rather than splitting out and isolating context. See: https://cognition.ai/blog/dont-build-multi-agents — but this is at odds with isolating agents that read untrusted input.

It might be clearer to link the original blog post (https://cognition.ai/blog/kevin-32b) rather than the Hugging Face model card. The charts and formulas are attractive, but the CUDA snippet feels quite elementary.

> Also... telling that they choose to benchmark against CodeForces rather than SWE-bench. They also worked with Devin to benchmark it on Devin's internal benchmarks, where it's twice as good as GPT-4o: https://x.com/cognition_labs/status/1834292718174077014 https://www.cognition.ai/blog/evaluating-coding-agents

Cognition (Devin) got early access. Interesting write-up: https://www.cognition.ai/blog/evaluating-coding-agents

It's a reference to Devin, one of the earlier (and most hyped) "autonomous" ai-agent-based software devs that it attempts to replicate/match in the open. Your interestingly different ire would be better-directed at the original project. https://www.cognition.ai/blog/introducing-devin Previous discussions on that fwiw include: https://news.ycombinator.com/item?id=39679787

Agreed. In fact, I thought this was an iteration on the previously announced "Devin" project until I realised that they have different spelling. https://www.cognition.ai/introducing-devin