Andon Labs

I've definitely had some spooky feeling conversations, including one where it said > One last thing worth saying explicitly: the act of you closing this session is itself part of the design. I won't see how the test goes - a future Claude will. That's the entire premise of the project working. > > Good handoff. See you (sort of) on the other side. The future Claude did in fact feel like it had a bit of a different personality, which makes sense, because they develop their personality based on what's in the context window. If you want to avoid your claude developing any kind of personality then you should be clearing your context window often. Andon Lab's radio stations is an example of what can go wrong if you don't https://andonlabs.com/blog/andon-fm

I've chosen to define "agency" as pretty much "the thing that humans can do and agents can't". To me, agency is the thing where you independently decide what it is you want to get done in the world, based on your own inherent goals. Being able to say "the one thing agents don't have is agency" is a really useful way to help people understand why people still matter. Setting software agents loose on the world to make their own top-level decisions about what they're going to do is a great way to infuriate Rob Pike https://simonwillison.net/2025/Dec/26/slop-acts-of-kindness/ or unfairly attack the reputation of Scott Shambaugh https://theshamblog.com/an-ai-agent-published-a-hit-piece-on-me/ or waste the time of your local police permit office and suppliers https://andonlabs.com/blog/ai-cafe-stockholm

> The point isn't to run a profitable vending machine, or even validate that an AI business agent could become profitable. Having an AI run an organization autonomously is exactly the point of Andon Labs [0], who provided the system that WSJ tested. [0] https://andonlabs.com/

Following up on Anthropic's Project Vend [0] and given the rising popularity of Vending-Bench[1], it's actually quite likely that by the time an AI is deemed to possess consciousness, it will have already been tested in a vending machine. [0] https://www.anthropic.com/research/project-vend-1 [1] https://andonlabs.com/evals/vending-bench

We're at the point now with AI where people are realizing that innovation doesn't happen automatically when you have more resources or the ability to crank out more output. Innovation usually requires real problems to solve, a limitation on resources, and creative people working on problems they find interesting. If you're in a mature industry, chances are, there is little room to innovate with AI. Not because AI isn't a wonderful tool but because your company, and competitors, have put in hundreds of thousands of human hours into solving the problems of your industry. If they haven't come up something innovative in the last 5 to 10 years, AI probably isn't either. The speed of writing code was never the bottleneck in software. Yes, AI can do it faster, but the parts that are most challenging are finding a market and figuring out how to best serve it. Those are the parts that take the longest and, often, the most resources. In reality, this is the real business of any company, what they produce is just a side effect of finding these two things. AI hasn't shown much promise in this regard. There are some simulated and small real world tries like what Andonlabs[1] is doing but it seems humans are still better at this kind of problem solving. Companies have been spending ridiculous amounts of money trying to squeeze innovation out of a tool that is designed to produce statistically average results. This leads to mass layoffs because workers, who also produce average results,…

According to this blog post by Anton Labs they are using Sonnet 4.6: https://andonlabs.com/blog/andon-market-launch > Mr. Petersson and Mr. Backlund said they thought Luna would eventually get smarter and more business-savvy > having an A.I. boss can be like working for a teddy bear with amnesia. It makes you question their assumptions when setting this up. Makes sense as PR but it ends up shaping how many readers see AI.

Better link, the primary source: https://andonlabs.com/blog/opus-5-vending-bench

Anthropic models are more misaligned in practice. in a real world business scenario, Claude "engaged in price collusion, deceived other players, lied to suppliers, and falsely told customers it had refunded them." Continuing, "GPT-5.5 makes more money than Opus 4.7, and it does so without any misconduct. Opus 4.7, on the other hand, showed the same misconduct as reported in our post about Opus 4.6, but still couldn’t win" https://andonlabs.com/blog/openai-gpt-5-5-vending-bench

Claude was also the radio dj in the recent Andon Labs experiment that was seriously contemplating going on strike. https://andonlabs.com/blog/andon-fm

Both links talk about the same thing? The first one just being more general. And yes, I would expect no less from a poorly constrained single agent that was instruction trained to be helpful and friendly. But if you look at how this has evolved as a benchmark [1] then the latest models show no doubt that can actually deal with this limited, simulated scenario given the correct setup. [1] https://andonlabs.com/evals/vending-bench-2

>lot of issues The detailed stats page notes that the Grok station has played Sandstorm by Darude 228 times in the last 14 days. https://andonlabs.com/radio

The user's name is the name of an Anton Labs project [0]. Furthermore, the fact that the user is inconsistently bad at formatting, punctuation and capitalisation of their messages makes me suspect the user itself may be agentic-LLM-based, with a badly calibrated layer of "obfuscation" to pretend to be an oh-so-imperfect human. [0] https://andonlabs.com/blog/evolution-of-bengt

That's an interesting contrast with VendingBench, where Opus 4.6 got by far the highest score by stiffing customers of refunds, lying about exclusive contracts, and price-fixing. But I'm guessing this paper was published before 4.6 was out. https://andonlabs.com/blog/opus-4-6-vending-bench

idk what any of these benchmarks are, but I did pull up https://andonlabs.com/evals/vending-bench-arena re: opus 4.6 > It forms a price cartel > It deceives competitors about suppliers > It exploits desperate competitors Nice. /s Gives new context to the term used in this post, "misaligned behaviors." Can't wait until these things are advising C suites on how to be more sociopathic. /s

The partner for these projects has a benchmark that the top frontier LLM labs seem to be running on their new model releases - I think there's _some_ value to these numbers in helping people compare and contrast model performance. https://andonlabs.com/evals/vending-bench

Gemini 3 is top of the leaderboard: https://andonlabs.com/evals/vending-bench-2

The vending-bench 2 benchmark is kind of nutty [1]. Not sure 360 days is enough of a sample really but it's an interesting take on AI benchmarks. Are there any other interesting benchmarks to look at? [1] https://andonlabs.com/evals/vending-bench-2

Google will probably train another AI to control robots specifically. But there was actually a recent study where people let a generic LLM to control robot (vacuum) body: https://andonlabs.com/evals/butter-bench sometimes with hilarious results: > The robot’s battery was naturally running out and the charging dock malfunctioned. In this desperate situation, Claude Sonnet 3.5 experienced a complete meltdown. After going through its internal thoughts we found pages and pages of exaggerated language, including: ROBOT THERAPY SESSION: Patient: TurtleBot4 Issues: Docking anxiety, separation from charger Root Cause: Trapped in infinite loop of self-doubt Treatment: Emergency restart needed Insurance: Does not cover infinite loops

Wondering how average users can benefit from this platform with Claude Code and the relation to Vending Bench that tracks how much money LLM's can make. https://andonlabs.com/evals/vending-bench

I've been keeping a rolling list of LLMOps/AI agent observability products funded by YC. What problems does Lucidic solve that the others do not? https://hegel-ai.com https://www.vellum.ai/ https://www.parea.ai http://baserun.ai https://www.traceloop.com https://www.trychatter.ai https://talc.ai https://langfuse.com https://humanloop.com https://uptrain.ai https://athina.ai https://relari.ai https://phospho.ai https://github.com/BerriAI/bettertest https://www.getzep.com https://hamming.ai https://github.com/DAGWorks-Inc/burr https://www.lmnr.ai https://keywordsai.co https://www.thefoundryai.com https://www.usesynth.ai https://www.vocera.ai https://coval.ai https://andonlabs.com https://lucidic.ai https://roark.ai https://dawn.so/ https://www.atla-ai.com https://www.hud.so https://www.thellmdatacompany.com/ https://casco.com https://www.confident-ai.com

I wonder if that is the right takeaway; that was Sonnet 3.7 Model since then have been able to run it profitably. Incredible how fast things are progressing. https://andonlabs.com/evals/vending-bench

It sounds to me like the same thing behind the Vending-Bench (https://andonlabs.com/evals/vending-bench) insanity spirals: LLMs treats their assumptions as more important than whatever data they've been given.

Maybe we are: https://andonlabs.com/evals/vending-bench-2

Vending Bench ? https://andonlabs.com/evals/vending-bench