Chroma Research

Incidentally, Chroma also produced the single best study on long-context degradation that I've come across: https://research.trychroma.com/context-rot Before that, I cited nolima (https://www.reddit.com/r/LocalLLaMA/comments/1io3hn2/nolima_longcontex… constantly to illustrate how difficult tasks involving reasoning or multi-step information gathering degraded much faster than the needle-in-haystack benchmarks cited by the major labs. Now Chroma is the first stop. Nice job on the research!

Artifacts can reduce token usage by a factor of 2 or more. I had an [agent evolution framework](danieltan.weblog.lol/2025/06/agent-lineage-evolution-a-novel-framework-for-managing-llm-agent-degradation) before that dumped the output analysis into chat. It often timed out before the 10th conversation. After dumping the analysis into an artifacts, and have the LLM only edit it as required, I can go to 15 or more rounds without hitting the context limit. While they seem to re-output the entire artifact each time, they don't actually consume the tokens for the entire artifact. This also greatly reduces the tendency of HALO-style rampancy, or AI psychosis which is also what the recent paper on context-rot/poisoning (https://research.trychroma.com/context-rot) is about.

High-level, Chroma lets you incorporate your own data into AI. That can be business data, like your help docs, or customer data, such as private documents. I think there's a misconception among many non-technical people that they should "fine tune" models - to make an LLM sound like their brand or know their internal data. But, the best practice today is "context engineering" - to use a commodity LLM, and add propriety information into the prompt. The hard part of context engineering is knowing which data to incorporate. For example, all help docs in every context creates context rot [1] that hurts accuracy, so you need a way to find which information to put in which context. The solution is essentially an internal search engine, which is Chroma. Chroma gives AI multiple ways to search, and you can either hard-code which one works best for you - or let an agent write its own queries to conduct research by itself. Vector search is a generation ahead of old search infrastructure, and is useful for relatedness queries - like "help docs about billing". Full-text search is useful for proper nouns, like "Next.js". Regex search is useful for code and laws. Metadata search is more nuanced, but becomes really important in document search (e.g., PDFs). Chroma lets you run all of these search methods against private data, and you can use it to even include citations in results. So, the high-level answer is: Chroma enables you to incorporate your business or customer data into…

this is a pretty important piece and the research backs you up. Moving that context out of your system prompt dynamically is going to help reduce your lost in the middle effect. Context rots almost immediately. I've got a project that is being built to address this directly as well, but I'm still very early days. Keep it up! you're on the right track. Hong, K., & Chroma Research Team. (2025). Context rot: How increasing input tokens impacts LLM performance. Chroma Research. https://research.trychroma.com/context-rot Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., & Liang, P. (2024). Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12, 157–173. https://doi.org/10.1162/tacl_a_00638

This is well established at this point, it’s called “context rot”: https://research.trychroma.com/context-rot

The issue is that once you pollute your context window with the “wrong” information even after you have guided the LLM to the right path, it is still more likely to go off the rails. https://research.trychroma.com/context-rot

Unfortunately, I think the context rot paper [1] found that the performance degradation when context increased still occurred in models using attention sinks. 1. https://research.trychroma.com/context-rot

As the context grows, the output (usually a structured function call) remains relatively short. This makes the ratio between prefilling and decoding highly skewed in agents compared to chatbots. The problem is that context engineering is still an emerging science, even though for agent systems, it's already essential. This is overall a really interesting post and makes me think of Chroma’s technical report on context rot: https://research.trychroma.com/context-rot

super-edit: Sorry, this is not a usage related question, I have move it to: https://news.ycombinator.com/item?id=47772971 Here is the question for which I cannot find an answer, and cannot yet afford to answer myself: In Claude Code, I use Opus 4.6 1M, but stay under 250k via careful session management to avoid known NoLiMa [0] / context rot [1] crap. The question I keep wanting answered though: at ~165k tokens used, does Opus 1M actually deliver higher quality than Opus 200k? NoLiMa would indicate that with a ~165k request, Opus 200k would suck, and Opus 1M would be better (as a lower percentage of the context window was used)... but they are the same model. However, there are practical inference deployment differences that could change the whole paradigm, right? I am so confused. Anthropic says it's the same model [2]. But, Claude Code's own source treats them as distinct variants with separate routing [3]. Closest test I found [4] asserts they're identical below 200K but it never actually A/B tests, correct? Inside Claude Code it's probably not testable, right? According to this issue [5], the CLI is non-deterministic for identical inputs, and agent sessions branch on tool-use. Would need a clean API-level test. The API level test is what I really want to know for the Claude based features in my own apps. Is there a real benchmark for this? I have reached the limits of my understanding on this problem. If what I am trying to say makes any sense, any help would be…

Okay, here's the tl;dr: Attention based neural network architectures (on which the majority of LLMs are built) has a unit economic cost that scales (roughly) n^2 i.e. quadratic (for both memory and compute). In other words, the longer the context window, the more expensive it is for the upstream provider. That's one cost. The second cost is that you have to resend the entire context every time you send a new message. So the context is basically (where a, b, and c are messages): first context: a, second context window: a->b, third context window: a->b->c. It's a mostly stateless (there are some short term caching mechanisms, YMMV based on provider, it's why "cached" messages, especially system prompts are cheaper) process from the point of view of the developer, the state i.e. context window string is managed by the end user application (in other words, the coding agent, the IDE, the ChatGPT UI client etc.) The per token cost is an amortized (averaged) cost of memory+compute, the actual cost is mostly quadratic with respect to each marginal token. The longer the context window the more expensive things are. Because of the above, AI agent providers (especially those that charge flat fee subscription plans) are incentivized to keep costs low by limiting the maximum context window size. (And if you think about it carefully, your AI API costs are a quadratic cost curve projected into a linear line (flat fee per token, so the model hosting provider in some cases may make more…

I've seen a couple of power users already switching to Pi [1], and I'm considering that too. The premise is very appealing: - Minimal, configurable context - including system prompts [2] - Minimal and extensible tools; for example, todo tasks [3] - No built-in MCP support; extensions exist [4]. I'd rather use mcporter [5] Full control over context is a high-leverage capability. If you're aware of the many limitations of context on performance (in-context retrieval limits [6], context rot [7], etc.), you'd truly appreciate Pi lets you fine-tune context for optimal performance. It's clearly not for everyone, but I can see how powerful it can be. --- [1] https://lucumr.pocoo.org/2026/1/31/pi/ [2] https://github.com/badlogic/pi-mono/tree/main/packages/coding-agent#sy… [3] https://github.com/mitsuhiko/agent-stuff/blob/main/pi-extensions/todos… [4] https://github.com/nicobailon/pi-mcp-adapter [5] https://github.com/steipete/mcporter [6] https://github.com/gkamradt/LLMTest_NeedleInAHaystack [7] https://research.trychroma.com/context-rot