Thinking Machines Blog

Yeah, besides that, the only big other open source US model that was worth looking at was Inkling (975B params), Jul 15, 2026. https://thinkingmachines.ai/news/introducing-inkling/

And still today we spend a great deal of effort trying to make our randomly-sampled LLM outputs reproducibly deterministic: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer…

Didn't many of the missionaries at OpenAI go to Thinking Machines Lab? https://thinkingmachines.ai/

Yes - THIS! I can't even believe how exhausting it is to read. I'm not sure why or what changed in Fable. Did they do this writing-style output to give it more token compression during/for training or to prefer output for less money? I love it for a few things, but it's gotten really hard to spend any extended amount of time with it because of the lack of mental model I seem to be able to hold while working with complicated problems. I'm guessing it's just not enough time doing RL on human feedback. Check out the anouncement of Inkling (https://thinkingmachines.ai/news/introducing-inkling/)... the section in the middle "Early in RL verbose, grammatical" (if you search) : We need to understand the operator. The 5D line element is ds² = e^{2A(x)} (ds²_4d + dx²), where A(x) = sin(x) + 4 cos(x), x in [0, 2π]. The internal coordinate is periodic. The background is a warped product: metric g_{MN} where M,N = 0..4. The internal direction has metric e^{2A(x)} dx²? Wait, the ds² is e^{2A} (ds²_4d + dx²). So the internal metric is e^{2A(x)} dx². Actually if the total metric is ds² = e^{2A(x)} (ds²_4d + dx²), then yes, internal metric is e^{2A} dx². vs. Post RL We need determine eigenvalue problem for spin-2 fluctuations h_{μν}(x,y) with TT in 4d and depend on x. For metric of form ds² = e^{2A(x)} (g_{μν}(y) + h_{μν}(y,x)) dy^μ dy^ν + e^{2A(x)}? Wait internal metric is e^{2A} dx²? Actually ds² = e^{2A} [ds_4² + dx²]. So internal metric is e^{2A} dx²; warp factor same for 4d and…

Determinism in agents is a complex topic because there are several different layers of abstraction, each of which may introduce its own non-determinism. But yeah, it is going to be difficult to induce determinism in a commercial coding agent, for reasons discussed below. However, we can start by claiming that non-determinism is not necessarily a bad thing - non-greedy token sampling helps prevent certain degenerate/repetitive states and tends to produce overall higher quality responses [0]. I would also observe that part of the yin-yang of working with the agents is letting go of the idea that one is working with a "compiler" and thinking of it more as a promising but fallible collaborator. With that out of the way, what leads to non-determinism? The classic explanation is the sampling strategy used to select the next token from the LLM. As mentioned above, there are incentives to use a non-zero temperature for this, which means that most LLM APIs are intentionally non-deterministic by default. And, even at temperature zero LLMs are not 100% deterministic [1]. But it's usually pretty close; I am running a local LLM as we speak with greedy sampling and the result is predictably the same each time. Proprietary reasoning models are another layer of abstraction that may not even offer temperature as knob anymore[2]. I think Claude still offers it, but it doesn't guarantee 100% determinism at temperature 0 either. [3] Finally, an agentic tool loop may encounter different…

Yeah, the dream is to have a "reasoning kernel" that you can plug specific knowledge and skills into ("I know kung fu"). I'm not aware of any progress on the dream though. Plenty of people are making LoRAs for general-purpose open models and there are companies like Thinking Machines that will help you. I'm not sure how you'd choose which base model to start with. I'm also not sure how to draw the boundary between LoRA and RAG. Some links I found but I can't vouch for them: https://github.com/oobabooga/textgen/wiki/05-%E2%80%90-Training-Tab https://thinkingmachines.ai/tinker/

Interesting paper by Thinking Machines where they solve this issue. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer… TLDR: It’s actually more about kernels changing with batch sizes, and you can solve it by making these kernels not depend on batch sizes. It took their inference time from 26s to 42s.

castform founder here. while it's "rare for a specialized model to beat a strong general model" today, i think the tech/knowhow on how to do so is getting there. we see some early signs of this in industry e.g. https://shopify.engineering/sidekicks-continual-learning-loop https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment… https://castform.com/blog/elsa/ at the end of the day, models are only as good as the data they're trained on. and if one has access to proprietary data, it should yield specialized models that do better than larger general models

This one from thinking machines: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer… Was discussed a lot also here on hn

They reference https://thinkingmachines.ai/blog/on-policy-distillation/ If I understand correctly, it's distillation via having a teacher model score each of the student's tokens for a problem based on their own probabilities of generating that token at each step in the sequence. The reward/loss is then applied as RL. The multi-teacher bit seems to imply they're distilling from multiple models. It's light on the details, but it seems like it could be part of distilling from frontier/closed models. Provided they calculate the logprobs, which OpenAI seems to allow via API but not Anthropic. Maybe they have a way of estimating the logprobs externally? This method can be used to learn any domain from the teacher. Biology included.

Is this the same model they are using for interaction models? https://thinkingmachines.ai/blog/interaction-models/

They have good docs on finetuning in general here: https://thinkingmachines.ai/tinker/ I used it last week for an application using a small model just as an experiment. It all went very well. The model did not turn out to be good though because my training data was of bad quality. I plan to work on it more this weekend.

yes. At least they want to use it for that, I am not sure they already do. (look under "multimodality" in the blog post: https://thinkingmachines.ai/news/introducing-inkling/)

Thinky's main commercial product AFAIK is Tinker [0] - companies pay them to host their fine-tuning workloads and then the resulting fine-tuned models. I don't know if this is a good business plan, but I'm sure at least one person there has read Joel on Software [1]. [0] https://thinkingmachines.ai/tinker/ [1] https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/

https://thinkingmachines.ai/model-card/inkling/ 975B parameter 41B active

They mention in the announcement link https://thinkingmachines.ai/news/introducing-inkling/ that they are still testing Inkling-Small and it will also still be multimodal. This makes it super interesting as a Deepseek V4 Flash replacement (and would be interesting with DwarfStar / ds4 if it gets supported).

They are one of the few labs (perhaps even the only one at this level) that are doing something both unique and useful, rather than simply imitating what the others are doing: https://thinkingmachines.ai/blog/interaction-models/

I looks like they took inspiration from Thinking Machines - http://thinkingmachines.ai/blog/interaction-models/

> However, due to things such as batching and even different kinds of floating point imprecisions for different algorithm implementations, the probability distribution itself often differs run-by-run, so what you sample from it also differs. Exactly. While I’m assuming this won’t be news for most here, for those that are still new and/or curious about some more explanation on e.g. the floating-point imprecisions, see this nice article: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer…

I mean they can be deterministic, thinking machines lab made deterministic LLM's: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer… Although obviously I'd rather they stay non-deterministic

You can literally make Cortana with modern LLMs. Or something close to it, at least, soon to be indistinguishable from it. Especially as models like this are trained: https://thinkingmachines.ai/blog/interaction-models/

LLM inference can be implemented in a way where nondeterminism depends only on the random seed, but that's not common. It ends up being more efficient/easier to implement kernels whose exact results depend on how many other prompts are being processed in parallel. See https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer… for a pretty extensive exploration.

"Defeating Nondeterminism in LLM Inference" ( https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer… has a repo: https://github.com/thinking-machines-lab/batch_invariant_ops which seems to have eventually been merged into vllm: https://docs.vllm.ai/en/latest/features/batch_invariance/ So you can get determinism locally. On a cursory search I wasn't able to find any LLM provider advertising determism; if you need it for research you might have to rent a dedicated GPU pod and run vllm there with the appropriate settings.

After the Thinking Machines announcement (https://thinkingmachines.ai/blog/interaction-models/) I've become interested in the ideas behind speech models, and the next "step-function" In Agent<>Human interaction. Was able to spin this up over the weekend. Here's how it works: GPT-Realtime-2 is the frontend. It has a tool to start a (Claude Code/Codex) task, and a tool to check the progress of said task. The agents run in an async worker while the Realtime loop can continue chatting. It becomes fairly straightforward to manage several agent runs at once, as if you are having a conversation.

I think the main benefit is improved speed and parallelism. Very similar to https://thinkingmachines.ai/blog/interaction-models/

GPUs are deterministic. It's not that hard to ensure determinism when running the exact same program every time. Floating point isn't magic: execute the same sequence of instructions on the same values and you'll get the same output. The issue is that you're typically not executing the same sequence of instructions every time because it's more efficient run different sequences depending on load. This is a good overview of why LLMs are nondeterministic in practice: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer…

Yeah you can: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer…

Quoting: "But why aren’t LLM inference engines deterministic? One common hypothesis is that some combination of floating-point non-associativity and concurrent execution leads to nondeterminism based on which concurrent core finishes first." From https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer…

> This isn't true From what I understand, in practice it often is true[1]: Matrix multiplication should be “independent” along every element in the batch — neither the other elements in the batch nor how large the batch is should affect the computation results of a specific element in the batch. However, as we can observe empirically, this isn’t true. In other words, the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies! This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs will also have this source of nondeterminism. [1]: https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-infer…