Hazy Research Blog

There are also interesting approaches to more directly compress a large document or an entire codebase into a smaller set of tokens without getting the LLM to wing it. For example, Cartridges: <https://hazyresearch.stanford.edu/blog/2025-06-08-cartridges> They basically get gradient descent to optimize the KV cache while freezing the network.

https://hazyresearch.stanford.edu/blog/2025-03-15-tk-blackwell cooperative execution yeah as you can tell I do not do CUDA for a living :D

I think Hopper's native matmul tile is 64x64, and Blackwell is 128x128. see this blog for a reference on Blackwell: https://hazyresearch.stanford.edu/blog/2025-03-15-tk-blackwell

https://hazyresearch.stanford.edu/blog/2023-12-11-truly-subquadratic is an approachable overview of the Monarch approach, for those interested! There's a lot of opportunity here. Just because matrix multiplication makes for a beautiful mathematical building block, and a very reasonable one to build high-level ML logic on, doesn't mean it needs to be computed the same way, and in the same order, that we learned in linear algebra courses. I'm quite curious if this is being used in practice at scale, or whether it's still in the lab at the moment!

They already "went brr" when they announced ThunderKittens a year ago: https://hazyresearch.stanford.edu/blog/2024-05-12-tk

- Paper(pdf): https://hazyresearch.stanford.edu/static/posts/2025-11-09-hk/hipkitten… - Code: https://github.com/HazyResearch/HipKittens

From their repo: tl;dr When we put lots of text (e.g. a whole code repo) into a language model's context, generation cost soars because of the KV cache's size. What if we trained a smaller KV cache for our documents offline? Using a test-time training recipe called self-study, we show that this simple idea can improve throughput by 26× while maintaining quality. Link to their blog post: https://hazyresearch.stanford.edu/blog/2025-06-08-cartridges

This project is from CMU. Hazy Research at Stanford talked about the megakernel too: https://hazyresearch.stanford.edu/blog/2025-05-27-no-bubbles Good to see the competition in this area. (Edited): Related paper covering the larger "mirage" project, but this doesn't cover the "megakernel" approach: https://arxiv.org/abs/2405.05751

> rather than the goal on only having to trust the design & chip-manufacturer If you'd rather self-host, then the HazyResearch Lab at Stanford recently announced a FOSS e2ee implementation ("Minions") for Inference: https://hazyresearch.stanford.edu/blog/2025-05-12-security / https://github.com/HazyResearch/Minions

Paper: https://arxiv.org/abs/2502.15964 Blogpost at Hazy Research: https://hazyresearch.stanford.edu/blog/2025-02-24-minions GitHub repo: https://github.com/HazyResearch/Minions

What? This is really wrong. The name of the game these days is optimizing memory and FLOPs usage on these GPUs. For example, Hopper (H100) cards introduced the WGMMA (Warp-group Matrix-Multiply Accumulate) instruction, and anyone who does LLM training jumped to use it ASAP because without it you can't fully utilize the FLOPs. Anyone training AIs at the cutting age cares a lot about CUDA details. https://hazyresearch.stanford.edu/blog/2024-05-12-tk

Something like this? https://hazyresearch.stanford.edu/blog/2024-07-01-jrt

Transformers actually have an quantifiable state size (see https://hazyresearch.stanford.edu/static/posts/2024-06-22-ac/tradeoff_… although it's anywhere between 200k and 2M floats (for 360M and 1.33B respectively iinm). So a sufficiently sized RNN could have the same state capacity as a transformer. (this is from the Based paper: https://arxiv.org/pdf/2402.18668)

Research suggesting that much of the power of the transformer architecture comes from associative recall over long sequences that does not require scaling model dimensions. They design state space models that narrow the gap. Overview https://hazyresearch.stanford.edu/blog/2023-12-11-zoology0-intro Zoology 2 https://hazyresearch.stanford.edu/blog/2023-12-11-zoology2-based Monarchs and Butterflies: Towards Sub-Quadratic Scaling in Model Dimension. Scaling in model dimension as opposed to sequence dimension scaling in the previous posts. https://hazyresearch.stanford.edu/blog/2023-12-11-truly-subquadratic

This blog about “Based” came out just before neurIPS: https://hazyresearch.stanford.edu/blog/2023-12-11-zoology2-based

I guess they don't compare with S4? https://hazyresearch.stanford.edu/blog/2022-01-14-s4-1

This is relevant also: https://hazyresearch.stanford.edu/blog/2023-03-07-hyena

Impressively applied here: https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna

>> The next major leap in LLMs (in the next year) is probably going to be the prompt context size. Right now we have 2k, 4k, 8k ... but OpenAI also has a 32k model that they're not really giving access to unfortunately. Saw this article today about a different approach that opens up orders of magnitude larger contexts https://hazyresearch.stanford.edu/blog/2023-03-07-hyena

I think Sam is referring to transition from "Deep" to "Long" learning [1]. What new emergent properties, if any, will 1 billion tokens unlock? [1] https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning

I think it's unlikely that the first model to be widely considered AGI will be a transformer. Recent improvements to computational efficiency for attention mechanisms [0] seem to improve results a lot, as does RLHF, but neither is a paradigm shift like the introduction of transformers was. That's not to downplay their significance - that class of incremental improvements has driven a massive acceleration in AI capabilities in the last year - but I don't think it's ultimately how we'll get to AGI. [0] https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning

Attention used to be quadratic. But there have been linear attention advances. In fact, gpt-4 is almost certainly using some linear attention variant, maybe flash attention. Then there's this https://hazyresearch.stanford.edu/blog/2023-03-27-long-learning I don't think a 300 page context window is too far away

Also Flying Squid, another interesting project from Stanford - http://hazyresearch.stanford.edu/flyingsquid

A medium-spicy take of mine is that a bytecode VM in a GPU kernel is not as bad of an idea as one might think, and in some cases it can actually be the most reasonable solution. Some fun examples: 1. As mentioned in the post above, the Dolphin emulator famously implements the entire Gamecube/Wii GPU pipeline in a single gigantic ubershader, and this is useful because it avoids shader compilation stalls [1]. 2. Blender's Cycles renderer implements its shading graph eval system as a bytecode VM in a GPU kernel [2]. IIRC early versions of Vray GPU did something similar. There are better ways of course, but a VM gets you surprisingly far as a general approach. 3. Finally, a lot of ML frameworks (Tensorflow, PyTorch, etc) by default use the GPU relatively suboptimally (especially without kernel fusion and such). Tensor frameworks can extract a lot more perf out of GPUs using a VM-in-a-giant-kernel approach [3]. If you think abstractly about how a GPU SM actually works (using CUDA terminology here), all threads in a warp must execute in lockstep and the cost of execution divergence across threads in a warp is that you effectively run serially, losing the parallel advantage of the SM. This penalty gets magnified enormously if you are doing memory reads after wherever the execution divergence happens, since you now have multiple slow memory stalls in serial instead of one big memory read at once for all threads. If you're clever about implementing a bytecode VM, you can load as…

Here's a list of tools for scaling up transformer context that have github repos: * FlashAttention: In my experience, the current best solution for n² attention, but it's very hard to scale up beyond the low tens of thousands of tokens. Memory use is O(n) but compute is O(n²). Code: https://github.com/HazyResearch/flash-attention * Heinsen Routing: In my experience, the current best solution for n×m attention, i.e., mapping n tokens to m tokens. It's like a souped-up version of attention. I've used it to pull up more than a million tokens as context. Memory use and compute are O(nm). It works, but in my (limited) experience, it doesn't work out-of-the-box as well as FlashAttention for n² attention. Code: https://github.com/glassroom/heinsen_routing * RWKV: A sort-of-recurrent model which claims to have performance comparable to n² attention in transformers. In my (limited) experience, it doesn't. Others seem to agree: https://twitter.com/arankomatsuzaki/status/1639000379978403853 . Code: https://github.com/BlinkDL/RWKV-LM * RMT (this method): I'm skeptical that the recurrent connections will work as well as n² attention or n×m routing in practice, but I'm going to give it a try. Code: https://github.com/booydar/t5-experiments/tree/scaling-report In addition, the group that developed FlashAttention is working on state-space models (SSMs) that look promising to me. The idea is to approximate n² attention dynamically using only O(n log n) compute. There's no code…

We (a few PhD students at Stanford Hazy Research) just released an open-source Python library called [Meerkat](http://meerkat.wiki) where we built an interactive data frame powered by foundation models to wrangle unstructured data (images, videos, text docs...). Unstructured data is rich, diverse, complex, and pervasive which also makes it incredibly challenging to visualize, interact with, and use. We think large foundation models (FMs) like GPT-3 or CLIP will change this by allowing us to directly operate on such data. FMs are improving rapidly in their ability to read, interpret and generate unstructured text, images, even videos. The main problem is that FMs need to be controlled, and made more reliable as a software abstraction. Right now, it's like you're on a ship in a storm, and you're fighting the waves without a steering wheel. We felt that a lot of existing tools (think pandas or even Streamlit) were tailored to structured data, and weren't really well suited to the new challenges that are emerging in this paradigm. We built Meerkat to help. Meerkat systematically combines data frames with FMs and interactivity in a way that preserves the flexibility of FMs with added control. We aren't trying to reinvent the wheel, and we're hoping that we can lean on familiar interfaces like data frames while bringing something new to the table. If you're interested in details, you can read more about Meerkat on our research blog:…

Scaling if context window size has been a problem but now good potential of solutions using mamba. HyenaDNA is one to look at wrt DNA. And here are some other interesting links from Erik Garrison—a leader in pangenomics. https://hazyresearch.stanford.edu/blog/2023-06-29-hyena-dna https://github.com/instadeepai/nucleotide-transformer https://dl.acm.org/doi/pdf/10.1145/3535508.3545512 https://github.com/dnbaker/bioseq https://huggingface.co/AIRI-Institute/gena-lm-bert-base https://discuss.huggingface.co/t/dna-long-sequence-tokenization/16154