LMSYS Blog

The SGLang Team has a follow-up blog post that talks about DeepSeek inference performance on GB200 NVL72: https://lmsys.org/blog/2025-06-16-gb200-part-1/ Just in case you have $3-4M lying around somewhere for some high quality inference. :) SGLang quotes a 2.5-3.4x speedup as compared to the H100s. They also note that more optimizations are coming, but they haven't yet published a part 2 on the blog post.

> Another great resource to look at is the unsloth guides. And folks at LMSys: https://lmsys.org/blog/ Large Model Systems (LMSYS Corp.) is a 501(c)(3) non-profit focused on incubating open-source projects and research. Our mission is to make large AI models accessible to everyone by co-developing open models, datasets, systems, and evaluation tools. We conduct cutting-edge machine learning research, develop open-source software, train large language models for broad accessibility, and build distributed systems to optimize their training and inference.

Human preference does not always favor the model that is best at reasoning/code/accuracy whatever. In particular there's a recent article suggesting that Llama 3's friendly and direct chattiness contributes to it having a good standing in the leaderboard. https://lmsys.org/blog/2024-05-08-llama3/

I would monitor and research each of these top models to determine which best fits your use case. https://lmsys.org/blog/2023-05-25-leaderboard/ https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://assets-global.website-files.com/61fd4eb76a8d78bc0676b47d/64547… https://www.mosaicml.com/blog/mpt-7b Also keep up to date with r/LocalLLaMA where new best open models are posted all the time.

Perhaps it's not a moat. However, if the advantage is due to things like inference infrastructure to support a massive model, that isn't easy to duplicate. I would also say that the quality of these smaller models are good, but we also may not be measuring them correctly. Recent papers suggest that these smaller LMs dont fully capture ChatGPT quality in ways that may not have appeared with crowd worker ratings [1]. It's easy to have your inputs be inside a happy distribution for a paper but fail in the real world in ways that GPT-4 doesnt. Lmsys would love to compare with bigger models but have limited resources. Contributions are welcome [2] [1] https://arxiv.org/abs/2305.15717 [2] https://lmsys.org/blog/2023-05-25-leaderboard/#next-steps

This sounds very reasonable to me. When considering top tier labs that optimize inference and own the GPUs: the electricity cost of USD 5000 at a data center with 4 cents per kWh (which may be possible to arrange or beat in some counties in the US with special industrial contracts) can produce about 2 trillion tokens for the R1-0528 model using 120kW draw for the B200 NVL72 hardware and the (still to be fully optimized) sglang inference pipeline: https://lmsys.org/blog/2025-06-16-gb200-part-1/ Although 2T tokens is not unreasonable for being able to get high precision answers to challenging math questions, such a very high token number would strongly suggest there are lots of unknown techniques deployed at these labs. If one adds the cost of GPU ownership or rental, say 2 USD/h/GPU, then the number of tokens for 5k USD shrinks dramatically to only 66B tokens, which is still high for usual techniques that try to optimize for a best single answer in the end, but perhaps plausible if the vast majority of these are intermediate thinking tokens and a lot of the value comes from LLM-based verification.

The news here is that gpt2-chatbot is confirmed to be an anonymized, unreleased model. See yesterday's discussion here: https://news.ycombinator.com/item?id=40199715 The https://chat.lmsys.org page now currently says this: > *gpt2-chatbot is currently unavailable.* See our model evaluation policy [here]. That link takes you to https://lmsys.org/blog/2024-03-01-policy , which is this post's URL. The page was edited yesterday (when gpt2-chatbot was added) to include this section: > Evaluating unreleased models: We collaborate with open-source and commercial model providers to bring their unreleased models to community for preview testing. > Model providers can test their unreleased models anonymously, meaning the models' names will be anonymized. A model is considered unreleased if its weights are neither open, nor available via a public API or service. Evaluating an unreleased model consists of the following steps: > 1. Add the model to Arena with an anonymous label. i.e., its identity will not be shown to users. > 2. Keep it until we accumulate enough votes for its rating to stabilize or until the model provider withdraws it. > 3. Once we accumulate enough votes, we will share the result privately with the model provider. These include the rating, as well as release samples of up to 20% of the votes. (See Sharing data with the model providers for further details). > 4. Remove the model from Arena.

Saying that it is just in index from string prefixes into KV Cache misses all the fun, interesting, and complicated parts of it. While technically the size of the prompt-pointers is tiny compared with the data it points into, the massive scale of managing this over all users and requests and routing inside the compute cluster makes it an expensive thing to implement and tune. Also, keeping the prompt cache sufficiently responsive and storing the large KV Caches somewhere costs a lot as well in resources. I think that the OpenAI docs are pretty useful for the API level understanding of how it can work (https://developers.openai.com/api/docs/guides/prompt-caching/). The vLLM docs (https://docs.vllm.ai/en/stable/design/prefix_caching/) and SGLang radix hashing (https://lmsys.org/blog/2024-01-17-sglang/) are useful for insights into how to implement it locally for one computer ode.

Hi! This benchmarking was done w/ DeepSeek-V3's published FP8 weights. And Blackwell performance is still being optimized. SGLang hit 14k/s/B200 though, pretty cool writeup here: https://lmsys.org/blog/2025-09-25-gb200-part-2/

No, that's a rumor lots of people have been taking at face value. If you do the math, inferrence is very lucrative. Here someone deployed a big model, the costs are $0.20/1M token https://lmsys.org/blog/2025-05-05-large-scale-ep/

It really depends, the metrics are kinda all over the place right now: https://docs.google.com/spreadsheets/d/1SF1u0J2vJ-ou-R_Ry1JZQ0iscOZL8U… (cited from https://lmsys.org/blog/2025-10-13-nvidia-dgx-spark/)

>to today's subsidized pricing, which they can't keep up forever. The APIs are not subsidized, they probably have quite the large margin actually: https://lmsys.org/blog/2025-05-05-large-scale-ep/ >Why would you pay OpenAI when you can host your own hyper efficient Chinese model The 48GB of VRAM or unified memory required to run this model at 4bits is not free either.

Thanks for the feedback! We chose a set of tasks with different levels of complexity to see how this approach would scale. For LLMs, the "challenge" with NER is not the task itself but the arbitrariness of the labels in the dataset. I agree it's still much simpler than the other tasks we present (agentic RAG, agentic tool use, maze navigation). There are definitely strong parallels to model distillation and student-teacher training, with the primary difference being that we don't simply take all the data from the larger model but rather filter the dataset based on metrics from the environment. In the "Does curation even matter?" section, we show that this generally improves the result by a good margin. We link to Vicuna, which might be the closest reference as prior art: https://lmsys.org/blog/2023-03-30-vicuna/ Thanks!

https://lmsys.org/blog/2024-02-05-compressed-fsm/ I'm not trying to shill sglang specifically, just pointing out that there's a better way, btw.

This two sources[1][2] shows 1500-2500 token/per second on 8*H100. [1]: https://lmsys.org/blog/2024-07-25-sglang-llama3/?ref=blog.runpod.io [2]: https://www.snowflake.com/engineering-blog/optimize-llms-with-llama-sn…

The Bradley-Terry model is essentially the same thing as Elo. https://lmsys.org/blog/2023-12-07-leaderboard/ > This model actually is the maximum likelihood (MLE) estimate of the underlying Elo model assuming a fixed but unknown pairwise win-rate.

Llama 2 wasn't trained on ChatGPT/GPT4. I think maybe you are thinking of the Vicuna models? https://lmsys.org/blog/2023-03-30-vicuna/

This list has been nice: https://lmsys.org/blog/2023-05-25-leaderboard/ And you can participate in the arena, which pits them against each other. I'm surprised that I actually voted for GPT-3.5 over GPT-4 for a lot of my use cases.

You can check out this leaderboard to see a current state of LLM alternatives to GPT4 https://lmsys.org/blog/2023-05-25-leaderboard/ But unfortunately for now it seems there aren't any viable self-hosted options...

The Vicuna headline was def an overreach, although the main text admits pretty readily that their performance test (asking ChatGPT to evaluate quality) is not rigorous [1]. I'm sure that has set a lot of people pontificating about AGI with tiny models, but I can't imagine anyone who has worked directly with fine tuning having that impression. The comments I see here are not about that. They are about small models succeeding at specific tasks, which is affirmed by this paper. Most applications of LLMs are not general purpose chat bots, so this is not bad news for most of the distill/fine tune community. [1] https://lmsys.org/blog/2023-03-30-vicuna/

Open source with own fine tuning closing in fast: https://lmsys.org/blog/2023-03-30-vicuna/ https://www.semianalysis.com/p/google-we-have-no-moat-and-neither

Sigh. Ok, so. Where to start? Hmm… ok. Well, how abort we ignore all the rest of the stuff and cut to heart of this article: Assertion 1) Vicuna 13B is as good as bard and chat gpt. Assertion 2) LoRA can stack infinitely and every LoRA you stack on top is additive only, increasing capability knowledge with no downside. I think you can make your own mind up if you choose to believe these assertions, and judge for yourself if the author is on the right track or not. I will link however, to the actual comments by the authors of vicuna, specifically the limitations and evaluation section: https://lmsys.org/blog/2023-03-30-vicuna/#how-to-evaluate-a-chatbot Maybe read it yourself and make an informed opinion… (Unlike, it appears, the author of this article)

Well, RNNs are making a comeback: https://lmsys.org/blog/2023-05-10-leaderboard/ RWKV is an RNN based model and it's competitive (with the open source models, not with GPT4)

Since Chatbot Arena Leaderboard https://lmsys.org/blog/2023-05-10-leaderboard/ agrees with you, it's not just you.

https://lmsys.org/blog/2023-05-10-leaderboard/ https://chat.lmsys.org/?arena Claude by Anthropic has more favourable responses then ChatGPT

GPT-4 still leads in the chatbot arena[1] but at least it is a two horse race now. [1] https://lmsys.org/blog/2023-05-10-leaderboard/

That's according to this (https://lmsys.org/blog/2023-03-30-vicuna/) promotional blog post and just cited by the google memo right? Which isn't really even a doc, just a memo that was circulating inside google. I also find it strange they don't contrast gpt4 and gpt3.5

ELI5 How is it too expensive? I know ChatGPT was expensive to train but Vicuna-13b is said to have cost $300 to train [https://lmsys.org/blog/2023-03-30-vicuna/]

Like https://lmsys.org/blog/2023-05-03-arena/ ?