SemiAnalysis

He runs one of the industries leading analyst firms, probably not worth him having a reputation of making numbers up. https://semianalysis.com/

https://www.semianalysis.com/ is the best blog by far...all the YouTubers (Asianometry as well) get the analysis from that blog.

https://semianalysis.com/ They have some of the best publicly available analysis on these topics. The full details and numbers are hidden behind the institutional accounts which are priced for investors (not something you sign up for personally) but they're generous with what they send out in their newsletter. If you're not familiar with resources like this I could understand how you'd assume that the providers are hemorrhaging money on inference costs, because that is that story that gets parroted around spaces like Hacker News. You could ignore all of that, though, and go check OpenRouter to see how much providers are selling high parameter count models. They're not entirely at the level of the SOTA models, but the biggest open weight models are not that far behind in complexity either. They're being sold an order of magnitude cheaper than what you pay for the APIs from the major players. We don't know exactly how big the major models are, but it's unlikely that they're more than 10X more compute intensive from the leaks we do have.

you might be looking for SemiAnalysis? I only read the free portions of articles but they have various paid options, mostly targeting investors with information and tools. https://semianalysis.com/

Google figured this out with their paper from 2023, We have no moat and neither does OpenAI. The moat now is the harness and being able to recursively self improve from RLHF, a great example is how Grok used to be pretty bad but since SpaceX bought Cursor, they used that data to train Grok 4.5 which is now very competent at coding and even exceeds frontier models in certain benchmarks. https://www.semianalysis.com/p/google-we-have-no-moat-and-neither

> So the question is can they keep the pricing up on the older ones a few years down the line They don't expect to keep the prices flat over time, and everyone involved will have planned for this. Prices are highest when they're the newest and greatest (part of why it's valuable for neoclouds to be first in line for new models), and drop year by year as newer GPU models can do equivalent work at lower cost. You can see a pretty cool dataset of this at [1]; H100 prices where $3/hr in 2023, and dropped linear-ish to $1.75/hr by 2025. And also the notable exception that prices are up this year due to shortage. [1] https://semianalysis.com/gpu-pricing-index/

> early testing shows that Jalapeño will deliver performance per watt substantially better than current state-of-the-art We're starting to see what really matters here, and though this is hand wavy the TPU makes similar claims. I think googles memo about having no moat still stands (see: https://newsletter.semianalysis.com/p/google-we-have-no-moat-and-neith… if you are unaware). It kind of makes sense that all of this is looking more like 60's to 90's IBM, DEC, Cray, Sun and the hardware race that happened then. History doesn't repeat but it often rhymes and I suspect that these efforts will follow the same trajectory.

> what Alibaba spends their time doing? Not most of the time (pre-training takes a long time), but post-training is where most of the value is, yes. Famously it is all that OpenAI did between GPT 4o and GPT 5.3 (or 5.2?) - they didn't manage to complete a pre-training run[1], and all their progress was done with post-training (!) Post training what Cursor spends their time doing, and that has built a model that is competitive with the best coding models out there. It isn't limited at all. If you want to complain about something not being open source, complain about the lack of good open source RL environments (Prime Intellect excepted). [1] https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-swing-at-th…

This recent article from Semianalysis did a great job explaining part of it: https://newsletter.semianalysis.com/p/are-ai-datacenters-increasing-el…

But this was likely locked up years ago before this boom. Will Apple still be the premier customer for TSMC's next node A14? Apple was actually told by TSMC to move off of N3 asap because Nvidia with its Vera Rubin and Google TPUs will take over. Semianalysis had a great and detailed article about TSMC & Apple and how the future might play out: https://newsletter.semianalysis.com/p/apple-tsmc-the-partnership-that-…

If you try to track down the actual source for that Tom's Hardware link, it becomes pretty obvious that the claim is not credible. [0] GPUs do not burn out in three years, H100 rentals are priced at the same level as two years ago, and are effectively sold out. [1] [0] https://news.ycombinator.com/item?id=46203986#46208221 [1] https://newsletter.semianalysis.com/p/the-great-gpu-shortage-rental-ca…

I agree with you on that metric being not great - I would have definitely swapped it for this: "Claude Code GitHub Commits Over Time" https://newsletter.semianalysis.com/p/claude-code-is-the-inflection-po… Sure - also an imperfect metric. But less imperfect? And more indicative of... something? Not nothing?

It's one of the most popular "inside baseball" blogs in AI. Dylan Patel covers the people, tech, hardware, business analytics, and has amazing insight and access to people. "Blog" isn't quite right, but if you subscribe, you get a ton of useful analysis and reporting and writing. https://semianalysis.com/about/

Just because people aren’t talking about how LLMs are boosting their productivity to ship “insanely great” products doesn’t mean it’s not happening. I don’t talk about every tool I use, and neither do many of the best teams who keep their eyes on the prize and relentlessly ship. Meanwhile this story just came out: https://newsletter.semianalysis.com/p/claude-code-is-the-inflection-po…

Indeed. Why do people follow these clowns? They seem to read high level takes and spew out their nonsense theories. They fail to mention Google's edge: Inter-Chip Interconnect and the allegedly 1/3 of price. Then they talk about software moat and it sounds like they never even compiled a hello world in either architecture. smh And this comes out days after many in-depth posts like: https://newsletter.semianalysis.com/p/tpuv7-google-takes-a-swing-at-th… A crude Google search AI summary of those would be better than this dumb blogpost.

Here's a sober analysis based on actual experts with access to more information than what is publicly released: https://newsletter.semianalysis.com/p/how-to-kill-2-monopolies-with-1-… Skeptics welcome to review the following section: > These are extraordinary claims and thus demand extraordinary evidence. Let’s take them one-by-one: And here is their conclusion: > Naysayers will point out a million reasons why this is improbable, difficult, etc. - and they are mostly correct. There is a big difference between lab-scale and industrialized, high-volume tools. Substrate itself realizes this and agrees they are in for a lot of development and scaling pain. Still, they have at least developed some impressive capabilities on the most complex part of the process (litho) in a short amount of time (2-3 years).

I think they will be, but more for hand-off. Local will be great for starting timers, adding things to calendar, moving files around. Basic, local tasks. But it also needs to be intelligent enough to know when to hand off to server-side model. Android crowd has been able to run LLMs on-device since LlamaCPP first came out. But the magic is in the integration with OS. As usual there will be hype around Apple, idk, inventing the very concept of LLMs or something. But the truth is neither Apple nor Android did this; only the wee team that wrote the attention is all you need paper + the many open source/hobbyist contributors inventing creative solutions like LoRA and creating natural ecosystems for them. That's why I find this memo so cool (and will once again repost the link): https://semianalysis.com/2023/05/04/google-we-have-no-moat-and-neither…

> In order to break even they would have to minimize the operating costs (by throttling, maiming models etc.) and/or increase prices. This would be the reality check. Is that true? Are they operating inference at a loss or are they incurring losses entirely on R&D? I guess we'll probably never know, but I wouldn't take as a given that inference is operating at a loss. I found this: https://semianalysis.com/2023/02/09/the-inference-cost-of-search-disru… which estimates that it costs $250M/year to operate ChatGPT. If even remotely true $10B in revenue on $250M of COGS would be a great business.

> DeeoSeek trained r1 for 1.25% (5M) of that money (using the same spot price) on 2048 crippled export H800s and is maybe a month behind. This is a great example of how a misleading narrative can take hold and dominate discussion even when it's fundamentally incorrect. SemiAnalysis documents that DeepSeek has spent well over $500M on GPUs alone, with total infrastructure costs around $2.5B when including operating costs[0]. The more-interesting question is probably why do people keep repeating this? Why do they want it to be true so badly? [0]: https://semianalysis.com/2025/01/31/deepseek-debates/#:~:text=Our%20an…

> Dylan Patel comes to essentially the same conclusions (and this is a better post I think): https://semianalysis.com/2024/12/09/intel-on-the-brink-of-death/ Thanks, good reading. Sharp retrospective and analysis, hopeful blueprint for the future: """ Intel Foundry won’t be laden with Intel’s lagging product team, Mobileye, or Altera. Intel Foundry will have one clear function, and it’s vital to national security and the future of America and the West. Who should lead the charge here? Maybe a “Chip Czar” charged with restoring American logic prowess. We know someone with a great CV who has just become freed up for new opportunities… """

> Replacing search is enough to warrant trillion dollar valuation for LLMs, question is just if they can get that good. I would add that it needs to be more than just good, it also needs to be cheaper for mass adoption. Spoiler: classic search is at least cheaper (old post, but relevant: https://www.semianalysis.com/p/the-inference-cost-of-search-disruption… The thing is it cannot get that much cheap without disrupting earning expectations from hyperscalers.

A reference to the good doc: https://www.semianalysis.com/p/google-we-have-no-moat-and-neither While I'm linking semianalysis, though, it's probably worth talking about how everyone except Google is GPU poor: https://www.semianalysis.com/p/google-gemini-eats-the-world-gemini (paid) > Whether Google has the stomach to put these models out publicly without neutering their creativity or their existing business model is a different discussion. Google has a serious GPU (well, TPU) build out, and the fact that they're able to train moe models on it means there aren't any technical barriers preventing them from competing at the highest levels

According to this and SemiAnalysis[1], the MI300X seems impressive. Better than Nvidia's H100 and maybe a bit inferior to the H200. Why is Nvidia worth $1 trillion more? [1] - https://www.semianalysis.com/p/amd-mi300-performance-faster-than

Previously posted about here: https://news.ycombinator.com/item?id=36671588 and here: https://news.ycombinator.com/item?id=36674905 With the original source being: https://www.semianalysis.com/p/gpt-4-architecture-infrastructure The twitter guy seems to just be paraphrasing the actual blog post? That's presumably why the tweets are now deleted. --- The fact that they're using MoE was news to me and very interesting. I'd love to know more details about how they got that to work. Variations in that implementation would explain the fluctuations in the quality of output that people have observed. I'm still waiting for the release of their vision model which is mentioned here but we still know little about, sans a few demos a few months ago.

There was a very detailed post with die shots etc on semi analysis a few weeks back https://www.semianalysis.com/p/zen-4c-amds-response-to-hyperscale In the subscriber only section it was mentioned there will be some Zen5 consumer parts using Zen5c as the equivalent of current Intel E-cores.

> nobody at this point expects a 13B parameter model to succeed with the same accuracy at the broad range of tasks supported by what may be a 1T parameter model I think a lot of people believe exactly that. To take one example from the "We Have No Moat" essay: "It doesn’t take long before the cumulative effect of all of these fine-tunings overcomes starting off at a size disadvantage. Indeed, in terms of engineer-hours, the pace of improvement from these models vastly outstrips what we can do with our largest variants, and the best are already largely indistinguishable from ChatGPT." - https://www.semianalysis.com/p/google-we-have-no-moat-and-neither

Following the leak of Llama, they now probably enjoy having a worldwide open-source community improving their models for free, as elaborated in this leaked (sic!) Google document: https://www.semianalysis.com/p/google-we-have-no-moat-and-neither

Odd to see this the day after the appearance of what's supposed to be a "leaked" memo from inside Google advocating the opposite course -- claiming that the outsider crowd playing with (mostly) LLaMa derivatives is getting good results, and that the best thing for Google might be to try to engage. It claims, for example, that the open-source crowd, trying to find cheaper training methods using smaller, more targeted data sets -- because they have to -- has found ones that work as well, by some metrics, as anything that the groups with huge compute clusters. And that because they work as well, with less resources, they can subsequently explore alternatives a whole lot quicker than anything involving training new base models from scratch. The (supposed, unsourced) leak: https://www.semianalysis.com/p/google-we-have-no-moat-and-neither

Are you sure they will? They seem to prefer dividend payouts rather than capex on new fabs[1]. Combined with them moving to FCF loss - I think that what you described here may be imaginary at best. Given what they said in the last earnings call, a slow IBM-like death seems more likely. 1: https://semianalysis.com/intel-cuts-fab-buildout-by-4b-to-pay-billions…

Huge fan of SemiAnalysis run by Dylan Patel https://semianalysis.substack.com/ I have yet to find a better in-depth source for hardware ongoings.