AMD ROCm Blogs

Oh, interesting! Thanks for the additional context. This is one area where it is clear that Nvidia is a leader by not only providing the underlying kernels, but also the overall product framework integrations. One would have to port that entire plugin project over, which would probably be a ton of work to maintain. For what it is worth, AMD just recently released two blog posts on hipDF, so at least they are putting that effort in. https://rocm.blogs.amd.com/artificial-intelligence/cupy_hipdf_portfoli… https://rocm.blogs.amd.com/artificial-intelligence/hipDF_pandas_accele…

It's great to see vLLM getting faster/better for DeepSeek. I tested vLLM vs SGLang a couple weeks ago and SGLang's DeepSeek support was much better/faster (on 2 x p5 H100 nodes). It's great that no one's standing still, I saw this recent AMD article that reported SGLang perf on MI300X has increased by 4X over the past couple weeks: https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1_Perf/REA… (w/ the extra memory V3/R1 fits on a single MI300X or H200 node) It'll be interesting to see if either project can take advantage/get any benefits from this FlashMLA implementation.

(Split into two parts due to comment length restrictions) > I work closely with OLCF and Frontier (I have a job running on Frontier right now). This is incorrect. The overwhelming majority of compute and resource allocation are not "nuclear stockpile modeling code" projects or anything close to it. AMD often gets directly involved with various issues (OLCF staff has plenty of stories about this). I know because I've spoken with them and AMD. I don't have any experience running a job on one of these national supercomputers, so I'll defer to you on this. (Atomic Canyon looks very cool!) Just two follow-ups then: is it the case that any job, small or large, enjoys this kind of AMD optimization/debugging support? Does your typical time-grant/node-hour academic awardee get that kind of hands-on support? And, for nuclear modeling (be it weapons or civilian nuclear), do you know if AMD engineers can get involved? (https://insidehpc.com/2023/02/frontier-pushes-boundaries-86-of-nodes-e… this article claims "86% of nodes" were used on at least one modeling run, which I imagine is among the larger jobs) > It is and has been miles beyond the competition and that's clearly all you need. Nvidia has > 90% market share and is worth ~10x AMD. 17 years of focus and investment (30% of their R&D spend is software) when your competitors are wandering all over the place in fits and starts will do that. No dispute here that NVIDIA is the market leader today,…

Spir-v is already happening: https://rocm.blogs.amd.com/software-tools-optimization/spir-v-rocm/REA…

https://rocm.blogs.amd.com/artificial-intelligence/rocjpeg-decoding-pe… https://alax.info/blog/1992 https://en.wikipedia.org/wiki/Unified_Video_Decoder added around H.265 in UVD 6.0

This reads like a badly done, sponsored hype video on YouTube. So if we look at what NVIDIA has to say about NVFP4 it sure sounds impressive [1]. But look closely that initial graph never compares fp8 and fp4 on the same hardware. They jump from H100 to B200 while implying a 5x jump of going with fp4 which it isn't. Accompanied with scary words like if you use MXFP4 "Risk of noticeable accuracy drop compared to FP8" . Contrast that with what AMD has to say on the open MXFP4 approach which is quite similar to NVFP4 [2]. Ohh the horrors of getting 79.6 instead of 79.9 on GPQA Diamond when using MXFP4 instead of FP8. [1] https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-… [2] https://rocm.blogs.amd.com/software-tools-optimization/mxfp4-mxfp6-qua…

The benchmark this article relies on is here[1], and is focused on a single-user case and uses QwQ-32B-AWQ with a context length of 4096 as far as I can see. Which, from what I can gather, is a quite unrealistic setup for someone seriously considering buying a H100 for the home lab. That said, the tensor parallelism[2] of vLLM is interesting. [1]: https://github.com/DeutscheKI/llm-performance-tests#vllm-performance-t… [2]: https://rocm.blogs.amd.com/artificial-intelligence/tensor-parallelism/…

I think a lot of stuff will get upstreamed eventually. PyTorch just moves slower and since it’s a stable library, I think it cannot rapidly adopt something like fused MoE until the dust has settled a little and it’s clear what the API would look like long-term. I think it’s ok that stuff is tried first in Torch extensions. That’s how Flash Attention started after all and the same is true for newer kernels in CUDA-land (fused MoE, MLA, Marlin, etc.). With regards to TorchScript, that’s really legacy - torch.compile is where it’s at. This post seems to suggest that the kernels work with torch.compile: https://rocm.blogs.amd.com/artificial-intelligence/DeepSeekR1-Part2/RE…

We have a replacement for CUDA, it is called C++17 parallel algorithms. It has vendor support for running on the GPU by Intel, AMD and NVIDIA and will also run on all your cores on the CPU. It uses the GPU vendors compiler to convert your C++ to something that can natively run on the GPU. With unified memory support, it becomes very fast to run computations on heap allocated memory using the GPU, but implementations also support non-unified memory Vendor support: - https://www.intel.com/content/www/us/en/developer/articles/guide/get-s… - https://rocm.blogs.amd.com/software-tools-optimization/hipstdpar/READM… - https://docs.nvidia.com/hpc-sdk/archive/20.7/pdf/hpc207c++_par_alg.pdf