Tim Dettmers

You really don't need NVLink, you won't saturate the PCIe lanes on a modern motherboard with dual 3090s. Tim Dettmers amazing GPU blog post posits NVLink doesn't start to become useful until you are at 128+ GPUs https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/#What_i…

It really depends on what you're trying to do. This is sorta _the_ guide on GPUs for DL and has a great decision tree https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ Personally, I'm limited to an RTX 2080 for my personal projects at the moment, and I find the constraint pretty rewarding. It forces me to find alternatives to the huge models, and you'd be surprised what you can eek out when you pour in the time to tweak models. Of course, good data is also pinnacle.

https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ This is the best one imo.

I built a DIY PC with used GPUs (2x RTX 3090) for around 2300€ earlier this year. You can probably do it for slightly less now (i also added 128GB RAM and NVLink). You can generate text with >10 tok/s with that setup. Make sure to get a PSU with more than 1000W. Air cooling is a challenge, but it's possible. Recommended reading: Tim Dettmer's guide https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

If you have a lot of money (but not H100/A100 money), get 4090s as they're currently the best bang for your buck on the CUDA side (according to George Hotz). If broke, get multiple second hand 3090s. https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/. If unwilling to spend any money at all and just want to play around with llama70b, look into petals https://github.com/bigscience-workshop/petals

If you want to learn about which GPUs make sense to buy or rent for various ML workload, i highly recommend this Tim Dettmers article: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

Tim Dettmers [0] has been updating the linked article for a while now. It is a great primer -> deep dive into the world of GPU's for ML/DL tasks. I don't need 4x RTX 4090's but I really want them. As it stands now, I use an old M40 that I modified for water cooling. The only tricky part was configuring Win10 to run the card in WDDM mode as opposed to TCC mode without onboard graphics. To be honest, I am not entirely sure how it works as the pass-through is an older quadro card. How do I justify the purchase of DGX-1 as a hobby programmer? Call it my mid-life crisis purchase? I mean, its not that much more than a mid-range 2023 Corvette Stingray. [0] https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

This post[0] by Tim Dettmers should help. Reading through the article, it seems like he covers a lot of ground and lays out choices quite well. tl;dr is that only Nvidia is your best bet for local computing power, stay away from AMD & Intel for now. [0] - https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/

See also Tim Detter's fantastic post on GPU performance (which doesn't use benchmarks for the latest cards but instead calculates performance with a model): https://timdettmers.com/2020/09/07/which-gpu-for-deep-learning/ HN Discussion: https://news.ycombinator.com/item?id=24400603

Here's a writeup on tpu vs got throughput: https://timdettmers.com/2018/10/17/tpus-vs-gpus-for-transformers-bert/ The underlying principle is that even if the silicon isn't strictly faster, you can shove a dumb amount of data through with the right architecture. And efficient training with data parallelism means that it's a worthwhile strategy.

There's an open-source effort that's begun since early last year: https://github.com/tensorflow/tensorflow/issues/22 Though that will take another year at least before it comes to fruition. What was your GPU budget? If you were able to run them CPU-based I think a mid-range Pascal GPU would do great. You can sell it later for at least half the price as well. This might be useful: http://timdettmers.com/2014/08/14/which-gpu-for-deep-learning/

This is a tough question to answer, because it depends a lot on what you want to do! One way to approach it may be to look at what models you want to run and check the amount of VRAM they need. A back-of-the-napkin method taken from here[0] is: VRAM (GB) = 1.2 * number of parameters (in billions) * bits per parameter / 8 The 1.2 is just an estimation factor to account for the VRAM needed for things that aren't model parameters. Because quantization is often nearly free in terms of output quality, you should usually look for quantized versions. For example, Llama 3.2 uses 16-bit parameters but has a 4-bit quantized version, and looking at the formula above you can see that will allow you to run a 4x larger model. Having enough VRAM will allow you to run a model, but performance is dependent on a lot of other factors. For a much deeper dive into how all of this works along with price/dollar recommendations (though from last year!), Tim Dettmers wrote this excellent article: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ Worth mentioning for the benefit of those who don't want to buy a GPU: there are also models which have been converted to run on CPU. [0] https://blog.runpod.io/understanding-vram-and-how-much-your-llm-needs/

nvidia specific, best write up I know: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ Across vendors, generally, Nvidia still dominates currently. People are adding more support into ML libraries for other vendors via (second-class imo) alternate backends but expect to be patient if you're waiting for the day when there is healthy competition. IMO, I'd say: if you can save up for it, get a 4090; if you can save up for half a 4090, get a 3090 - seen many going for 600-800 now. If you can save up for half a 3090, I'm not sure - depends on if you prefer speed or VRAM. If it were me, I'd pick more VRAM first. re: compute capability, you can see here: - which GPUs have what cc: https://developer.nvidia.com/cuda-gpus - what cc comes with what features: https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#featu… I think the main qualitative change (beyond bigger numbers in the spec) for an enduser of machine learning libraries from 8.6 -> 8.9 (ie 3090 -> 4090) is this line: > 4 mixed-precision Fourth-Generation Tensor Cores supporting fp8, fp16, __nv_bfloat16, tf32, sub-byte and fp64 for compute capability 8.9 (see Warp matrix functions for details) ie new precisions will be builtin to eg pytorch with hw-level/tensor core support edit: btw you probably ought to stick to a consumer gpu (ie not professional) if you want it to be generally versatile while also easy to use at home.

I think Arubis didn't push the point hard enough. Nvidia is a long way ahead of AMD in AI/ML hardware capability, from my very limited experience. I got a Stable Diffusion thing up and running locally on a relatively old AMD RX 580 with 8GB of memory, and default settings of 512x512 with 10 (I think) passes, and it takes about 10 minutes to generate an image. On a similarly aged Nvidia 1060 with 6GB of memory that a colleague of mine uses for their local Stable Diffusion instance, it takes about 20 seconds to generate an image. I don't know how apples and oranges this comparison is, my OS is Linux and his is Windows, for example, but 10 minutes versus 20 seconds is, for me, a massively disappointing difference. I'm sure the gap has decreased in the intervening 6-7 years, but I don't know how much - such that I'd be opting for a second hand Nvidia over a new AMD (and I philosophically prefer AMD). This may be a worthwhile read if you're looking to squeeze the most performance: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

For a general guide, I recommend: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ There's a subreddit r/LocalLLaMA that seems like the most active community focused on self-hosting LLMs. Here's a recent discussion on hardware: https://www.reddit.com/r/LocalLLaMA/comments/12lynw8/is_anyone_else_su… If you're looking just for local inference, you're best bet is probably to buy a consumer GPU w/ 24GB of RAM (3090 is fine, 4090 more performance potential), which can fit a 30B parameter 4-bit quantized model that can probably be fine-tuned to ChatGPT (3.5) level quality. If not, then you can probably add a second card later on. Alternatively, if you have an Apple Silicon Mac, llama.cpp performs surprisingly well, it's easy to try for free: https://github.com/ggerganov/llama.cpp Current AMD consumer cards have terrible software support and IMO isn't really an option. On Windows you might be able to use SHARK or DirectML ports, but nothing will run out of the box. ROCm still has no RDNA3 support (supposedly coming w/ 5.5 but no release date announced) and it's unclear how well it'll work - basically, unless you would rather be fighting w/ hardware than playing around w/ ML, it's probably best to avoid (the older RDNA cards also don't have tensor cores, so perf would be hobbled even if you could get things running. Lots of software has been written w/ CUDA-only in mind).

Thanks for extensive answer. Do you have a few of these cutting-edge algorithmic advances papers in mind, could you list them? I guess I got too pessimistic because of things like "emergent features" [1] / "grokking" [2] that seems to happen only with a lot of compute, and also the fact that the original (vanilla) transformer architecture remains (one of) the best, despite many additional ideas and "advances" (but that is only evident at large scale) [3]. Because of the points above, it's really hard for me, as a non-expert, to assess which papers are true advancements, and which were only published in pursuit of vanity metrics (e.g. publication counts) but actually represent overfit/cherry-picked results rather than robust progress. [1] https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/ [2] https://www.lesswrong.com/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interp… [3] https://twitter.com/YiTayML/status/1551657355036676096

Nick Bostrom and his ilk are exhausting. I really don't think that the singularity / intelligence explosion is going to happen in the way that they think; I don't think ASIs will work the way they think they do; and I don't think we're anywhere close to AGI, even if some of the ideas happening right now are pretty cool. Some good sources on the matter in general: https://kevinkelly.substack.com/p/the-singularity-is-always-near https://idlewords.com/talks/superintelligence.htm https://spectrum.ieee.org/rupturing-the-nanotech-rapture https://timdettmers.com/2025/12/10/why-agi-will-not-happen/

I don't know whether Ed Zitron is telling the truth. I do know that Suleyman, Altman, and Amodei have lied, lied, and lied repeatedly, whether intentional or not. For that matter, I do not believe AGI will happen in our lifetimes. https://timdettmers.com/2025/12/10/why-agi-will-not-happen/

Tim Dettmers had an interesting take on this [1]. Fundamentally, the philosophy is different. >China’s philosophy is different. They believe model capabilities do not matter as much as application. What matters is how you use AI. https://timdettmers.com/2025/12/10/why-agi-will-not-happen/

This is a good counter in my view to the singularity argument: https://timdettmers.com/2025/12/10/why-agi-will-not-happen/ I think if we obtain relevant-scale quantum computers, and/or other compute paradigms, we might get a limited intelligence explosion -- for a while. Because computation is physical, with all the limits thereof. The physics of pushing electrons through wires is not as nonlinear in gain as it used to be. Getting this across to people who only think in terms of the abstract digital world and not the non-digital world of actual physics is always challenging, however.

Check these out: - https://www.hyperstack.cloud/technical-resources/tutorials/how-to-choo… - https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ - https://preview.redd.it/the-llm-gpu-buying-guide-august-2023-v0-4nve5p…

Multi-GPU in desktop chassis gets crazy pretty quickly. If you don't care about aesthetics and can figure out both the power delivery and PCI-E lane situation, https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/ has an example that will make a Bitcoin mining rig look clean. Water cooling can get you down to 2x slot height, with all of the trouble involved in water cooling. NVIDIA really segmented the market quite well. Gamers hate blower cards, but they are the right physical dimensions to make multi-GPU work well, and they are exclusively on the workstation cards.

This blog post might shed some light on the matter. If I'm understanding it correctly, it claims there are emergent features on the LLM weights that make it easier to "compress" the floats into smaller bits without losing much precision. https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/#Mix… Note that 2 bit quantization is generally regarded as too aggressive. Generally 4bits+ achieves a good tradeoff, see eg. https://arxiv.org/abs/2212.09720

You can buy multiple 4090s for that money and will get real GPUs including tensor cores. Still relevant it seems: https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

There’s a 2023 article by the same author here it looks like https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

Either you only use the GPU sporadically or your math is very different from Tim Dettmers` [1]: "The break-even point for a desktop vs a cloud instance at 15% utilization (you use the cloud instance 15% of time during the day), would be about 300 days ($2,311 vs $2,270):" And that was written before the current GPU crunch and assumes availability in the cloud, which currently is not a given at all. [1] https://timdettmers.com/2023/01/30/which-gpu-for-deep-learning/

Some very interesting discussion of outlier features and quantization: https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/ * Outlier values are used to prune values. * Transformers seem to undergo a "phase shift" in how outlier features are treated around 6.7B parameters. This could complicate research on removing them. Maybe you and Tim Dettmers would have a lot to talk about :)

One of the most significant quantization papers of the last year [1] found precisely that these outliers only start occuring with LLMs at 6.7B parameters and above. One of the most important keys to the success of deep learning in the last couple years has been the fact that emergent features exist after certain scales, so I wouldn't be too quick to dismiss things that don't help at smaller scales, nor would I be certain that all the tricks that help in small data/parameter regimes will necessarily help in larger models. Unfortunately! [1] https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/

Here is an article that explains more about the outliers that emerge in large transformer models, which is what this modified softmax is being proposed to fix: https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/ The fact that these only emerge in larger models is likely one reason the author hasn't actually tried it.

See also the blog post. https://timdettmers.com/2022/08/17/llm-int8-and-emergent-features/ I personally found most interesting of this work the emergent sparse behavior starting around 2B parameters but not before. That suggest that the model is operating in a different mode at such model size or larger.