EXO Labs Blog

The author mentions it but I want to expand on it: Apple is a seriously good option here, specifically the M4 Mac Mini. What makes Apple attractive is (as the author mentions) that RAM is shared between main and video RAM whereas NVidia is quite intentionally segmenting the market and charging huge premiums for high VRAM cards. Here are some options: 1. Base $599 Mac Mini: 16GB of RAM. Stocked in store. 2. $999 Mac Mini: 24GB of RAM. Stocked in store. 3. Add RAM to either of the above up to 32GB. It's not cheap at $200/8GB but you can buy a Mac Mini with 32GB of shared RAM for $999, substantially cheaper than the author's PC build but less storage (although you can upgrade that too). 4. M4 Pro: $1399 w/ 24GB of RAM. Stocked in store. You can customize this all the way to 64GB of RAM for +$600 so $1999 in total. That is amazing value for this kind of workload. 5. The Mac Studio is really the ultimate option. Way more cores and you can go all the way to 192GB of unified memory (for a $6000 machine). The problem here is that the Mac Studio is old, still on the M2 architecture. An M4 Ultra update is expected sometime this year, possibly late this year. 6. You can get into clustering these (eg [1]). 7. There are various Macbook Pro options, the highest of which is a 16" Mackbook Pro with 128GB of unified memory for $4999. But the main takeaway is the M4 Mac Mini is fantastic value. Some more random thoughts: - Some Mac Minis have Thunderbolt 5 ("TB5"), which is up to…

They're probably referencing this article: https://blog.exolabs.net/nvidia-dgx-spark/

I'd be interested in seeing numbers that split out the speed of reading input (aka prefill) and the speed of generating output (aka decode). Those numbers are usually different and I remember from this Exo article that they could be quite radically different on Mac hardware: https://blog.exolabs.net/nvidia-dgx-spark/

It was about this use case: https://blog.exolabs.net/nvidia-dgx-spark/

Even an M3 Ultra won't put up similar GPU compute to a DGX Spark: https://blog.exolabs.net/nvidia-dgx-spark/ Fill up the memory with a large model, and most of your memory bandwidth will be waiting on compute shaders. Seems like a waste of $5,000 but you do you.

For training, yes, you will need to share the parameters (i.e., weights and bias); the number is huge. But for inference, you don't need that much high bandwidth to run it in a distributed manner. According to the author of Exo https://blog.exolabs.net/day-1/: > When Shard A finishes processing its layers, it produces an activation that gets passed to Shard B over whatever network connection is available. In general these activations are actually quite small - for Llama 3.2 3B they are less than 4KB. They scale approximately linearly with the size of the layers. Therefore the bottleneck here is generally the latency between devices, not the bandwidth (a common misconception). I think that makes sense because the activations are the numbers coming out of the whole neuron network (or part of it). Compared to the number of parameters, it's not at the same magnitude.

Check out exo labs blog. https://blog.exolabs.net/day-2/