Apache TVM Blog

The model is compiled for apple silicon/metal with 4-bit quantization using mlc-llm https://mlc.ai/mlc-llm/ which uses TVM Unity https://tvm.apache.org/ under the hood.

How did AITemplate performance to state of art inference engine like tvm or onnx runtime ? Did AITemplate optimize/quantify network? Edit: link for TVM https://tvm.apache.org/

As mentioned in sibling comments, Torch is indeed the glue in this implementation. Other glues are TVM[0] and ONNX[1] These just cover the neural net though, and there is lots of surrounding code and pre-/post-processing that isn't covered by these systems. For models on Replicate, we use Docker, packaged with Cog for this stuff.[2] Unfortunately Docker doesn't run natively on Mac, so if we want to use the Mac's GPU, we can't use Docker. I wish there was a good container system for Mac. Even better if it were something that spanned both Mac and Linux. (Not as far-fetched as it seems... I used to work at Docker and spent a bit of time looking into this...) [0] https://tvm.apache.org/ [1] https://onnx.ai/ [2] https://github.com/replicate/cog

In an ideal world, absolutely! It's a hard problem and there are many attempts to make that happen automatically including polyhedral optimization (Polly[1]) and tensor compiler libraries (XLA[2] and TVM[3]). I work on a project called LoopTool[4] which is researching ways to dramatically reduce the representations of the other projects to simplify optimization scope. [1] https://polly.llvm.org [2] https://www.tensorflow.org/xla [3] https://tvm.apache.org [4] https://github.com/facebookresearch/loop_tool

> Is anyone using ML in compilers yet? Yes :) Check out TVM for something mainstream: http://tvm.apache.org/ And LibNC for something esoteric (but elegant): https://bellard.org/libnc/

Apache TVM [1], which is a tensor compiler stack focussed on deep neural networks. It is mostly written in C++ and Python, integrates external libraries like LLVM, OpenCL, CUDA, etc. [1] https://tvm.apache.org/

So, just wondering from a practical perspective, how do you train/serve models from these? I doubt they play nice with e.g. PyTorch or TF. Maybe they're meant to be used with a DL compiler like TVM [0]? [0] https://tvm.apache.org/

It is compiled to XLA so should be a lot faster then pure PyTorch but probably will be slower then TVM (https://tvm.apache.org/) i can prepare some benchmarks in next few days if u are interested :)

see also tvm https://tvm.apache.org/

The sibling comment has already touched on the compiler side of things, so let me touch on the rest: > A stronger culture of functional programming could make your code faster and easier to understand. It's often (though not always) easier for tooling to optimize pure functions over immutable data structures (and arrays). Arrays are currently mutable, and there is too much emphasis on mutating functions. Yes and no. Immutable data structures shine in 2 scenarios: 1. Small types that can be represented with a couple of machine words. Julia supports this already through via stack allocated immutable struct types and packages like StaticArrays [1] 2. Persistent data structures as found in most FP languages. Note how neither of these capture the large, (semi-)contiguous array types used for most numerical computing. These arrays are only "easier to optimize" if one has a Sufficiently Smart Compiler to work with. Here we don't even need to talk about Julia: the reason even Numba kernels in Python land are written in a mutating style is because such a compiler does not exist. You may be able to define something for a limited subset of programs like TensorFlow does, but the moment you step outside that small closed world you're back to needing mutation and loops to get a reasonable level of performance. What's more, the fancy ML graph compiler (as well as Numpy and vectorized R) is dispatching to C++/Fortran/CUDA routines that, not surprisingly, are also loop-heavy and…

The data that is out there is reasonably promising with WebGPU already in use in some production ML inference engines. TVM of course is way ahead of the curve as usual - https://tvm.apache.org/2020/05/14/compiling-machine-learning-to-webass… though this post is quite old now. It's still early days for pushing compute use cases to WebGPU (OctoML being super early notwithstanding). There's a small matmul in the examples directory but it only has the most basic tiling optimizations. One of my goals the next few weeks is porting the transformer block kernels from llm.c - I think that will flesh out the picture far better. If there's interest, happy to collaborate + could potentially do a writeup if there's enough interest. There's always some tradeoffs that comes with portability, but part of my goal with gpu.cpp is to create a scaffold to experiment and see how far we can push portable GPU performance.

Yeah! I've started to look some WebGPU compute applications with other students in my group and I think there could be some cool use cases, like the ones you mention. It sounds a bit odd, but yeah WebGPU on native (by directly using Dawn, or wgpu-rs) is actually pretty compelling as a cross-platform low-level graphics API. What's really cool is that compute and rendering using WebGPU can get near-native level performance. So a lot of scientific applications (which typically rely on more FLOPs/parallel processing) can be implemented in WebGPU compute without sacrificing much performance. I'm not sure how many simulations would be ported to WebGPU, since they usually end up targeting large scale HPC systems and CUDA, but for visualization applications I think the use case is pretty compelling, especially for portability and ease of distribution. On the compute side, I implemented a data-parallel Marching Cubes example: https://github.com/Twinklebear/webgpu-experiments , and found the performance is on par with my native Vulkan version. You can try it out here: https://www.willusher.io/webgpu-experiments/marching_cubes.html#vol=Sk… . There is a pretty high first-run overhead, but try moving the slider around some to see the extraction performance after that. WebGPU for parallel compute, combined with WebASM for serial code (or just easily porting older native libs), will make the browser a lot more capable for compute heavy applications. You could also combine these more…