Modal Blog

I recommend checking out https://modal.com It’s “serverless” and you only pay for the compute you use.

Not in love with axolotl but appreciate the advantages. This is an interesting approach, but you can also finetune easily on providers who wrap axolotl like Replicate [1], Modal [2], or if you want to run the infra, LLM Engine [3]. My only gripe with Helix would be that it's smaller than the above and my org would be peeved about data security. The ability to self host is cool, but too much can go wrong too quickly with plain Docker ML. Would love to see, for example, a `cog` version of the images that we can deploy distributed with more confidence/bravado. [1] https://replicate.com/mistralai/mistral-7b-instruct-v0.2 [2] https://modal.com [3] https://llm-engine.scale.com/

Just use https://modal.com/ :) At Canva I built auto-scaling GPU infra on K8s for model training[1], and it's way too much work and operational expense to be worth building yourself. I went work at Modal because building it properly once and then distributing the solution was going to be just way better and more efficient. 1. https://canvatechblog.com/supporting-gpu-accelerated-machine-learning-…

Start here https://modal.com/docs/examples/vllm_inference or give this a go https://modal.com/docs/examples/opencode_server You get $30 free credits each month on Modal which is enough to play around (i have no affiliation, just think they run a great service)

Personally: love the decorator pattern after I got used to it :) Posted this yesterday as well, but seems like a really nice emerging pythonic way to call out to remote infrastructure (see: Modal[1]). [1]: https://modal.com/docs/examples/hackernews_alerts#defining-the-functio…

I built a web viewer for visualizing Gaussian Splat .ply files generated by Apple's ml-sharp, which converts a single photo into a 3D Gaussian Splat in under a second [1]. Features: - Upload and view .ply file directly in the browser - Multiple camera trajectory animations (rotate, swipe, shake, forward) - Interactive orbit controls (drag to orbit, scroll to zoom, right-drag to pan) - No installation required – runs entirely client-side The original ml-sharp can render the same video trajectories but requires a CUDA GPU. This viewer lets you explore the 3D output on any device with a browser. I also added cloud GPU inference via Modal so you can generate splats without a local GPU (free tier available) [2]. [1] https://github.com/apple/ml-sharp [2] https://modal.com/

There's lots of interesting tooling in my space. Currently open in my browser are https://github.com/liquidmetal-dev/flintlock https://github.com/e2b-dev https://www.daytona.io https://modal.com/ https://render.com/ There's lots of others. I'd love to see a proper comparison somewhere.

right now we're using docker -- we're planning to support modal (https://modal.com/) for remote sandboxes and a "local" mode that might use something like worktrees

You should look at [Modal](https://modal.com/), not affiliated.

Modal's GPU glossary is a good overview about how GPUs work [0]. Karpathy's LLM overview is a good high level overview on LLMs [1]. 3b1b's video (and subsequent videos) on transformers was excellent at helping me understand the math at a high level [2]. This matrix multiplication optimization worklog helped me understand writing better CUDA (not for beginner intro though) [3]. During this process I also asked ChatGPT a lot of questions. I'm definitely open to suggestions about "how to learn" with all the new tools we have. I felt this has not been straightforward to figure out. [0] https://modal.com/gpu-glossary [1] https://www.youtube.com/watch?v=7xTGNNLPyMI [2] https://www.youtube.com/watch?v=wjZofJX0v4M [3] https://siboehm.com/articles/22/CUDA-MMM

I know of https://modal.com/, which I believe is used by Codegen and Cognition. Anecdotally-speaking, I hear that many companies in the LLM agent space roll their own sandbox solutions - I've heard of both Firecracker- and Kubernetes-based implementations.

Yes. We use Modal (https://modal.com/), and are big fans of them. They are very ergonomic for development, and allow us to request GPU instances on demand. Currently, we are running our real-time model on A100s.

Give https://modal.com a spin -- email me at deven [at] modal.com and happy to help you get set up

There are a few tools here like e2b and modal. No affiliation. [1] https://e2b.dev [2] https://modal.com I’m in the same space, but we run the code that devs already write using LLMs to orchestrate them.

I'm going to make a left field suggestion and say Modal Labs if you're mostly Python based: https://modal.com/ It should work with other languages through Docker and their Python API, but I can't speak to the ease of deployment in that case.

I'm curious if people's qualms around abstracting the cloud/network also apply to the https://modal.com product. Differential seems like a similar project at its core, just focused on the Typescript + microservices ecosystem.

> The author apparently does not have any experience in building systems/infrastructure. Well, he built https://modal.com , one of the coolest things since sliced mangoes, and before that https://github.com/spotify/luigi

I wonder if new serverless platforms, like Modal [0], are even simpler than monoliths. I've been reading some books on software engineering, and it seems clear that network calls, multiple services, integration points, etc. all cause complexity. However, I do wonder if things like Modal allow you to focus only on writing business logic. No dealing with deploys, network comms, memory, etc. (No, I don't work at Modal, just wondering "what comes after monolith" -- Modal, Temporal, etc. all seem quite good) [0] https://modal.com/

Yep, that’s right. Modal (https://modal.com) is actually what you’re describing with Python, but Modal is just the compute layer (Unison is going for the whole cloud), and Modal isn’t doing anything to handle version alignment across service boundaries

I saw this a few weeks ago, haven't tested it yet but it looked cool and not so pricey: https://modal.com/ The three musketeers of cloud also allow you to use GPUs but it may be harder to get, especially GCP that needs human whitelisting of your request to use GPU instances.

That's a pretty cool showcase of modal [1]. From a marketing perspective I have to congratulate, this is a really well done way to get people to check out your platform. 1: https://modal.com/

How do you plan to compete against https://autocode.com/ https://pipedream.com/ https://modal.com https://telefunc.com/ ?

Thanks for sharing this interesting project and approach! One suggestion for improvement: Add some more info to your website/GitHub about the need for a provider and which providers are compatible. It took me a bit to figure that out because there was no prominent info about it. Additionally, none of the demos showed a login or authentication part. To me, it seemed like the VMs just came out of nowhere. So at first, I thought "Cloudrouter" was a project/company that gave away free VMs/GPUs (e.g. free tier/trial thing). But that seemed too good to be true. Later, I noticed the e2b.app domain and also then found the note at the bottom of the site that says "Provider selection -> Use E2B provider (default)". Then I got it. However, I should mention that I don't know much about this whole topic. I hadn't heard of E2B or Modal before. Other people might find it more clear. For those that are wondering about this too, you will need to use a provider like https://e2b.dev/ or https://modal.com/ to use this skill, and you pay them based on usage time.

It depends on what you mean by replacing. The integrated developer experience is much better on Vapi, etc. The goal of the Pipecat project is to provide state of the art building blocks if you want to control every part of the multimodal, realtime agent processing flow and tech stack. There are thousands of companies with Pipecat voice agents deployed at scale in production, including some of the world's largest e-commerce, financial services, and healthtech companies. The Smart Turn model benchmarks better than any of the proprietary turn detection models. Companies like Modal have great info about how to build agents with sub-second voice-to-voice latency.[1] Most of the next-generation video avatar companies are building on Pipecat.[2] NVIDIA built the ACE Controller robot operating system on Pipecat.[3] [1] https://modal.com/blog/low-latency-voice-bot - [2] https://lemonslice.com/ = [3] https://github.com/NVIDIA/ace-controller/

For most people, before it makes sense to just buy all the hardware yourself, you probably should be renting GPUs by the hour from the various providers serving that need. On Modal, I think should cost about $72/hr to serve Kimi K2 https://modal.com/pricing Once that's running it can serve the needs of many users/clients simultaneously. It'd be too expensive and underutilized for almost any individual to use regularly, but it's not unreasonable for them to do it in short intervals just to play around with it. And it might actually be reasonable for a small number of students or coworkers to share a $70/hr deployment for ~40hr/week in a lot of cases; in other cases, that $70/hr expense could be shared across a large number of coworkers or product users if they use it somewhat infrequently. So maybe you won't host it at home, but it's actually quite feasible to self-host, and is it ever really worth physically hosting anything at home except as a hobby?

Very cool! > Unless you've actually watched tokens stream at those rates, the numbers are hard to internalize. This is the rendering. I built something similar recently, for the same reason: https://modal.com/llm-almanac/token-timing-simulator. I like that the output rendering is closer to typical UIs -- syntax highlighting in code mode, tool calls, dim-italic reasoning. One feature mine has that the author, or anyone else who vibe codes their own version after seeing this, might like to steal is modeling the distribution of output latencies. My implementation is hacky (log-normal roughyl estimated from p50, p90, and p99 values), but still, when you set those to realistic values, it recreates the "jitter" you see in many LLM UIs. antirez is right that generation tok/s isn't flat as a function of context length, which is a weakness of both simualtors.

Groups of gpu threads are called warps. https://modal.com/gpu-glossary/device-software/warp

Relevant post: https://modal.com/blog/dollars-per-token-considered-harmful (disclaimer: I work with the author)