Twitch Engineering Blog

I suppose! Honestly, I wish it worked like Humble Bundle wherein you have a sliding scale on who gets your money; the platform (Twitch/Amazon), the streamer, and the charity. I'm personally subscribed to PirateSoftware and GDQ because I don't want to see ads and hope a few of my bucks make it to them. Currently it's hard to know[0] how much you're giving the content creator versus Twitch/Amazon. [0] https://blog.twitch.tv/en/2024/01/24/an-update-to-several-streamer-pay…

Talking about other languages... My fav Go feature is the memory ballast - it's real, lots of go programs have command-line to set it up, and no one cares about it anymore https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

I assure you if that code was to be ported to Go 1:1, Go GC would simply crawl to a halt. Write code badly enough and no matter how good hardware and software is, it won't be able to cope at some point. Even a good tool will give, if you beat it down hard enough. For example, you may be interested in this read: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to… Issues like these simply don't happen with GCs in modern JVM implementations or .NET (not saying they are perfect or don't have other shortcomings, but the sheer amount of developer hours invested in tuning and optimizing them far outstrips Go).

The new `runtime.SetMemoryLimit` is actually a pretty big deal and solves a lot of long standing issues regarding the garbage collector not being very configurable, I believe including the one described by the viral "Go memory ballast" article [0]. [0]: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

Related: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

The author of the article mentions that trick and links to https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

> I suspect Go's GC would be particularly well-fit for this kind of application considering it typically runs in <1ms with no exceptionally large pauses (except perhaps in pathological cases). Lol. " 30% of CPU cycles were being spent in function calls related to GC. " https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

Broadly, yes, Go's GC is quite fast and unobtrusive. They've done a good job tuning it for many purposes. On the post's programs though it's quite possible it doesn't come into play at all, as the compiler keeps things on the stack when possible. But like all GCs, there are cases where it has issues. Anecdotally, I've seen multiple cases where "ballast"[1] reduced RPC response times by around 25%. A bit of GC tuning on some tests sped them up by over 10x (due to pathological thrashing - each test would go from just-below the GC threshold to just-over). Etc. Granted, it's usually on not-great code, but the point still stands - if you care about performance, GC cannot be ignored, regardless of how good its marketing is. [1] https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

Sounds like Discord was running gigantic in memory Go caches. Not that it's the wrong approach, but this is a very real-time system-y problem that Rust is designed for. So I understand why this made sense. Obviously rewriting in Rust worked for them, but I'm curious if they tried the Ballast[0] approach, along with smaller cache's per service. That would make the GC scan on the cache smaller while still avoiding some of the Go startup heap issues. I'm not sure if the target heap size ever was accepted by the Go team, but it may have been easier than a full rewrite. [0] https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

i'm curious if allocating a memory ballast would have helped discord in this case? a one line patch in main() with something like ballast := make([]byte, 10<<30) might help their GC spikes, since garbage collection would only trigger when the heap size reaches 20GB, at which point the heap will be reduced back down to 10GB or so. https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

Have you read this - https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

Actually, I think this was meant as a reference to Twitch's experiences with the Go garbage collector: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

It's not "wasted" in the Go case. On modern systems, the ballast is only a few numbers in the virtual memory table as long as you don't touch it, which isn't that hard. The original blog post confirm it: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to… search for "Now onto 2. Won’t this use up 10Gib of my precious RAM?" It may be "nonsensical" but on the grand scale of "things done for memory management's sake" I find it unimpressive. (If 10GB were actually allocated and unavailable, I would find it impressive.)

> Go ... has top notch garbage collection Is ballast[1] still required for certain use cases? [1]: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

The Twitch folks were facing a related situation with the GC. They developed a workaround that they called Ballast, reducing the overall latency and making it more predictable. Quite impressive results [0]. The Go's GC is groundbreaking in several aspects, but probably needs to provide ways to fine-tune it. Posts like this make me believe that one-size-fits-all settings are yet to be seen. [0]: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

Funnily enough, something similar happened at Twitch regarding their API front end written in Go: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

If you want to force it you can call "runtime.GC()" but that's almost always a step in the wrong direction. It is worth it to read and understand: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

The good news are that Go's GC has basically no tunables, so you wouldn't have spent weeks on that. The bad news is that it has basically no tunables so if it's a tuning issue you're either fucked or have to put "tuning" hacks right into the code if you find any that works (e.g. twitch's "memory ballast" to avoid overly aggressive GC runs: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

Go has no proper solution to garbage collector ballast, a hack which businesses you've heard of are using in the wild. see https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to… and https://github.com/golang/go/issues/23044. A golang team member's reasons for not adding a minimum heap size include that it would require more testing each release, and that they might want to have a max heap size hint instead. I posted a comment similar to this one recently, but it seems more relevant here.

10 gb if your twitch. https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to…

The original blog post[1] does mention GOGC, and a list of pitfalls that made them prefer ballast / the pending heap size proposal[2] over it. Are they in error? Or do you find their reasoning unconvincing? [1] https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to… [2] https://github.com/golang/go/issues/23044

Definitely checkout the follow on post as well. It's got simple and concrete steps for making and eliciting forecasts quickly: https://blog.twitch.tv/flash-forecasting-f8384cfe4f6c

>Twitch has shied away from dark patterns that lock out adblocking users. I don't see any big modals or disabled video if I run an adblocker on Twitch. This is no doubt on purpose. I vividly remember this not being the case, whereas (at least in partnered streams) if twitch detected an ad-blocker it would stop the stream and then fill your screen with recommendations to subscribe to the streamer. What ended up happening if I recall correctly was the adblockers would just block the anti-adblock screens so they were rendered ineffective. I'm pretty sure there was other efforts, but I found this on their blog [1] [1] https://blog.twitch.tv/experiment-with-advertising-on-partner-channels…

As a subscriber, I can confirm that it does auto-renew. Also, twitch recently added the option to share the fact that you have paid for another month (by not cancelling) [0]. Taking note of these notifications is possible and would capture some of the resubscription data. [0] - http://blog.twitch.tv/2015/01/subscription-announcements/

> The only way to accomplish this at scale is to build something that is legit better and let the market decide. Anything else is just principled wishful thinking. Classic neo-liberalism BS (pardon my french). Markets are not some natural law written in the atoms, it's a human construction, and we shape it the way we want. Countries can create or destroy markets just with laws, you put a tax here, you put a legal requirement there. That's for example the reason that big american tech companies have been kicked out of South Korea: - https://www.theguardian.com/world/2025/aug/08/south-korea-google-maps-… - https://blog.twitch.tv/en/2023/12/05/an-update-on-twitch-in-korea/ Sure, if there are 2 competing companies that play with the exact set of rules, the mArKeT wIlL deCiDe, but that would be a really stupid decision from any government to not shape the rules in its favor. Europe is slowly waking up to this reality, better late than never I guess. Did the "market decide" that Nvidia chips won't be shipped to China ? Did "the market decide" to put tariffs to get benefits from other countries ? Did "the market decide" to put embargo to Cuba, Iran, Venezuela.. ? Hearing that regulations and laws is "wishful thinking" makes no sense at all. It's more the opposite, it's the only way to shape the markets the way you want to.

I agree with this overall. It seems that C is a language/compiler/toolchain combo that lets you shoot yourself in the foot, but gives you enough access to stuff to be maximially performant in a random (i.e. not specialized) use case. (C++ is in this camp too). Rust is a language/compiler/toolchain combo that tries its best to not let you shoot yourself in the foot by presenting those footguns for you to fix up front, without giving up much of the access to stuff (C++ tried some things in this camp like destructors and smart pointers. GC is also something that's in this camp) Golang is a language/compiler/toolchain combo that lets you shoot yourself in the foot in some circumstances, and does not give you enough access to be maximally performant in every random case, but rather it seems to posit to give you opinions that will get you really, good performance in most random cases (if you need to tune the GC, for instance, you're probably worse off, like here: https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to… Meaning it's making the tradeoffs of upfront complexity vs runtime issues, and Tim Toady vs The One True Way for you, because in the general case those are the most optimized tradeoffs. I don't disagree, especially when the closed-world ecosystem is good enough to not need to reach out to external tools whether those are via cgo or in the build toolchain. In general, I think that if you reach the point where…

I can't speak for why Herman Li, or any other musician, would be banned for DMCA. But, its my understanding that Twitch pays for a live performance license from the major music publishers to enable musicians to stream on their platform [1]. Of course, this didn't work for Herman Li, but I think it at least speaks to their long-term goals for the platform. Additionally, while they say to never play recorded music on stream, unless from a source where the copyrights are taken care of, their specific recommendations even in this "clarification" post are very opaque. Never play recorded music on stream, unless you own all the rights, but you probably don't, if you're not sure turn off your VODs. None of that is consistent with some overarching idea of why people get DMCA'd; would I get DMCA'd for playing recorded music live, but not having VODs? I would guess even Twitch doesn't know; its outside of their control. In short, music on Twitch is REALLY risky, primarily because their rights enforcement is so immature. Its akin to them wildly shooting in the air with a machine gun. YouTube's Content ID is a bit more mature and predictable, so diversifying there may be a good idea. [1] https://blog.twitch.tv/en/2020/11/11/music-related-copyright-claims-an…

V8 still uses STW for mark sweep. The benchmark image on this https://v8.dev/blog/concurrent-marking shows 50+ms pause time, quite bad compared to ~5ms or less in new Java collectors. This might be due to STW move phase though? They don't really explain, but the long pause times show that there's definitely still long STW pauses in V8. For Go, I'm going to be a bad HN user and not read the whole article. Sorry, it's just too long for this time of night. It does appear that my understanding of Go GC is out of date. There's been many improvements in the last couple years. Some strange behavior due to not enough knobs to configure GC, but it appears to have a near constant GC pause? https://blog.twitch.tv/en/2019/04/10/go-memory-ballast-how-i-learnt-to… I'm annoyed that Google doesn't offer much benchmarking results for V8. Huge articles about improvements made with a single benchmark image. And they didn't use standard benchmarks for either so it's unclear what they're even benchmarking. The Go slides you linked include benchmarks from some guys production server he tweeted images of, a bunch of standard benchmarks but they only show % throughout improvement, and no pause times.

I work in a small shop, much "behind the times", and recently tried introducing gRPC. It went okay. The problem is it was a hard sell and for our limited needs, a lot of gRPC's features weren't something we didn't happily solve with REST. Some of the other more complex features that REST didn't support were things we didn't feel we needed. All together, gRPC was a big burden for me to push, with little payoff for people. I think I was the only one that liked it, and even I disliked parts of it. So before pulling the switch too far, I decided to take a step back and assess with benefits we all got from gRPC. Mainly it was Protobuf spec, RPC design (ie not REST verbs and whatnot), and code generation. Because those were the things we all agreed upon, I went with Twirp[0]. It's been a great experience so far. The rollout is just a bit ahead of where our gRPC was, and it's been mostly effortless. The most effortless thing? I've got an (internal) API user switching from the old REST API to the proto RPC spec, and all they're changing is the URL and a few field naming schemes. It's still JSON for them. They're not generating code, they're not doing anything. It all "just works", which is pretty impressive. TL;DR: We love simpler options that let us gain the benefits of Protobufs, without the complexity of gRPC when we don't need it. [0]: https://blog.twitch.tv/twirp-a-sweet-new-rpc-framework-for-go-5f2febbf…

Software for scaling up live-streaming CDN points of presence (POPs) is a pretty crazy domain. For on-demand video, you can think of a CDN as a cache, getting known-ahead-of-time chunks. But what about for live streaming? It's not feasible to stream frame-by-frame directly from your encoding backend to all the viewers of the World Cup, over something like RMTP - you'd want to use a CDN. So typically, you distribute meaty (multi-second) HLS segments as individual video files, or collections of files, to your CDN; once available, they then need to be requested by browsers/mobile clients as a whole segment, over HTTP(S). Works well with existing CDN infrastructure (provided they can handle the write volume and have big enough inbound pipes)... but the huge issue is that the length of the segment plus round-trips is a lower bound on effective latency. And when interactivity is required, multi-second delays can be horrible. https://www.wowza.com/blog/hls-latency-sucks-but-heres-how-to-fix-it is a great writeup. Another overview of the problem, and a proposed solution, is in this excellent article by Twitter here: https://medium.com/@periscopecode/introducing-lhls-media-streaming-eb6… > In HLS live streaming, for instance, the succession of media frames arriving from the broadcaster is normally aggregated into TS segments that are each a few seconds long. Only when a segment is complete can a URL for the segment be added to a live media playlist. The latency issue is…