Netflix Tech Blog

I have nothing against Capital One, but I must say that Netflix published this article in November 2018: https://medium.com/netflix-techblog/netflix-information-security-preve… I would expect folks to be prepared for this kind of vulnerability. Anyway the repo is for educational purposes and it's sarcastic name is, well, saracastic. Cap One is a great technology company.

We also are doing a blog series on how we're approaching notebooks at https://medium.com/netflix-techblog/notebook-innovation-591ee3221233 (the scheduling post is the second in the series). Also I'd recommend listening to talks from the conference with an open mind. There were a lot of great discussions and positive energy around notebooks. Hopefully these help describe a few of the new patterns and tools available in the notebook space.

I get what you're saying about the benefits of PaaS or other managed services, but Netflix serves most content (by volume) off of appliances [1] they ship to ISPs that feed into the network at points-of-presence somewhere close to an edge. They use AWS for logic and compute (including transcoding [3]), but video streams from this globally-distributed CDN [2]. [1] https://openconnect.netflix.com/en/ [2] https://media.netflix.com/en/company-blog/how-netflix-works-with-isps-… [3] https://medium.com/netflix-techblog/high-quality-video-encoding-at-sca…

I don't think time efficiency is the primary concern for the big encoders like Netflix and YouTube. They're more interested in lowering the bitrate while maintaining image quality and are willing to throw compute resources at slower encoders which achieve that. Netflix, for example, does multiple encodes per scene in multiple formats to get the best possible quality for each format for each scene. VP9 outperforms H.264 (libvpx versus x264) for Netflix's use case. Here are some articles from the Netflix TechBlog on their experiences with VP9 and their encoding approach in general: https://medium.com/netflix-techblog/optimized-shot-based-encodes-now-s… https://medium.com/netflix-techblog/more-efficient-mobile-encodes-for-… https://medium.com/netflix-techblog/netflix-downloads-on-android-d79db…

People might find this interesting to compare and constrast to (certainly I noticed parallels with[1]) Netflix's 'serverless' platform, discussed yesterday: https://medium.com/netflix-techblog/developer-experience-lessons-opera… Edit: As a sibling commenter notes this is most eminently doable with e.g. Common Lisp and BEAM (Erlang/Elixir), but more folks are (publicly) attempting this in other environments now (I've experimented with a number of approaches to this the last few years, so I'm trying to keep score - would love to see any comments on other attempts below). [1]: Quote: "At the core of the redesign is a Dynamic Scripting Platform which provides us the ability to inject code into a running Java application at any time. This means we can alter the behavior of the application without a full scale deployment."

Don't forget about the personalization and recommendations technology, which accounts for 75% of viewing. https://medium.com/netflix-techblog/netflix-recommendations-beyond-the…

for people who need more 9s of availability on a single cloud provider, you have to start going multi-region. aws takes region isolation/independence very seriously, and along with geographic independence gives you effectively two entirely independent clouds which just so happen to have the exact same APIs. Some of the (really great) Netflix blog posts[0] have talked about multi-region services. If you do go multi-cloud, I would be wary of picking regions that are located very close to each other. While you'll obviously get independent code and (likely) independent deployments, you're still susceptible to issues correlated with the physical location. [0] https://medium.com/netflix-techblog/global-cloud-active-active-and-bey…

Total content is likely closer to 100 PB. The "25% of consumer bandwidth" is a proxy for how heavily Netflix is used. Take a look at some of their open source projects and their tech blog to get a feel for the problems they solve at their scale. https://netflix.github.io/ https://medium.com/netflix-techblog Some things I find personally interesting: They run a 5000 node Cassandra cluster. They can failover an entire datacenter in 27 minutes.

Here is a blog post from Netflix staff members where they explain the recommendations: https://medium.com/netflix-techblog/netflix-recommendations-beyond-the… . It's from 2012 so things might very well have changed since then. apenwarr talks about this on his blog. He claims Netflix purposely switched from a good recommendation algorithm to a worse one after the transition from their DVD-by-mail days to their streaming offering. See https://apenwarr.ca/log/20190201 (find the section titled "Netflix"). I recommend reading that section (or even the whole article, though the main point of the article is something only tangentially related), but I'll try to summarize. The argument goes that Netflix doesn't need to find the best movie or TV show for you; instead it tries to find one that is just good enough for you, but at the same time good enough for lots of other people too (apparently in your case they fail at that). From their standpoint it doesn't matter if you watch something you really really like instead of something mildly entertaining: Netflix noted that the probability of people cancelling depends on the number of hours they watch Netflix content, not on the quality of that content. Nudging people to select what they watch out of a smaller pool of offerings enables them to save on bandwidth costs because a lot of content can be served out of distributed caches. There's a word for it even, apparently: "satisficing"…

Super cool! It reminds be of Netflix's Flamescope[0] also from Brendan Greggs work at https://medium.com/netflix-techblog/netflix-flamescope-a57ca19d47bb. The reasons heatmaps are "news" is because not everyone has any formal exposure to data/stats curricula. The concept is easy, straight forward, and nearly obvious but unless you're in the position to have datasets to try to understand (such as a class where they have datasets and questions posed for you!) you just don't hear about this stuff. What often happens is that you have a problem you're trying to understand: e.g. "Why does this crash?" or "Why is this slow on Wednesday?", and you use the data you have available to solve it. You don't normally have access to the kinds of data at the resolutions you need for something like heat-=maps to come into play. You're ops not a the application engineer and you just have 50-90-95-99 latency percentile graphs pre-aggregated in minute windows in nagios or graphana and maybe a few more of those for IO, NET, CPU, and thread counts and you're trying to correlate between these to form a hypothesis. If it's important enough AND you have bad luck AND it can't be solved with more hardware, THEN you go deeper. You get to start actually trying to decide what new data is worth collecting ad-hoc, during what time intervals, etc. Only then do you even have data with the resolution where it's worth talking about heatmaps. That's why this is and interesting reintroduction or new to…

The "pressure stall information" looks interesting. It's exponentially weighted moving averages of the fraction of time "some" or "full" (all non-idle) tasks are stalled on each of CPU, memory, and IO. Also a counter of absolute time. [1] Printing this seems like a good addition to "Linux Performance Analysis in 60,000 Milliseconds". [2] More useful that the load average printed by uptime, which hasn't aged well. Linux mixes cpu-bound and io-bound stuff into one bucket [3] which is a terrible idea, and the number of tasks is confusing at best when you have a mix of thread-per-CPU and thread-per-connection stuff running. Looks like it's also exported on a per-cgroups basis, which is great. [1] http://git.cmpxchg.org/cgit.cgi/linux-psi.git/tree/Documentation/accou… [2] https://medium.com/netflix-techblog/linux-performance-analysis-in-60-0… [3] http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html

That really depends on the application, who has access to the keys, how they learn about the keys etc. There isn't a one size fits all solution and that's kinda why eventually consistency is a pain in the ass. The "I want strong consistency but have to use S3 because its the cheapest way to store stuff and its great at serving reads and just otherwise great" option is to maintain an external index that maps between persistent ids (e.g. file paths) and S3 keys. We use this for hosting user content at my company. Our index is in MS SQL of all things ;) The MVP option is to not to delete stuff. :) This can work well in some scenarios (e.g. you want to keep all versions around anyway) but be careful to not dig a hole that's hard to get out of. A more esoteric option is lifecycle based cleanup. This can work well for things like a build artifact cache (where an expired key isn't a big deal because you can regenerate it.) Some of this is discussed in this blog post by Netflix: https://medium.com/netflix-techblog/s3mper-consistency-in-the-cloud-b6… (caveat: s3mper is dead(?) and there is an AWS-native solution for HDFS now.)

Chaos engineering. You have to randomly simulate failure cases of all shapes and sizes to be prepared for failure cases of all shapes and sizes. You don't know if your plan will survive contact until it's survived contact. https://medium.com/netflix-techblog/the-netflix-simian-army-16e57fbab1…

That's not completely accurate. Hardware & OSes have not been designed for heavy multi-tenancy, leading to (among other issues) cache interference. At Netflix [1], we have to do quite a bit of work to undo that. Bigger orgs like Google have spent more than a decade improving the Linux scheduler in their own fork for similar reasons. [1] https://medium.com/netflix-techblog/predictive-cpu-isolation-of-contai…

Hey Everyone, one of the Mantis maintainers here and I have two others over my shoulders here to answer your questions. To elaborate a little Mantis is a platform and ecosystem for building cost-effective operationally focused stream processing applications. We've used it within Netflix for the past several years to provide real-time alerting on SPS [0], real-time anomaly detection [1] and real-time operational outlier detection. [2] [0] https://medium.com/netflix-techblog/sps-the-pulse-of-netflix-streaming… [1] https://www.youtube.com/watch?v=6UwcqiNsZ8U [2] https://medium.com/netflix-techblog/tracking-down-the-villains-outlier…

> and there's no automated tool that will automagically sense the AWS API calls that you perform and then write them for you in a least-privilege manner. Netflix released one two years ago: https://medium.com/netflix-techblog/introducing-aardvark-and-repokid-5…

Netflix also wrote about the drawbacks on this approach https://medium.com/netflix-techblog/netflix-edge-load-balancing-695308…

You might be thinking of Netflix, who developed Hystrix and Ribbon. These are being slowly replaced with other components from Spring Cloud or Resilience4J: https://medium.com/netflix-techblog/netflix-oss-and-spring-boot-coming… Disclosure: I work for Pivotal, but not on Spring.

You could look into functional reactive programming as in rxjs/rxjava etc (https://medium.com/netflix-techblog/reactive-programming-in-the-netfli… or command query responsibility segregation (https://microservices.io/patterns/data/cqrs.html).

This might be of interest: https://medium.com/netflix-techblog/notebook-innovation-591ee3221233

Netflix is in fact getting quite granular with production pipelines, as evidenced by this blog post - https://medium.com/netflix-techblog/studio-production-data-science-646…

Netflix has written about their internal platform as well[1]. Has anyone come across other articles similar to these? I haven't been able to find many outside of these two. 1 - https://medium.com/netflix-techblog/engineering-to-improve-marketing-e…

This is a repost but since you’re interested - Netflix data science is very pro-Jupyter/Python/data analysis packages. https://medium.com/netflix-techblog/notebook-innovation-591ee3221233

Yes - that’s why I used them as an example because they’ve published details about the benefits and how they compare codecs. https://medium.com/netflix-techblog/toward-a-practical-perceptual-vide…

Netflix actually wrote a blog post about this recently. It varies depending on your network connection like video: https://medium.com/netflix-techblog/engineering-a-studio-quality-exper…

Python at Netflix 2013 edition: https://medium.com/netflix-techblog/python-at-netflix-86b6028b3b3e Though it is outside my area of expertise, Node usage continues to increase as well. Different use-cases.

Notably, Netflix doesn't (or least didn't): https://medium.com/netflix-techblog/towards-true-continuous-integratio…

Interesting that bring up the cover art topic. Artwork Personalization at Netflix.. https://medium.com/netflix-techblog/artwork-personalization-c589f074ad… Excerpt: Another challenge is to understand the impact of changing artwork that we show a member for a title between sessions. Does changing artwork reduce recognizability of the title and make it difficult to visually locate the title again, for example if the member thought was interested before but had not yet watched it? Or, does changing the artwork itself lead the member to reconsider it due to an improved selection? Clearly, if we find better artwork to present to a member we should probably use it; but continuous changes can also confuse people.

The Netflix issue isn't really DRM, it's just a way to minimize bandwidth[1]; they found HQ 720 encodes to be sufficient quality for most consumers. I personally find it lacking. [1]: https://medium.com/netflix-techblog/per-title-encode-optimization-7e99…

My understanding is that most commercial CDNs (Limelight, Akamai, etc) are also more cost-effective than serving directly from AWS. However, commercial CDNs don't know a-priori what files are going to be popular, and how to distribute them in different regions and on different types of storage to maximize efficiency. But that's one of the things that Netflix's CDN team can predict. See https://medium.com/netflix-techblog/distributing-content-to-open-conne…