Instagram Engineering

One of my favorite ones is Instagram's: http://instagram-engineering.tumblr.com/

Instagram has used python and django at scale. They have written about it in their engineering blog[0]. Not sure what their current stack is. They did resort to all kinds of tricks. But your overal point still stands. The performance of python is lacking memory and it's embarrassingly slow. I hope python4 will have scripted for developing and compiled for production, like Dart. And a great compiler like Rust. [0]https://instagram-engineering.com/static-analysis-at-scale-an-instagra…

Great post, entertainingly written. Back in 2015, Instagram did a blog post on similar challenges they came across implementing emoji hashtags [1]. Spoiler alert: they programmatically constructed a huge regex to detect them. [1] https://instagram-engineering.com/emojineering-part-ii-implementing-ha…

Follow-up blog post a year later: https://instagram-engineering.com/copy-on-write-friendly-python-garbag…

When they get big enough, they add the static analysis on top of the language to make it more robust. There's a good read by Instagram (all Python) on this: https://instagram-engineering.com/static-analysis-at-scale-an-instagra…

Instagram uses basic base62 with a very standard alphabet ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_ They started using hashing functions from Facebook after they got acquired [0]. This function is used across the whole FB and the main benefit is that it's time sensitive. That means they will sort based on time. First few millions of IDs are just autoincrements and then they swapped to the custom sort. If you are bored, you can dig through who used instagram in the early days. BTW base(62/63/64) is very popular way to encode IDs. Some sites will use much more custom alphabet. My favorite was Vine which used BuzaW7ZmKAqbhMOei5J1nvr6gXHwdpDjITtFUPxQ20E9VY3Ll [0] https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a…

The primary reason is of course encapsulation. Many things you don't want to leave publically accessible. For instance if you're creating a thread safe api, you need to be mindful of references/mutexes/locks which can easily be enforced with wrappers. If you expose the underlying objects directly you could easily dead lock, corrupt memory, create race conditions, etc. It also greatly simplifies testing and guarantees/provability since you can guard against access patterns or certain inputs or states. Personally, I have run into enough horror caused by mutating a global state that I write code which is quite strict about what it exposes. If you're a python dev, I'd suggest reading this (posted on hn recently but I can't find the hn link currently): https://instagram-engineering.com/python-at-scale-strict-modules-c0bb9…

I had asked a similar question on Quora [1] few years back. Here is the summary: - Facebook: https://www.facebook.com/Engineering?sk=notes - Google: http://google-engtools.blogspot.com/ - Twitter: http://engineering.twitter.com/ - Linkedin: http://engineering.linkedin.com/blog - Dropbox: https://tech.dropbox.com/ - Instagram: http://instagram-engineering.tumblr.com/ - Netflix: http://techblog.netflix.com/ - Flickr: http://code.flickr.com - Etsy: http://codeascraft.etsy.com/ - Yelp: http://engineeringblog.yelp.com/ - Heroku: https://blog.heroku.com/ - Airbnb: http://nerds.airbnb.com/ - Bitly: http://word.bitly.com/ - Evernote: https://blog.evernote.com/tech/ - Highscalability: http://highscalability.com/ [1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-bes…

I remember reading about a tool Instagram used for SSH management, but that might not be a "best practice". (http://instagram-engineering.tumblr.com/post/11399488246/simplifying-e…

And Esty: https://codeascraft.com/ Netflix: http://techblog.netflix.com/ Facebook: https://code.facebook.com/posts/ Instagram: https://instagram-engineering.tumblr.com/ Pinterest: http://engineering.pinterest.com/ These are all excellent blogs to keep up on in your free time.

Oddly enough, Tumblr doesn't seem to support HTTPS for subdomains, so Instagram's is just http://instagram-engineering.tumblr.com/

Similarly, Instagram faced similar issues and developed Neti for the task. https://github.com/Instagram/neti And the blog article: http://instagram-engineering.tumblr.com/post/89992572022/migrating-aws… Now, Instgram's path was AWS EC2-Classic -> AWS VPC -> FB, however it is still relevant.

Instagram had a nice post on how they did their PK generation/sharding a couple years back. Still a good read, http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-… Previously discuss here, http://news.ycombinator.com/item?id=3058327

Actually don't use UUIDs. We use something very similar to what Instagram does with Postgres : http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-… This allows far greater flexibility with regard to sharding and scale. Bigint with a function generated nextval will sustain up to 9223372036854775807. If you need more than that, your next option is Numeric, but I don't think we'll exhaust bigint any time soon.

I enjoyed this article and also found a link to this one which I found equally interesting: http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-… I worked with the Flickr-style ticket DB id setup at Etsy, and while it was lovely once it was all set up, it's way more complicated (requiring two dedicated servers and a lot of software and operations stuff.) The solution outlined by instagram of just having a clever schema layout and stored procedure that safely allocates IDs locally on each logical shard is elegant and I'm having a hard time blowing holes in it.

"Now if you need either [Sharding or JSON] (very common) PostgreSQL is a non-starter." Instagram used sharded postgres, and they certainly started up. http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-… Native JSON support is coming in 9.2 (being released very soon) and will rapidly improve. PostgreSQL is being used by startups and major enterprises alike to great effect. If you have any interest in it, I encourage you to dive in to some real problems and see how postgres can help you solve them. Yes, your criticisms may be valid, and postgres is always improving. But if you keep an open mind and actually try to solve real problems with it using all of the tools it has to offer, you may be pleasantly surprised.

Color is really, really hard. I love the clarity of black-and-white imaging, but the thing that keeps me coming back to it more than anything else is the ability to successfully render my intended image on more electronic platforms. When working in color, I either gravitate toward images with unambiguous color or accept the fact that only those with calibrated screens will see what I intended to create. Moreover, those calibrated screens need to be calibrated to the same reference I'm using, or they'll see something different. Even with prints, without controlling the viewer's ambient light, you'll also get different perception. It is really hard. Agreed that debugging is hard, too. I've recently come across the notion of 'canary images' [1] , which can make some of the debugging effort into binary search. [1] https://instagram-engineering.com/bringing-wide-color-to-instagram-5a5…

"Ideas that diverged too much from the existing successful tech had a huge obstacle in their way: they had to recreate all of the existing solutions in their new model.” You’re so right! There are great examples of companies that preserved the existing ecosystem of solutions (Facebook with PHP, Microsoft with C++, Google with Java and C++, maybe also Amazon with Perl 5 and C++). There are also great examples of successful companies that in a certain sense did or had to do things in an entirely new model (WhatsApp with the Open Telecom Platform, F5 Networks with their data center FPGA-powered hardware load balancers, Tesla and SpaceX, and so on). There are also great examples of companies that did both, where they preserved an existing ecosystem but also added another ecosystem on top to fix flaws (NeXT Computer with Smalltalk and C, Stripe with the modern version of that as Ruby and Go, Twitter with Ruby and Java, so on). Microsoft is the best example of a company that can succeed even with what seems like a crazy choice. Almost everything being written in C++ sounds insane to the point of being business suicidal, but they worked hard enough to make it work. If Microsoft can succeed in the ways that they have with C++, and if Facebook could succeed for as long as they did with PHP, and Instagram can allegedly succeed with Django[1], then anyone can succeed with anything as long as you can endure the stress that a seemingly peculiar decision might cause you. And those…

Django is the closest I can think of. Many unicorns built on Django. Both Rails and Django are horribly slow though, so once you get to some critical scale you gotta start doing some real weird stuff like Instagram did with turning off Python's GC [1], etc. [1] https://instagram-engineering.com/copy-on-write-friendly-python-garbag…

For those wondering how to make all of this HA and production-worthy, this talk was a good watch: https://www.youtube.com/watch?v=Mw8O9d0ez7E Explains how to get HA for both the coordinator nodes and the storage nodes. While I haven't put this into practice myself, I'd be interested in hearing how things like resharding/scaling play out, similar to some of the recent posts on postgres (re)sharding: https://news.ycombinator.com/item?id=43364668 https://www.notion.com/blog/sharding-postgres-at-notion https://www.notion.com/blog/the-great-re-shard https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a…

Maybe this one for Instagram? https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a…

This is from several years ago (2017), but this has very similar vibe as Instagram disabling Python GC - https://instagram-engineering.com/dismissing-python-garbage-collection…

It looks like figma mostly just implemented the Instagram sharding paper, and then did the obvious next level of semi-intelligent yet delicate and fragile query routing. Definitely a technically challenging and overall fun / satisfying engineering exercise! https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a…

If you know which type of uuid you have (v1, v4, etc) then you can take a look at how many bits of randomness it has, how many total items you have, and compute the probability of a collision if you just take a subset of the bits and use that as an ID. In theory it's definitely possible. The 128 bits you get in a UUID is a LOT of randomness for an identifier. Postgres BIGINTs are just 64 bits. Instagram's sharded IDs are just 64 bits. (See below.) You can test it. If you're using uuidv4 (which is 100% random bits, minus a few for the version), you could make a new column in your table in Snowflake, populate it with the first 64 random bits of your existing uuid column, then see if you have any collisions. https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a…

I am using a variant of SnowflakeId [^1] in order to have 64 bit keys too. It's similar to UUIDv7 (it leaks the creation time), but it's not an issue for me. So I am able to have a single 64 bit key, which can easily be formatted into a small string for user-facing urls. [^1]: https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a…

I was wrong thruflo pasted the correct link You are probably right. The Instagram Engineering blog [0] points to the Fabric documentation on Read the Docs [1] which is empty. "Read the docs" links to the GitHub repo [2] but it's 404 and even the GitHub organization zwsyff888 is gone. [0] https://instagram-engineering.com/what-powers-instagram-hundreds-of-in… [1] https://fabric.readthedocs.io [2] https://github.com/zwsyff888/fabric

Instagram engineers themself wrote a bit about their backend infrastructure. One of the more important topics was how they shard the data [0] and this is also linked to in this blog post. [0]: https://instagram-engineering.com/sharding-ids-at-instagram-1cf5a71e5a…

Source: https://instagram-engineering.com/what-powers-instagram-hundreds-of-in…

Instagram's backend is a Django monolith "with several million lines of code and a few thousand Django endpoints" as of 2019, and the same architecture was used for Threads! They now run Django on a JIT-enabled fork of CPython, and leverage a ton of internal tooling for static analysis and strong typing. https://instagram-engineering.com/types-for-python-http-apis-an-instag… https://engineering.fb.com/2023/09/07/culture/threads-inside-story-met… https://engineering.fb.com/2022/05/02/open-source/cinder-jits-instagra… https://engineering.fb.com/?s=django Django is also excellent for e-commerce with projects like https://github.com/saleor/saleor - or you can roll your own e-commerce system quite easily, which we've done at my travel startup.

Unfortunately, python garbage collector messes up copy on write. Here’s a blog from instagram on how they fixed it - https://instagram-engineering.com/copy-on-write-friendly-python-garbag…