I
Instagram Engineering
engineering.instagram.com · Company blog · on HN since 2015
Instagram's engineering team writes about scaling, Python and Django, infrastructure, and mobile and open-source tools.
What HN saysHN lists it among respected engineering blogs, and commenters cite it for Python at scale, PgQ cache invalidation and Cassandra latency work.
51HN people shared it
17linked it in comments
40stories submitted
3.0Kstory points
9 years agolast shared
2015people per year2026
Most-shared posts
- 1Bringing Wide Color to Instagram6 people · 396 points · first shared 2017 · 117 comments on HN
- 2Dismissing Python Garbage Collection at Instagram6 people · 265 points · first shared 2017 · 85 comments on HN
- 3Open-sourcing MonkeyType – Let your Python code type-hint itself5 people · 443 points · first shared 2017 · 229 comments on HN
- 4Web Service Efficiency at Instagram with Python5 people · 200 points · first shared 2016 · 51 comments on HN
- 5Scaling our infrastructure to multiple data centers4 people · 49 points · first shared 2015 · 11 comments on HN
- 6Copy-on-write friendly Python garbage collection3 people · 198 points · first shared 2017 · 52 comments on HN
- 7Sharding and IDs at Instagram3 people · 2 points · first shared 2017 · discussion on HN
- 8React Native at Instagram2 people · 589 points · first shared 2017 · 233 comments on HN
- 9Open-sourcing a 10x reduction in Apache Cassandra tail latency2 people · 408 points · first shared 2018 · 164 comments on HN
- 10Flexible Feature Control at Instagram2 people · 51 points · first shared 2015 · 11 comments on HN
- 11Instagram and Android: Four Years Later2 people · 5 points · first shared 2016 · 2 comments on HN
- 12Homepage2 people · first shared 2016
- 13Continuous Deployment at Instagram1 people · 239 points · first shared 2016 · 91 comments on HN
- 14Performance and Usage at Instagram1 people · 87 points · first shared 2016 · 29 comments on HN
- 15Lessons learned with 3D Touch1 people · 51 points · first shared 2015 · 31 comments on HN
What HN people wrote when they linked it
altern8tif · 2017-04-21
Great design + engineering teams = Awesome products
Instagram (https://engineering.instagram.com/)
Stripe (https://stripe.com/blog/engineering)
Airbnb (http://nerds.airbnb.com/)
dimitri-gnidash · 2017-01-19
Python is a wonderful language where the speed of development and maintainability is concerned.
Django is a great framework for getting your web application and/or APIs built. The library support is also phenomenal (looking at Django Rest Framework for API development)
It is also unlikely you will outgrow Python/Django stack as Instagram and Pintereset are doing fine. https://engineering.instagram.com/what-powers-instagram-hundreds-of-in…
Rarely does the stack matter to startup success, why not pick up Steve Blank's
The Startup Owner's Manual about building THE RIGHT thing? Building the wrong product is the number one killer of startups why not focus there?
tedmiston · 2016-08-02
I have met some members of Instagram's engineering team at Python conferences. They also run a dedicated engineering blog (https://engineering.instagram.com).
Despite the fact that they use an old version of Django, and the product has not exactly changed drastically since it became popular, they don't seem jaded.
Sometimes it's fun to build your own version of something regardless of what exists in the rest of the world. I'm sure nuances will become more prominent in time.
rbranson · 2016-04-15
There is also PgQ from the https://wiki.postgresql.org/wiki/SkyTools package, which is fantastic and mature. It's like Kafka inside your PostgreSQL. When I was at Instagram, we implemented a multi-region cache invalidation system using PgQ that took advantage of atomicity between PgQ and writes to the source-of-truth data table. http://engineering.instagram.com/posts/548723638608102/
mspoof · 2018-03-12
Rocks Storage engine in Cassandra which claims to reduce the GC stalls
https://engineering.instagram.com/open-sourcing-a-10x-reduction-in-apa…
rojoca · 2018-02-13
Also the source(?) of this change from instagram: https://engineering.instagram.com/copy-on-write-friendly-python-garbag…
bpicolo · 2018-01-01
Memory is typically the scaling bottleneck for Python workers for web servers. One way to cut back drastically on this is to load as much as possible in startup and then fork for request serving processes. The problem is that python refcounting causes a lot of copy-on-writes for data that's really just used as read-only data.
This change allows you to run more workers with less memory.
Instagram has had a few good posts about how they've approached this problem, here's another:
https://engineering.instagram.com/dismissing-python-garbage-collection…
grzm · 2017-10-27
Trivially, a UUID is wider (128 bits as opposed to 64). You can choose to generate your UUIDs using a Instragram[0]-like strategy so you can, for example, benefit from temporal/spatial locality in the indexes. There may be performance consequences of using the wider datatype, but I suspect this could be overwhelmed by other application concerns if you're dealing with data that benefits from the wider type.
[0] https://engineering.instagram.com/sharding-ids-at-instagram-1cf5a71e5a…
Edit: Update to correct snowflake to Instagram's strategy. I make this mistake often enough that it's now self-reinforcing :/
chrisco255 · 2017-08-20
Not true, see link: https://engineering.instagram.com/react-native-at-instagram-dd828a9a90…
Push Notifications, Edit Profile, Photos Of view, Post Promote, Comment Moderation, Lead Gen Ads, SMS Captcha...and that was in February. They are ramping up their usage all the time.
flunhat · 2017-07-03
Instagram is one, I'm sure there's others [1].
[1] https://engineering.instagram.com/web-service-efficiency-at-instagram-…
jeeyoungk · 2017-06-15
See issues like https://engineering.instagram.com/dismissing-python-garbage-collection…
If you have to turn off a crucial language feature to increase performance, I'm not sure whether a language is considered "scalable".
Guyag · 2017-05-11
Instagram engineering did a blog post a while back on implementing emoji hashtags explaining (among other things) how some are represented by >1 unicode character. Pretty interesting read [0].
[0] https://engineering.instagram.com/emojineering-part-ii-implementing-ha…
SEJeff · 2017-04-28
Honestly I think Uber switching was more of a business decision as their employees had trouble figuring out how to properly use Postgres. Instagram definitely has different data needs, but is probably much bigger than Uber wrt data. They use Postgres successfully via Django no less!
https://engineering.instagram.com/sharding-ids-at-instagram-1cf5a71e5a…
Symmetry · 2017-02-03
I'm not sure what the order of events was but I should certainly link that blog post.
https://engineering.instagram.com/dismissing-python-garbage-collection…
ParadoxOryx · 2017-01-09
Looks like the OP link is 404'ing.
Here's the working one — https://engineering.instagram.com/bringing-wide-color-to-instagram-5a5…
mburst · 2016-10-27
I believe Instagram uses Django https://engineering.instagram.com/web-service-efficiency-at-instagram-… In any case it's still a pretty popular site/app on it's own. I don't see Facebook killing it anytime soon.
paultela · 2016-01-29
There's a mention of Facebook regularly doing this in the summary section of this instagram engineering post: http://engineering.instagram.com/posts/548723638608102/
EDIT: Here's more info: http://www.datacenterknowledge.com/archives/2014/09/15/facebook-turned…