Tumblr Engineering
engineering.tumblr.com · Company blog · on HN since 2011
Tumblr's engineering team posts about its infrastructure, including MySQL sharding, PHP 7, notifications, GIF handling and file copying.
What HN saysHN commenters cite it for a useful MySQL sharding talk, a fast file-copy technique with pigz and netcat, and its PHP 7 move.
36HN people shared it
8linked it in comments
36stories submitted
1.3Kstory points
3 years agolast shared
2011people per year2026
Most-shared posts
- 1Efficiently copying files to multiple destinations3 people · 2 points · first shared 2011 · discussion on HN
- 2PHP 7 at Tumblr2 people · 303 points · first shared 2016 · 51 comments on HN
- 3Staircar: Redis-powered notifications2 people · 55 points · first shared 2011 · 15 comments on HN
- 4StreamBuilder: Tumblr's open-source framework for powering your dashboard2 people · 4 points · first shared 2023 · 1 comments on HN
- 5Tumblr's new engineering blog2 people · 3 points · first shared 2011 · discussion on HN
- 6Tumblr for iPhone is now 100% native2 people · 2 points · first shared 2012 · discussion on HN
- 7GIFs at tumblr2 people · 1 points · first shared 2014 · discussion on HN
- 8How I review code1 people · 234 points · first shared 2018 · 138 comments on HN
- 9Jetpants: a MySQL toolkit for managing billions of rows and hundreds of DBs1 people · 191 points · first shared 2012 · 23 comments on HN
- 10Flow and TypeScript1 people · 146 points · first shared 2017 · 97 comments on HN
- 11Colossus: A New Service Framework from Tumblr1 people · 77 points · first shared 2014 · 27 comments on HN
- 12Tumblr Opensources Kubernetes Utilities1 people · 72 points · first shared 2019 · 17 comments on HN
- 13Memcache top1 people · 67 points · first shared 2013 · 11 comments on HN
- 14Categorizing Posts on Tumblr1 people · 43 points · first shared 2016 · 5 comments on HN
- 15Everything that went wrong while building the Tumblr iOS share extension1 people · 19 points · first shared 2014 · discussion on HN
What HN people wrote when they linked it
versteegen · 2017-03-15
Use gifsicle. It's excellent and very fast, I don't think there's anything that produces smaller gifs than it. [0]
You can also use ImageMagick to produce optimised gifs (it has about 4 different options for that, eg -layers OptimizeTransparency). See [1]. But in my own tests, gifsicle always beats it (often by a huge margin) and is vastly faster. That ImageMagick page may be out of date. Also, Tumblr contracted the author of gifsicle, Dr. Eddie Kohler, to make some improvement for them, hopefully that made it back into gifsicle [2].
[0] https://www.lcdf.org/gifsicle/
[1] http://www.imagemagick.org/Usage/anim_opt/#compress_opt
[2] https://engineering.tumblr.com/post/80003435120/who-doesnt-love-animat…
notJim · 2012-06-07
Slightly OT, but Tumblr gave a great talk about their sharding architecture here: http://engineering.tumblr.com/post/12652551894/slides-from-our-velocit…
It's really quite a good intro to the subject, I think.
iworkforthem · 2012-03-29
http://engineering.twitter.com/
http://engineering.foursquare.com/
http://engineering.tumblr.com/
To name a few... know of one, please help add.. Thanks.
Xorlev · 2022-10-17
Pretty similar to that, we used pigz and netcat to bring up new MySQL read replicas in a chain at line speeds.
I recall learning the technique from Tumblr's eng blog.
https://engineering.tumblr.com/post/7658008285/efficiently-copying-fil…
kkoppenhaver · 2016-12-19
DailyMotion[1] and Tumblr[2] have recently moved to PHP7. In addition, WordPress[3] is moving towards PHP7 as well.
[1] http://engineering.dailymotion.com/php-7-deployment-at-dailymotion/
[2] https://engineering.tumblr.com/post/152998126990/php-7-at-tumblr
[3] (from 2015) https://make.wordpress.org/core/2015/09/10/wordpress-and-php7/
cmelbye · 2012-12-17
Add Tumblr to that list as well. They recently converted their iOS app from essentially pure HTML5 to pure native, which was no easy feat considering the dynamic nature of blog posts on Tumblr, which can contain almost arbitrary HTML content.
http://engineering.tumblr.com/post/35271768127/tumblr-for-iphone-is-no…
Of course, HTML5 proponents will continue to ignore these major data points.
iuguy · 2012-06-08
It looks interesting but one small thing. I looked into the transferring large files quickly link[1] and saw that they were using netcat and tar to transfer files. This is not necessarily optimal[2], and applying some compression can go a long way, although this will be dependent on the use case. Compression also has the added bonus of transferring less data across the network and (if you don't uncompress) less space at the other end.
SSH is also an option (a slow option, but an option) that provides certain things (encryption, authentication) that make it ideally suited for transfers across network boundaries.
[1] - http://engineering.tumblr.com/post/7658008285/efficiently-copying-file…
[2] - http://www.ndchost.com/wiki/server-administration/netcat-over-ssh
didip · 2012-02-14
I hope they managed to release staircar(http://engineering.tumblr.com/post/7819252942/staircar-redis-powered-n… before phasing it out with Scala.