Yahoo Engineering
yahooeng.tumblr.com · Company blog · on HN since 2013
Yahoo's engineering team blogs about its infrastructure, open-source releases, and projects such as object storage, YUI, and mail.
What HN saysHN links it for details on Yahoo's Ceph-based object storage, the end of YUI development, and tools like MDBM.
53HN people shared it
18linked it in comments
71stories submitted
1.7Kstory points
3 years agolast shared
2013people per year2026
Most-shared posts
- 1Benchmarking Streaming Computation Engines at Yahoo9 people · 45 points · first shared 2015 · 3 comments on HN
- 2Evolving Yahoo Mail5 people · 16 points · first shared 2014 · 3 comments on HN
- 3Yahoo Cloud Object Storage at Exabyte Scale5 people · 8 points · first shared 2015 · 1 comments on HN
- 4Open Sourcing a Deep Learning Solution for Detecting NSFW Images3 people · 453 points · first shared 2016 · 145 comments on HN
- 5Yahoo stopping all new development of YUI3 people · 450 points · first shared 2014 · 223 comments on HN
- 6MDBM – High-speed database2 people · 159 points · first shared 2014 · 111 comments on HN
- 7Screwdriver: Yahoo's Continuous Delivery Build System for Dynamic Infrastructure2 people · 158 points · first shared 2017 · 58 comments on HN
- 8Open-Sourcing Yahoo's Pulsar, Pub-Sub Messaging at Scale2 people · 104 points · first shared 2016 · 18 comments on HN
- 9Cronshot2 people · 5 points · first shared 2014 · discussion on HN
- 10Under the Hood: Delivering the First Free Global Live Stream of an NFL Game2 people · 5 points · first shared 2015 · discussion on HN
- 11Elide: Simplify Your CRUD2 people · 5 points · first shared 2015 · discussion on HN
- 12NodeJS High Availability2 people · 4 points · first shared 2013 · discussion on HN
- 13Pistachio: co-locate the data and compute for fastest cloud compute2 people · 3 points · first shared 2015 · discussion on HN
- 14Yahoo Engineering's Tumblr Page2 people · 1 points · first shared 2015 · discussion on HN
- 15Yahoo open-sources Kafka Manager for Kafka cluster management1 people · 92 points · first shared 2015 · 12 comments on HN
What HN people wrote when they linked it
ot · 2024-01-08
The repo does not appear to have had substantial changes for ~8-9 years https://github.com/yui/yuicompressor
Yahoo announced development was stopped in 2014: https://yahooeng.tumblr.com/post/96098168666/important-announcement-re…
Is this still relevant?
ymse · 2017-11-15
This post reminded me of a very old article from Yahoo/Tumblr explaining how they were (ab)using Ceph to generate thumbnails on the fly as pictures were uploaded using the Ceph OSD plugin interface.
Unfortunately the post seems to have disappeared from the internet (it was probably around 6 years ago), so here are some other teasers:
https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-object-store…
https://ceph.com/geen-categorie/dynamic-object-interfaces-with-lua/
Disclaimer: not affiliated with Ceph apart from being a happy sysadmin.
dryman · 2017-05-09
Thanks for raising this up. I spent way way more time on OPIC (almost a year) than robin hood hash map (2 weeks to complete the POC). Glad to see people noticing this project :)
The other comment pointed out MDBM, which I didn't know about.
From their performance number I think this may show that why OPIC robin hood is quite optimal.
https://yahooeng.tumblr.com/post/104861108931/mdbm-high-speed-database
MDMB gives users raw access to the mmaped data and pointers. And from its benchmarks it results 10x faster than rocksdb and leveldb. The design of OPIC has even less overhead (may not be a good thing) than MDBM, and it also works on a mmaped file (or anonymous swap). There's no lock, transaction, or WAL in OPIC. OPIC just brings you the raw performance a hash table can gives you.
Veratyr · 2017-01-12
They're already on Ceph-based Cloud Object Store, which uses 8/3 erasure coding: https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-object-store…
Erasure coding isn't something new, unknown or unique to Minio, it's built into Ceph, GlusterFS and Openstack Swift, the largest distributed object stores.
twiss · 2017-01-12
The first link in the article [1] says Yahoo deploys Ceph internally for object storage.
[1]: https://yahooeng.tumblr.com/post/116391291701/yahoo-cloud-object-store…
BinaryIdiot · 2016-09-30
The Yahoo blog[1] post is far more interesting than this techcrunch "article". Suggest changing URL to the Yahoo Blog please.
[1] https://yahooeng.tumblr.com/post/151148689421/open-sourcing-a-deep-lea…
yk · 2016-09-30
I would suggest, that the link should go to Yahoo's blog post
https://yahooeng.tumblr.com/post/151148689421/open-sourcing-a-deep-lea…
which contains some technical details. (And furthermore, I guess the HN crowd has enough Internet experience to come up with stupid jokes of their own design.)
abc_lisper · 2016-05-26
> Most hip-fired benchmarks, including that awful Yahoo! one that everybody cites, use Flink poorly
Why are Yahoo! benchmarks awful? How did they manage to use Flink poorly?
For people who don't know what he is referring to, check this: https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-c…
cprayingmantis · 2016-04-12
As with most tools it's depending on what you want to do with it. Storm is meant to handle streaming data. It's wonderful for low latency big data applications. You can check out the benchmarks for it at Yahoo Developer Tumblr https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-c…
doctorcroc · 2016-03-09
The benchmarking study done by Yahoo was fairly comprehensive, and quantitatively assesses the different stream frameworks (including Flink and SS) https://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-c…
We also did a podcast about it if you're interested in digging deeper -- http://softwareengineeringdaily.com/2016/02/03/benchmarking-stream-pro…
faizshah · 2016-03-09
Eifoov7h · 2016-02-04
This is the reference benchmark for the blog post: http://yahooeng.tumblr.com/post/135321837876/benchmarking-streaming-co…
shawndumas · 2015-04-15
Hey, they use trendy JS frameworks too!
http://discuss.emberjs.com/t/big-things-are-happening-here-at-yahoo/39…
http://yahooeng.tumblr.com/post/101682875656/evolving-yahoo-mail
jameswragg · 2014-11-26
Here's the associated Yahoo Engineering blog post that goes with the slides.
http://yahooeng.tumblr.com/post/101682875656/evolving-yahoo-mail
caridy · 2014-10-14
The official announcement is now live:
http://yahooeng.tumblr.com/post/100006468771/announcing-format-js-inte…
hopefully it will help to clarify few of the questions...
mcescalante · 2016-08-04
I'm mainly wary of Squarespace because their entire frontend compiles to YUI which stopped being maintained in August, 2014 (see https://yahooeng.tumblr.com/post/96098168666/important-announcement-re… You don't write YUI, but that's what the code ends up as when you visit the site. I tried out their developer platform early after they launched it, and trying to be productive with it put my brain in a pretzel.
I feel as though the SS CMS is overpriced for the features you get, and you're locked into their tech stack & templating which I find leave something to be desired. You can use Wordpress as a CMS and use the API to get the data yourself, or just compile the site to static for free with a plugin, not to mention the large number of paid pluggable CMS solutions for developers. The list goes on, but a local development server shouldn't be this big of news for a CMS developer platform.
pdevr · 2015-05-19
These are some of the top sites' engineering blogs which I collected over a period of time - I removed those which have already been posted:
Google: http://googledevelopers.blogspot.com/ http://google-engtools.blogspot.com/ http://googleresearch.blogspot.com/
YouTube: http://youtube-eng.blogspot.com/
Baidu: http://usa.baidu.com/category/baidutechblog/
Yahoo: http://yahooeng.tumblr.com/ http://yahoodevelopers.tumblr.com/
Wikipedia (closest I could find): https://blog.wikimedia.org/c/technology/
Amazon: https://aws.amazon.com/blogs/aws/
Twitter: https://engineering.twitter.com/
eBay: http://www.ebaytechblog.com/
Microsoft: http://microsoftengineering.com
FourSquare: http://engineering.foursquare.com/
Apple (closest): https://developer.apple.com/
Paypal: https://www.paypal-engineering.com/ https://devblog.paypal.com/
Dropbox: https://blogs.dropbox.com/tech/
Edit: More (collected from this link: http://www.quora.com/What-are-the-top-startup-engineering-blogs):
Yelp: http://engineeringblog.yelp.com/
Quora: http://engineering.quora.com/
Yammer: http://eng.yammer.com/blog/
AirBnB: http://nerds.airbnb.com/
Square: https://corner.squareup.com/
Flickr: http://code.flickr.net/
Soundcloud: https://developers.soundcloud.com/blog/
Cloudera: http://blog.cloudera.com/blog/
Slideshare: http://engineering.slideshare.net/
Heroku: http://engineering.heroku.com/
Pinterest: http://engineering.pinterest.com/
vjeux · 2015-04-06
React (outside of React Native) is being used on many high traffic websites.
- Facebook comments, chat, search, ads creation, photo upload, tablet website... and basically every new product being written.
- Entire Instagram website
- Yahoo Mail: http://yahooeng.tumblr.com/post/101682875656/evolving-yahoo-mail
- A lot of the Khan Academy front-end: http://khan.github.io/react-components/
- Netflix website and TV experiences: http://conf.reactjs.com/schedule.html#beyond-the-dom-how-netflix-plans…
- Many Airbnb new products: http://www.slideshare.net/spikebrehm/the-evolution-of-airbnbs-frontend
- Flipboard mobile website: http://engineering.flipboard.com/2015/02/mobile-web/
- ...