Yelp Engineering Blog

This announcement makes sense to me, having done something similar before personally (1). The difference between a screenshot of a photo saved as a PNG and a JPG is huge, usually over 1MB, and I'm sure Twitter gets plenty of these. Publishing their thresholds for the JPG vs PNG size comparison instead of referring to it just an internal metric would clear up some disconnect. Plus, I'd love to improve the static values I chose if there is better data available. I doubt the people working on this project wanted to change any PNG unless it's clear cut but the line has to go somewhere. These will be better handled by JPG. Numbers about avg file size would be awesome too, how much data will this save in a typical user session? 1 - https://engineeringblog.yelp.com/2017/06/making-photos-smaller.html (Section: Large PNG Detection)

I had asked a similar question on Quora [1] few years back. Here is the summary: - Facebook: https://www.facebook.com/Engineering?sk=notes - Google: http://google-engtools.blogspot.com/ - Twitter: http://engineering.twitter.com/ - Linkedin: http://engineering.linkedin.com/blog - Dropbox: https://tech.dropbox.com/ - Instagram: http://instagram-engineering.tumblr.com/ - Netflix: http://techblog.netflix.com/ - Flickr: http://code.flickr.com - Etsy: http://codeascraft.etsy.com/ - Yelp: http://engineeringblog.yelp.com/ - Heroku: https://blog.heroku.com/ - Airbnb: http://nerds.airbnb.com/ - Bitly: http://word.bitly.com/ - Evernote: https://blog.evernote.com/tech/ - Highscalability: http://highscalability.com/ [1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-bes…

This looks great! Looking forward to playing around with it. I loved the idea of Airbnb's Synapse, but it's tricky to configure (you basically have to write an haproxy config from scratch plus learn how synapse config sections map to the haproxy config). Plus it seemed like the non-zookeeper backends were pretty unstable, I had to fix a few things to get it working with ec2 tags (and fwiw, at this point it's been over a month and my PR to merge the changes back upstream hasn't even been commented on). How does Baker Street handle restarting haproxy, does it do anything like this[0] automatically to get zero-downtime configuration reloads? [0] http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-rel…

MRJob is a great way to get off the ground and running with MapReduce: http://engineeringblog.yelp.com/2010/10/mrjob-distributed-computing-fo… https://github.com/Yelp/mrjob

(disclaimer: I work on the Data Pipeline project @ Yelp - but these opinions are my own and not necessarily representative of Yelp) @jack9 - If I am understanding your point correctly, it is that you could just use S3 directly as the 'unified buffer' and the Kafka part is unnecessary. I'll try to shed some light on why we made Kafka part of this infrastructure. - Regarding "near real-time" this is our cheeky way of saying we internally haven't set an SLA or a formal definition of the maximum latency in the system for us to declare it 'real time'. Hard numbers wise from the time a MySQL[0] event happens to a transformed version of it being in redshift is roughly in the realm of 10-30 seconds (unofficial and not a guarantee of course, just what I am anecdotally seeing at this point, and we are certain we can bring that down). - As ora600 points out, we not only have multiple data sources, but also multiple data targets. So it's important to us to reduce the IN*OUT down to IN+OUT. It's important to note that nearly all of these connection points more naturally 'speak' in streaming than batch processing. - Kafka has been fantastic tech for this use case for us: aside from this s3/redshift OUT connector we generally are dealing with connectors which want to be streaming. In-fact if it were performant to do so we would love to stream message-by-message into redshift as well (it's not). - We already have a lot of infrastructure for batch processing - specifically a service for…

Most of the stream processing in the Data Pipeline happens inside of an internal project called PaaStorm, which is storm-like. It was built to take advantage of our platform as a service (http://engineeringblog.yelp.com/2015/11/introducing-paasta-an-open-pla… which handles process scheduling really well. Architecturally, it's pretty similar to Samza, with distributed processes communicating using Kafka. We do use Spark streaming, and are starting to use Kafka Streams and Data Flow, where they're a better fit. I'm personally most excited about Beam/Flink. We'll probably end up replacing the PaaStorm internals with some other tool, when one with good python support matures. Beam's event-time handling and windowing seem really promising at this point. https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102 is a great overview of the different concerns for stream processing.

The most significant issue with HAProxy and large SOAs is the lack of really good dynamic re-configuration. You don't want to have to manually change your configuration every time someone wants to add or change load balancing to a service, and if you do it automatically you run into HAProxy reloads being a reliability problem as they drop traffic on the floor due to Linux being goofy in how it implements SYN handling. It looks like there is a lot of good advice on this thread for how to mitigate the configuration problem (bamboo, consul template, etc ...) but then you run into reloads, and even if you use zero downtime restarts as I described in my HAProxy deep dive (http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-rel… you still end up with periodic ~50ms blips in your timings. We've found that Synapse (https://github.com/airbnb/synapse) is best in class at managing HAProxy for us in a fairly large SOA. It is really intelligent about doing as much as possible dynamically over the HAProxy stats socket as opposed to reloading for every configuration change, and we can finely tune how often we are reloading HAProxy. The best part is that it is fully pluggable. Use DNS, marathon, zookeeper, etcd, etc...: Synapse can manifest that service registration information into HAProxy configurations.

I had a similar reaction when reading https://engineeringblog.yelp.com/2024/10/migrating-from-postgres-to-my…

I found this on their tech blog from 2016: https://engineeringblog.yelp.com/2016/08/how-we-scaled-our-ad-analytic… Whether they really needed cassandra or whether it was just using fancy complicated tech for the sake of it, I couldn't say.

For those that care about how it got corrupted: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassandra-cluste… > The investigation around the exception revealed that at-least one of the SSTable (Sorted String Table) rows was unordered, which caused the compaction operation to fail. SSTables are immutable files that are always sorted by the primary key This was just... a bug in Cassandra? Is there anyone that can shed light on this? There seem to be plenty of people using Cassandra at scale -- is constantly repairing it normal practice?

Source: https://engineeringblog.yelp.com/2023/01/rebuilding-a-cassandra-cluste… (Jan 30, 2023)

assert on the call_count attribute of a mock instead of trying to use methods on it like .assert_called_once_with() "a mock’s job is to say, “You got it, boss” whenever anyone calls it. It will do real work, like raising an exception, when one of its convenience methods is called, like assert_called_once_with. But it won’t do real work when you call a method that only resembles a convenience method, such as assert_called_once (no _with!)." https://engineeringblog.yelp.com/2015/02/assert_called_once-threat-or-…

A visually impaired software engineer at Yelp recently published a blog post titled, "A Guide to Software Engineering for the Visually Impaired"[1]. He also gave talks at PyOhio[2] and North Bay Python[3]. [1] https://engineeringblog.yelp.com/2018/10/a-guide-to-software-engineeri… [2] https://www.youtube.com/watch?v=L3WpnG49XLc [3] https://www.youtube.com/watch?v=FvlzaOrLBNw

> haproxy can't do that. HAProxy reloads don't interrupt existing connections (unless you want it to). There is a ~microsecond downtime in accepting new connections. https://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-re…

Think it's a "process Graph"... and Process ID 1 is systemd on the[/her] machine. Edit: turns Out, she explains: [...]systemd: rkt will run a program called systemd-nspawn as the init (PID 1) process inside your container. This is because it can be bad to run an arbitrary process as PID 1[0] -- your process isn't expecting it and will might react badly. It also run some systemd-journal process? I don't know what that's for yet.[...]" [0] https://engineeringblog.yelp.com/2016/01/dumb-init-an-init-for-docker.…

The front-facing system is the smallest part The admin and back-office system is deeper. Beyond the reviews you also have: events, mailing list management, profile management, messages, i18n, search, etc Example: http://engineeringblog.yelp.com/2015/10/how-we-use-deep-learning-to-cl… Edit: still, 3Mi lines is massive. However, I think there's something that contributes significantly: HTML and CSS

A year later, there is dumb-init[1]. Released as statically linked binary, it is simple ADD + chmod +x to get running simple init without any dependencies. ^1: https://engineeringblog.yelp.com/2016/01/dumb-init-an-init-for-docker.…

Is that new in 1.6? As of 1.5.11 there was a lot of hacking required to get a true zero downtime reload: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-rel…

> Anytime I add a new machine (e.g., web4) i've to change the conf of HAProxy and restart it. This brings down the system for a moment. (is this the right approach btw?) You might find this story from a year ago interesting: "True Zero Downtime HAProxy Reloads" http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-rel… HN discussion (with some answers from the post author): https://news.ycombinator.com/item?id=9369051 I'm curious as to how much of an actual issue you experience though. Barring an error that prevents HAproxy from starting, it should be pretty quick? Maybe not quick enough for streaming media/realtime audio-visual communication though.

There is, sort of. If you're hosting on Linux, you can use iptables to artificially delay responses. That and some other methods are described in a post on the Yelp engineering blog[1]. 1: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-rel…

I think this was the problem [1] was targeting [1] http://engineeringblog.yelp.com/2016/01/dumb-init-an-init-for-docker.h…

I disagree with the "big data" line in the sand being whether "a kid doing a paper run could afford your servers". Using spot instances on AWS, you can have a 100 machine cluster with 1.5TB RAM for only $3 per hour. Working at Common Crawl, some of the coolest projects I've seen have been side projects or weekend projects by interested volunteers. WikiReverse[1] cost only $64USD to parse the metadata of all 3.6 billion pages (even cheaper if you avoided EMR fees) whilst Yelp extracted 748 million US phone numbers from 2 billion pages for $10.60USD. These days, big data (regardless of how you define the term) is within the reach of the kid with a paper route. [1]: https://wikireverse.org/ [2]: http://engineeringblog.yelp.com/2015/03/analyzing-the-web-for-the-pric…

I was under the impression the `SO_REUSEPORT` implementation on Linux was rather broken, as evidenced by HAProxy's graceful reload still dropping packets[1]. I'm curious as to why Nginx has implemented this when you can design your Nginx config around the problem while not encountering the broken implementation. [1] http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-rel…

This way is even better, as it queues up the SYN packets until HAProxy is ready, leading to no faster responses than waiting 1s until the client retries with another SYN: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-rel…

Relevant blog post: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-rel…

Yep, I found a number of them, but I promptly forgot them in excitement when I found the one I actually wanted. If you want to check them out yourself, they appear to congregate in the kernel tree at net/sched/sch_* [1]. The only other qdisc I have much experience with is sch_netem, which emulates behaviors of a WAN (delay, loss, etc). I used it in this post [2] to conduct adversarial testing of MySQL replication (search 'tc qdisc'). [1] http://lxr.free-electrons.com/source/net/sched/ [2] http://engineeringblog.yelp.com/2014/03/mysql-replication-network-issu…

Yelp adds places to their database, and they have had mixed success with Mechanical Turk. http://engineeringblog.yelp.com/2011/02/towards-building-a-high-qualit…

At one point we used Dumbo for a lot of stuff at Yelp, but we finally ended up writing our own framework: http://engineeringblog.yelp.com/2010/10/mrjob-distributed-computing-fo… It's open source and easy to use, and makes hadoop on EMR exceptionally clean. Check it out.