Michael Mitzenmacher

Isn't this idea similar to the classic "power of 2 random choices" https://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf

https://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf and > Best of 2 is good because it combines the best of both worlds: it uses real information about load to pick a host (unlike random), but rejects herd behavior much more strongly than the other two approaches. https://brooker.co.za/blog/2012/01/17/two-random.html

It's in the source, Less Hashing, Same Performance: Building a Better Bloom Filter: https://www.eecs.harvard.edu/~michaelm/postscripts/rsa2008.pdf I've used this trick at scale on network gear and it works great.

Mitzenmacher, the author of Probability and Computing, has an interesting survey on cookoo hashing. In it there are a number of open research problems that, for those of you with interest, a worth looking over. The 7th is very interesting to me, regarding optimal ways to maintain a hash table in a parallel computing environment. http://www.eecs.harvard.edu/~michaelm/postscripts/esa2009.pdf

> few applications of random walks for random human walking I've heard a hypothesis that an advantage of dubious fortune-telling techniques (entrails, tea leaves, etc.) for hunter-gatherer societies is that random choice of prey and direction each day keeps them from over-exploiting any given resource to extinction. A weird flex Indian Rajas were said to have done is to release a horse and then follow it, offering to fight anyone they met upon the resulting (not so random) walk. https://en.wikipedia.org/wiki/Ashvamedha I have read both ancient sword fighters and modern fencers advise to consciously plan an attack, but unconsciously trigger it: by letting noise from the environment decide the moment T=0, you're less likely to give your opponent any tells. One might imagine that divination before battle would provide what the modern world calls a "mixed strategy" and could indeed by superior to pure strategies. (compare the SSBN; see also https://www.atlanticcouncil.org/in-depth-research-reports/issue-brief/… ) On a more peaceful note, if you wish to choose between popular dinner spots, it may be best to make the selection at hazard: http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf

Two random choices has a few significant advantages over "find the least filled one": - It's O(1), rather than O(buckets). - It's robust to stale data and concurrency. One way systems try to avoid the O(buckets) work is by caching, or by scanning in parallel, either of which cause stale load estimates. "Pick the best" tends to lead to significant over-filling http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf or https://brooker.co.za/blog/2012/01/17/two-random.html). - Best-of-k (rather than best-of-two) provides a parameter `k` that allows the system to tradeoff between optimality (higher k is more optimal), and robustness to stale data and concurrent accesses (lower k is more robust). Mitzenmacher's core observation is that the step up from k=1 to k=2 is a massive improvement in optimality with only a small loss in robustness (and much more than most people would expect).

>assuming you have a perfect cryptographic hash function Except there is no such thing (not in the sense you need it) - it too would be a perfect source of randomness, and you're just pushing the problem down the road. As an example, NIST publication 800-90B recommends multiple randomness extractors, one of which is SHA. They recommend using twice the amount of entropy in as the entropy out to get random "enough" bits. Thus even SHA is not going to mix bits well enough as you want. (The things called perfect hash functions in the literature map N items into N slots with no collisions, which is not what is needed here). As a perhaps surprising counterexample to any simple solution, here [1] is a math paper with a proof that there can be no optimal algorithm that is best for all values of coin bias. Here's [2] a cool walkthough on some of the ideas in theory that have been investigated. [1] https://web.eecs.umich.edu/~qstout/abs/AnnProb84.html [2] http://www.eecs.harvard.edu/~michaelm/coinflipext.pdf

I certainly already agreed that Rails' architecture is bad (though the reason that it has this problem is its memory usage, and not any of the other reasons you mention). Herokus architecture is bad as well. It's the combination of these that causes the problem. But that does not mean that it's impossible, or even hard, to solve the problem at Herokus end. > I'm not sure there are such providers, and if there aren't, I think it's safe to point the finger towards Rails. This is not sound logic. I described above two methods for solving the problem: (1) increase the memory per Dyno (see below: they're doing this, going from 512MB to 1GB per Dyno IIRC, which although still low will be a great improvement if that means that your app can now run 2 concurrent processes per Dyno instead of 1), or (2) do intelligent routing for small groups of Dynos. Do you understand the problem with random routing, and why either of these two would solve it? If not you might find the paper I linked to previously very interesting: "To motivate this survey, we begin with a simple problem that demonstrates a powerful fundamental idea. Suppose that n balls are thrown into n bins, with each ball choosing a bin independently and uniformly at random. Then the maximum load, or the largest number of balls in any bin, is approximately log n / log log n with high probability. Now suppose instead that the balls are placed sequentially, and each ball is placed in the least loaded of d >= 2 bins chosen…

It depends a bit on exactly what you mean. But eg if you want to write a new hash table (with a new hash function), you'd want to do pretty similar-ish analysis to figure out whether it's a good idea. I used some neat math in my time to justify much simpler algorithms than what we were using before. (But not hash table related.) https://www.keithschwarz.com/darts-dice-coins/ is also a joy to read. (As mentioned in the discussion on https://news.ycombinator.com/item?id=8006336 ) Have a look at eg https://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf for some more in this flavour.

Remember you also have to determine the coin is biased and the results are in a biased order. "An ideal unbiased coin might not correctly model a real coin, which could be biased slightly one way or another. After all, real life is rarely fair." https://www.eecs.harvard.edu/~michaelm/coinflipext.pdf

Ah nice, a "Power of 2 (random) choices" strategy. Article credits "The power of two choices in randomized load balancing" (1996) by Michael Mitzenmacher. PDF link: https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf I heard about someone using PO2C for cache eviction recently. Maybe Oxide talking about their storage work? Seems obviously smart. No need to keep LRU or carefully track & decide, just use statistics to compare two random elements and drop the least used one.

Thanks for the pointer. The file comment says: // This is a block Bloom filter (from Putze et al.'s "Cache-, Hash- and Space-Efficient Bloom Filters") with some twists: // 1. Each block is a split Bloom filter - see Section 2.1 of Broder and Mitzenmacher's // "Network Applications of Bloom Filters: A Survey". // 2. The number of bits set per Add() is contant in order to take advantage of SIMD instructions. Mentioned papers: https://www.cs.amherst.edu/~ccmcgeoch/cs34/papers/cacheefficientbloomf… https://www.eecs.harvard.edu/~michaelm/CS222/bloomsurvey.pdf

How does it compare to "random two choices" [1]? [1] https://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf It is simpler to implement than LRU - a single "access timestamp" needed to be attached to data. It also shows performance on par or better than LRU when applied to database caching.

Have you read The Power of Two Choices in Randomized Load Balancing?[0] [0] https://www.eecs.harvard.edu/~michaelm/postscripts/tpds2001.pdf

It comes from the paper "The Power of Two Random Choices: A Survey of Techniques and Results" - http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf

Link to the paper mentioned in the blogpost: http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf

The article linked does, yes. The paper the article is based on (linked in the article) has a proof for worst case load if you’re interested in the details. Edit: Link to paper http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf

There's also a more mathematically rigorous paper on the random-pick-2 topic in the references section of the Dropbox blog post -- https://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf

The Power of Two Random Choices: it's a surprisingly decent heuristic https://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf

The method is called "Power of Two Random Choices" (http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf). And the two-choices paradigm is widely applicable beyond load balancing. In particular, it applies to hash table design (e.g. cuckoo hashing) and cache eviction schemes (https://danluu.com/2choices-eviction/).

It is indirectly referenced near the bottom of the page, with a pointer to http://brooker.co.za/blog/2012/01/17/two-random.html, which references http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf

This 2 random cache scheme reminds me a lot of the 2 random load balancing scheme laid out in "The Power of Two Choices in Randomized Load Balancing" https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf Which shows an exponential improvement in choosing the less loaded of two randomly chosen servers over just random.

Good talk, I feel like I've tread exactly this same thinking and given a similar talk internally (i accidentally ended up as the LB SME), even used poisson distribution to model the requests and do some code demos. He references a simple algorithm/paper on simply chosing between two servers, and the results on the smoothing out of variance was impressive (paper: https://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf ). So, no real conclusion, it's just interesting to peel back the layers of LB.. so many people treat it as magic, when really its a statistical lie - and impossible to achieve perfect distribution (hence the title)!

Most of the 'Power of two choices' I've read about assumes the presence of a global queue (http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf) -- there's a parallel variety, but they go light on details in that text. I'm unaware of how Heroku does things. I'd guess they dropped the global queue because it's unpractical (failure prone, not scalable as it's a single point of contention). I'm mostly surprised to see people happy being able to handle 1 or 2 requests in parallel per instance in general. That sounds absolutely insane to me.

randomized routing is not necessarily bad if they look at 2 choices and pick the min. See http://en.wikipedia.org/wiki/2-choice_hashing and http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf

Randomized routing isn't all bad. In fact, if Heroku were to switch from purely random routing to minimum-of-two random routing, they'd perform asymptotically better [1]. [1]: http://www.eecs.harvard.edu/~michaelm/postscripts/mythesis.pdf

I was thinking that you could have the client pass in a couple of hash values, and then use linear combinations of them to get a bunch of independent-enough hashes. This is mathematically fine, and hash functions are a dime a dozen when you're not in redis-lua-land. Paper justifying this technique: http://www.eecs.harvard.edu/~michaelm/postscripts/tr-02-05.pdf

This stuff has just been in the air the past few years - Network coding ( http://www.eecs.harvard.edu/~michaelm/CS222/contentdist.pdf ) does the inverse operation to increase efficiency in Bittorrent-like distribution schemes. It is quite cool that randomness can help in both the creation and solving of these problems.

Alas, digital fountains are tied up in patents, and people have been leery: http://www.eecs.harvard.edu/~michaelm/postscripts/itw2004.pdf

There's interesting work on queuing theory. In practice, each shopper picks the queue that she thinks is least loaded. How many queue lengths do you need to sample? It turns out that if you sample just two queue lengths, you get a huge benefit (in terms of reducing the avg queue length), but not so much if you sample more. See "Power of two random choices": http://www.eecs.harvard.edu/~michaelm/postscripts/handbook2001.pdf EDIT: link