Something Similar

> Instead of building storage that can fail between nodes, consider building applications that can survive node loss and are not relying on persisting their state in a black box that is assumed to be 100% available. These assumptions about reliablity of systems and their consistent behavior under network partitions has bitten countless people before and will continue to claim victims unless application paradigms are fundamentally rethought. I'd recommend this... except that it's really, really hard [0]. And if you think about it, you have to do this for every new application, which is a bit wasteful. So IMO, doing it at filesystem level is a good trade off. It's abstract enough to be useful over a wide range of use-cases (unlike block storage), because if you think about it file systems are just frameworks. To be fair, block can work really well. I think block storage vendors have themselves to blame. Half the "standards" aren't, interop can be terrible. I know I'd rather have simple, reliable storage - stop stuffing "enterprise" features into a rotting, increasingly technical-debt laden codebase. All back-up solutions for block suck - you'd think this problem would be fixed, especially with cloud storage. I could go on. [0] https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-syste…

When Jeff Hodges gave a presentation of his "Notes on Distributed Systems for Youngbloods"[1] at Lookout Mobile Security back in like 2014 or 2015, he did this really interesting aside at the end that changed my perception of my job, and it was basically this. You don't get to avoid "politics" in software, because building is collaborative, and all collaboration is political. You'll only hurt yourself by avoiding leveling up in soft skills. No matter how correct or elegant your code is or how good your idea is, if you haven't built the relationships or put consideration into the broader social dynamic, you're much less likely to succeed. [1] https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-syste…

It's subtly different from a cache. http://www.somethingsimilar.com/2012/05/21/the-opposite-of-a-bloom-fil…

Nice catch. I'm not so sure about: A simple fix will be just crawling the links without the request parameters so that we don’t have to suffer. Many links would fail/have different content if the request parameters were removed from the URL. Perhaps the crawler could use some kind of reverse bloom filter [1] to be more careful/back off if it receives the same content from multiple URLs. However nothing is simple at Google scale so there are probably issues with this approach too. [1]: http://www.somethingsimilar.com/2012/05/21/the-opposite-of-a-bloom-fil…

Another really good article on distributed systems along the same lines is this one: http://www.somethingsimilar.com/2013/01/14/notes-on-distributed-system…

Guys, this is real. http://www.somethingsimilar.com/2012/05/21/the-opposite-of-a-bloom-fil… Yes, it's a hash table without conflict resolution. No, that's not the joke. From the article: The simplest data structure that fits the description of a “opposite of a Bloom filter” is a hash map. With a large corpus like the one we were dealing with, the memory consumed by a hash map is far too large and its growth is bound only by corpus size. However, from this simple idea a more fruitful one comes out. The design I pitched uses an array, a hash function, and the ability to swap ids in and out of the array concurrently. The hash function and the array are used in the same way they are inside of a hash map implementation. Unlike a typical hash map implementation, the array size is bounded and hash collisions are not avoided.

The use case in mainly deduplication of a high volume data stream (though it's got a few other uses). The write volume is fairly stable so it's sized in such away that we'll never emit dupes even when the upstream source crashes and needs to be rebuilt from backups (for this case that means > a billion cache entries). Something like the opposite of a bloom filter (https://www.somethingsimilar.com/2012/05/21/the-opposite-of-a-bloom-fi… didn't work because we don't want false negatives either. Since the cache is fed by a Kafka log HA is achieved simply by having multiple consumers individually populating their own cache instance. The persistence mechanisms are to allow for code deployments that don't blow away the cache, not HA. We actually experimented with grid caches (ignite in particular since it offers off-heap in memory storage as well), but the performance simply isn't there. At the volume we're writing even millisecond latency is a non-starter. We did explore both memcached and redis, but we need strict FIFO and both of those solutions provide nondeterministic LRU.

Only one person in this comment thread mentioned CAP theorem, which is the (highly informal*) rationale for NoSql's existence and also its deliberate aversion to join dependencies (which more or less violate the core principles of eventual consistency modeled distributed data stores.) Useful links for erudition: https://arxiv.org/pdf/1302.0309.pdf https://codahale.com/you-cant-sacrifice-partition-tolerance/ https://martin.kleppmann.com/2015/05/11/please-stop-calling-databases-… https://www.somethingsimilar.com/2013/01/14/notes-on-distributed-syste… https://www.microsoft.com/en-us/research/publication/replicated-data-c…

As starter read Mikito Takada (mixu) book "Distributed Systems For Fun and Profit" http://book.mixu.net/distsys/single-page.html Notes on Distributed Systems for Young Bloods (http://www.somethingsimilar.com/2013/01/14/notes-on-distributed-system… A Note on Distributed Computing (http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.41.7628) http://en.wikipedia.org/wiki/Fallacies_of_distributed_computing You can read additional pointer from Henry blog :http://the-paper-trail.org/blog/distributed-systems-theory-for-the-dis…