Peter Bailis

I love the reference to the ACIDRain paper in there. > They analyzed “12 popular self-hosted eCommenrce applications written in four languages and deployed on over 2M websites” and identified and verified “22 critical ACIDRain attacks that allow attackers to corrupt store inventory, over-spend gift cards, and steal inventory”. According to the paper, “Of the 22 vulnerabilities, five were level-based, meaning that the default weak isolation level led to the anomalies behind the vulnerabilities. http://www.bailis.org/papers/acidrain-sigmod2017.pdf

My favorite mental framework to reason about consistency in distributed systems, Invariant Confluence, was formulated by Peter Bailis at al in Coordination Avoidance in Database Systems (overview and link to paper http://www.bailis.org/blog/when-does-consistency-require-coordination/… Invariant confluence, determines whether an application requires coordination for correct execution. You can tailor the invariants to your requirements making invariant confluence a much better tool than CAP

Agreed! Peter has actually done some really great research into this, finding real vulnerabilities in applications due to insufficient transaction isolation: http://www.bailis.org/papers/acidrain-sigmod2017.pdf

I'm not thinking of strict serializability - I do actually mean linearizability. My previous comments were referring to the basic algorithm presented as the main contribution of this paper in section 3. In this algorithm, "Every command accesses only one object o." With each operation applying to a single object and each object satisfying linearizability, the system as a whole will be linearizable. The distinction between linearizability and strict serializability is somewhat subtle. I highly recommend this blog post [1] by Irene Zhang and this blog post [2] by Peter Bailis for some really great discussion on the subtleties involved. [1] https://irenezhang.net/blog/2015/02/01/consistency.html [2] http://www.bailis.org/blog/linearizability-versus-serializability/

I link to this in my post, but Peter Bailis's post on this is worth a read: http://www.bailis.org/blog/understanding-weak-isolation-is-a-serious-p…

> if I understand correctly there's no way to avoid an extra round trip? With HATs, you only need to contact one replica for every key. This is to goal behind our definition of "high availability" (http://www.bailis.org/blog/hat-not-cap-introducing-highly-available-tr… In general, I haven't seen algorithms guaranteeing serializability or atomicity that complete without a round trip to at least one other replica (or a possibly long trip to the master). Intuitively, the impossibility results dictate that this must be the case, otherwise partitioned replicas could safely serve requests. Daniel Abadi has a great post about this latency-consistency trade-off: http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-l…

That may be the case, but all other protocols that guarantee consistency suffer from similar subtle issues (which can be resolved through careful algorithm design). Eventual consistency, however, is never the answer, unless your question is "how do I get eventual consistency?" The consistency required by an application is entirely imposed by the domain, and it is that requirement that causes algorithmic difficulties. Using less-consistent protocols to achieve a consistent view simply pushes the problem to a different layer. Also, eventual consistency is a weak requirement that, when sufficient, enables a certain level of fault tolerance and low latency not achievable with other consistency levels (see [1]), although other consistency levels may offer good-enough fault-tolerance and latency (rivaling EC in many cases). EC is not in general the solution to the problem of "how do I best distribute my data?" [1]: http://www.bailis.org/blog/when-does-consistency-require-coordination/

That depends on what level of book, and what you want to learn from it. On "how real distributed systems are built", there isn't a really good book available in my opinion. I liked The Datacenter as a Computer, but it's scope is fairly narrow and is more focused on global infrastructure issues than software design issues. On the math behind distributed systems, Nancy Lynch's book Distributed Algorithms (http://www.amazon.com/Distributed-Algorithms-Kaufmann-Management-Syste… is my favorite, but it's getting to be quite out-of-date. Absent a book, there are a lot of great resources online. High Scalability(http://highscalability.com/) has a good amount of quality links to distributed systems content, and some editorial content of varying quality. In general, it's a good site to follow if you're interested in how distributed systems are being used in industry. I also like to follow Aphyr's blog(http://aphyr.com/), Daniel Abadi's blog(http://dbmsmusings.blogspot.com/), and Henry Robinson's blog(http://the-paper-trail.org/blog/) and Peter Bailis's blog(http://www.bailis.org/).

Cassandra needs ACID like a fish needs a bicycle ,-) "When is "ACID" ACID? Rarely." - https://www.bailis.org/blog/when-is-acid-acid-rarely/

Does RAMP or CURE offer any possibility of conditional writes with CRDTs? I have had these papers on my list to read for months, specifically wondering if it could be applied to Garage https://dd.thekkedam.org/assets/documents/publications/Report/RR-8858.… http://www.bailis.org/papers/ramp-sigmod2014.pdf

You don't easily get ACID with Relational Databases. The world works with Eventual Consistency, just embrace it. :-) "When is "ACID" ACID? Rarely" - http://www.bailis.org/blog/when-is-acid-acid-rarely/

This is a rehash of all the things explained in the paper http://www.bailis.org/papers/feral-sigmod2015.pdf TLDR: Not a good idea.

For similar isolation level anomalies in real world applications check out this SIGMOD '17 paper: ACIDRain: Concurrency-Related Attacks on Database-Backed Web Applications: http://www.bailis.org/papers/acidrain-sigmod2017.pdf

> Your business won't succeed if you're hacked daily. Which can be the consequence of setting the wrong isolation level: http://www.bailis.org/papers/acidrain-sigmod2017.pdf

I believe what we spanner calls 'external consistency' we call strict serializability. Read-write transactions in FaunaDB are strictly serializable, whereas reads default to serializable. This allows us to serve reads independently from the closest region to the client, cutting down on latency in geo-distributed clusters. This is a bit weaker than external consistency or strict serializability; Peter Bailis has a good writeup on the meaning of serializability vs strict serializability here: http://www.bailis.org/blog/linearizability-versus-serializability/

Also relevant: http://www.bailis.org/blog/how-to-make-fossils-productive-again/ Choice quote from there: > Kill the reference architecture and rethink our conception of “database.” The article titled “Architecture of Database System” should be considered harmful. If a system doesn’t have a buffer pool, it can still be a database, and, in fact, I’d prefer not to read any more papers on “databases” that have buffer pools. Instead, I’d prefer you shock me with your radical, new (and useful!) conception of a data management platform.

There's a quote that's gone around by Peter Bailis: "Despite the ubiquity of weak isolation, I haven’t found a database architect, researcher, or user who’s been able to offer an explanation of when, and, probably more importantly, why isolation models such as Read Committed are sufficient for correct execution. It’s reasonably well known that these weak isolation models represent “ACID in practice,” but I don’t think we have any real understanding of how so many applications are seemingly (!?) okay running under them." http://www.bailis.org/blog/understanding-weak-isolation-is-a-serious-p…

Also, I find the link in the page also very useful to understand the "serializable" and "linearizable". It makes clear definition and has more potential readings on ACID. http://www.bailis.org/blog/linearizability-versus-serializability/ Thanks, bailis.

It's typically called Write Skew. Wikipedia discusses it on the snapshot isolation page (http://en.wikipedia.org/wiki/Snapshot_isolation). Also, Peter Bailis's blog has info and links to papers if you want more detail (http://www.bailis.org/blog/when-is-acid-acid-rarely/)

My suspicion is that the author was trying to distinguish total order of events vs. partial order of events. I feel like the reason vector clocks and the like were created was to deal with the fact that real-time is problematic in distributed system. There is some response from the author though on this question: http://www.bailis.org/blog/linearizability-versus-serializability/#com…

And a TL;DR http://www.bailis.org/blog/scalable-atomic-visibility-with-ramp-transa…

The new paper the blog post introduces: http://www.bailis.org/papers/ramp-sigmod2014.pdf

HAT (http://www.bailis.org/papers/hat-vldb2014.pdf) looks pretty promising in terms of multi-term transactions. It turns out that you can push the problem off to garbage collection in order to make transaction id generation easy, and garbage collection is easier to be sloppy and heuristic about. The only problem is it isn't clear yet that HATs are as rich as 2PL-based approaches, and that nobody's built an industrial strength implementation yet.

I don't see the connection between your reply and the linked article's content. A major point is already made right at the start when an ACID definition is quoted from a book - which makes it clear that that term is too nebulous and, in conclusion, that simply regurgitating some definition is not helpful in practice. On the other hand, if you are actually able to design for all those complex points you may still not have one short definition to quote available, which would not speak against you at all. Not having a short definition for ACID to quote may happen to both real experts as well as people who really don't know anything. Because for real experts that definition is too simplified to be of any use. I saw similar issues a lot in school and in academia, when having to answer some test questions: When you actually know a lot more you may actually get many answers wrong, because the one asking used a much simplified model (and therefore more inaccurate than yours) as basis for creating their question. For example in biology, there are some key tenets, but when you go deeper you learn about all the additional quirks and exceptions. Using the simplified tenet an answer to a question "can this or that happen" would be never, but for someone who knows a lot more that answer would be wrong. As an expert, you no longer use that simplified model you learned early in school. Example blog post showing the world is more complex than the simple definition:…

This is a bit dumbed down, and ignores the domain terminology required to properly discuss the trade-offs here (which is puzzling given that it links to a post by Aphyr, where you can find incredibly thorough discussions around isolation levels and anomalies). > The fundamental problem with using Kafka as your primary data store is it provides no isolation. This is false. I can only assume the author doesn't know about the Kafka transactions feature? To be specific, Kafka's transaction machinery offers read-committed isolation, and you get read-uncommitted by default if you don't opt-in to use that transaction machinery (the docs: https://kafka.apache.org/0110/javadoc/index.html?org/apache/kafka/clie… Depending on your workload, read-committed might be sufficient for correctness, in which case you can absolutely use Kafka as your database. Of course, proving that your application is sound with just read-committed isolation is can be challenging, not to mention testing that your application continues to be sound as new features are added. Because of that, in general I think that the underlying point of this article is probably correct, in that you probably shouldn't use Kafka as your database -- but for certain applications / use-cases it's a completely valid system design choice. More generally this is an area that many applications get wrong by using the wrong isolation levels, because most frameworks encourage incorrect implementations…

It's not surprising that a simple theorem doesn't deal with practical trade-offs. In my experience if you feel you are running up against the CAP theorem you need to take a moment to think about the real needs of your application and looking for a more detailed theorem isn't going to guide you well, instead you need to think about what you actually want to happen when various kinds of failure occur. In finance for example, often trading systems will choose to be available and accept transactions when there is a failure and then reconcile afterwards - swapping the risk of making mistakes but in a trusted environment to prevent the potentially catastrophic condition of not being able to trade. The article claims that the CAP theorem only prohibits linearisability as a consistency model. The original CAP paper talks about "Atomic consistency" which is clearly at least linearisability. However as the proof formalizes, it's obvious that if you accept writes to totally split and separated systems which cannot communicate together (network partition), you cannot be consistent across the two parts. This is confusing because this kind of consistency only loosely matches the definitions in the ACID acronym which are geared around invalid database states in a single machine database, not about different requests to different database machines giving inconsistent answers. Linearizability versus serialisability (and strict serialisability) are explained well by Peter Bailis…

I've been using Rails for a decade now, across many jobs, and one thing I've experienced fairly consistently is the worst Ruby web apps I have ever dealt with are the ones that weren't written in Rails. I've both seen and spent a lot of time rewriting overgrown Sinatra monstrosities with assorted non-AR ORMs. These apps were generally riddled with XSS since they didn't handle escaping correctly, often had unmaintainable code that sometimes was thousands of lines long crammed into a single file, and lacking any sort of basic structuring principles mixed parameter handling with business logic all over the place. I've seen several attempts to "build a better Rails" that's "more OO", but even frameworks that solve the problems I've just described in the previous paragraph have one huge drawback: Rails is a lingua franca for web development in Ruby, and whatever incremental gains you get from having a better framework are heavily outweighed by the costs of making people learn a new framework-per-project, the lack of features and documentation, and the lack of the library ecosystem that surrounds Rails. At points I've tried to keep three ORMs/DB libraries in my head at a time: AR, DataMapper, and Sequel, which just left me wishing I was dealing with AR (the devil you know...) ORMs need massive feature sets to jam the proverbial square peg of relational databases through the round hole of objects, and AR is the only one I've used that's both expressive enough to cover all the…

I am afraid your premise is incorrect. Foreign keys can be enforced (particularly on insert-only or insert-mostly workloads) with very little contention, even in a distributed system: http://www.bailis.org/papers/ramp-sigmod2014.pdf And lest you think there is some hidden constant factor here that destroys real performance at the cost of "scalability," the techniques from that paper were used to completely destroy the TPC-C benchmark: http://arxiv.org/pdf/1402.2237.pdf Admittedly this stuff is not yet implemented in traditional RDBMSes, but there is absolutely no reason why it could not be and I fully expect it to in the future. Certainly, if you are having performance problems now, this knowledge doesn't help. But I'd venture to guess (from personal experience) that most people removing foreign keys are doing it preemptively, rather than evaluating whether it is actually causing performance problems.

> Cassandra is eventually consistent. Consistency is pretty important. Your above statement unfortunately has no match in distributed systems theory or practice. Basically eventually consistent means that the system is not consistent from the point of view of CAP, so it does not feature strong consistency. As an alternative eventual consistency means: our contract with the user is, that, at least, when there are no partitions, the system will no show split-brain conditions, so a given information will converge to a given value for all the nodes. This alone is a contract with the user, but does not tell a lot, it just tells you: 1) No strong consistency, sorry. 2) No split-brain conditions when the partitions heal. So "It features eventual consistency, consistency is important" statement, I'm afraid, means that you don't have the tradeoffs about distributed systems clear. Now, if EC alone does not mean a lot, how to classify an EC system? It depends on the exact implementation of what happens when there is a partition, and what happens when the partition heals. There are EC systems that are configured by default to accept writes in the minority side (even in partitions when there is a single DB node if we want to honor "A" of CAP), and later will merge depending on "wall clock" timestamps. This is a form of consistency (and is weak), but is not Consistency with capital-C of CAP. Other systems will use vector clocks instead of timestamps, which is stronger but slower.…

Looks like you cross-posted this comment here and on my blog, so I'll also cross-post in my response. I am quite confused by your statement that YugaByte does not claim linearizability. The C of CAP is linearizability (see the original CAP theorem https://users.ece.cmu.edu/~adrian/731-sp04/readings/GL-cap.pdf). By claiming to be CP from CAP, you are claiming linearizability. CockroachDB also makes the same CP claim. However, they explicitly walk back from this claim (https://www.cockroachlabs.com/blog/living-without-atomic-clocks/). In contrast, YugaByte documentation makes no effort to walk back from this claim. Rather, the documentation seems to indicate that YugaByte is linearizable at: https://blog.yugabyte.com/jepsen-testing-on-yugabyte-db-database/ and https://docs.yugabyte.com/latest/develop/learn/acid-transactions/. Also, I would encourage you to read the Herlihy-Wing paper on linearizability published in 1990. You seem to be confusing linearizability with ACID isolation levels, which are actually a different concept. Peter Bailis has a good blog post on the difference: http://www.bailis.org/blog/linearizability-versus-serializability/. On your third point, we agree. However, the point of my post is that relying on max clock skew without hardware support is dangerous. I’m going so far as to say that it’s so dangerous that it is incorrect to claim consistency guarantees when you make such assumptions. I would really encourage YugaByte to consider using global…