Daniel Abadi

I was just looking into this, and found this really interesting post about foundationdb and sql. https://www.voltactivedata.com/blog/2015/04/foundationdbs-lesson-fast-… Also this, excellent comparison of distributed database consistency methods, mentions cockroach, but not foundation. http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-fa… This brings up the question, what method/protocol does foundation use for distributed consistency?

This post inpsired a lot of great discussion about the Spanner family of distributed databases. Someone asked about TiDB there, but there was no follow up. Can someone answer that question? http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-fa…

For further reading, Dan Abadi's post at http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-fa… addresses the Spanner issues more directly. I don't think there's a great reference for the Percolator transaction architecture specifically, but there is a brief description here: https://blog.octo.com/en/my-reading-of-percolator-architecture-a-googl… One thing I learned recently is that FoundationDB is essentially the Percolator model (timestamp oracle), even though FoundationDB predated the Percolator paper.

It is more of an introduction but I enjoyed 7 databases in 7 weeks from the pragmatic bookshelf (https://pragprog.com/book/rwdata/seven-databases-in-seven-weeks). I have also learned a lot from Aphyr's Jepsen posts about failure modes of mostly NoSQL databases: https://aphyr.com/tags/Jepsen I enjoy following Daniel J Abadi (https://www.cs.umd.edu/~abadi/), for instance about distributed databases and transactions (https://dbmsmusings.blogspot.com/2015/10/why-mongodb-cassandra-hbase-d… Thanks for asking the question, I hope more references show up as I would love to learn more as well.

> if I understand correctly there's no way to avoid an extra round trip? With HATs, you only need to contact one replica for every key. This is to goal behind our definition of "high availability" (http://www.bailis.org/blog/hat-not-cap-introducing-highly-available-tr… In general, I haven't seen algorithms guaranteeing serializability or atomicity that complete without a round trip to at least one other replica (or a possibly long trip to the master). Intuitively, the impossibility results dictate that this must be the case, otherwise partitioned replicas could safely serve requests. Daniel Abadi has a great post about this latency-consistency trade-off: http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-l…

That depends on what level of book, and what you want to learn from it. On "how real distributed systems are built", there isn't a really good book available in my opinion. I liked The Datacenter as a Computer, but it's scope is fairly narrow and is more focused on global infrastructure issues than software design issues. On the math behind distributed systems, Nancy Lynch's book Distributed Algorithms (http://www.amazon.com/Distributed-Algorithms-Kaufmann-Management-Syste… is my favorite, but it's getting to be quite out-of-date. Absent a book, there are a lot of great resources online. High Scalability(http://highscalability.com/) has a good amount of quality links to distributed systems content, and some editorial content of varying quality. In general, it's a good site to follow if you're interested in how distributed systems are being used in industry. I also like to follow Aphyr's blog(http://aphyr.com/), Daniel Abadi's blog(http://dbmsmusings.blogspot.com/), and Henry Robinson's blog(http://the-paper-trail.org/blog/) and Peter Bailis's blog(http://www.bailis.org/).

Bringing this interesting read back to the surface, I (like many) had initially been bewildered by the CAP theorem. This post clarifies the ambiguities and extends the categories of a distributed system. Note, I pasted the wrong link in the submission: this is the post I intended to share: https://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-…

Prof. Daniel Abadi talks about the two meanings of consistency in this blogpost: https://dbmsmusings.blogspot.com/2019/07/overview-of-consistency-level… I'd argue that in your second case, there's no consistency "external" to the system; you've now defined a larger, distributed system that includes your external parties, and now you're in consistency-as-in-CAP territory, rather than consistency-as-in-ACID.

Is there a consensus / standard on the wording of the "consistency level" and "system guarantee"? https://dbmsmusings.blogspot.com/2019/06/correctness-anomalies-under.h… Classification of serializable systems * (consistency level) READ UNCOMMITTED (ISO) READ COMMITTED (ISO) REPEATABLE READ (ISO) SNAPSHOT ISOLATION SERIALIZABLE (ISO) ONE COPY SERIALIZABLE STRONG SESSION SERIALIZABLE STRONG WRITE SERIALIZABLE STRONG PARTITION SERIALIZABLE STRICT SERIALIZABLE * System Guarantee Dirty read (ISO) Non-repeatable read (ISO) Phantom Read (ISO) Write Skew Immortal write Stale read Causal reverse

It's not about whether clients wait for acks, but rather whether participants of the consensus (instances of the database servers, basically) wait for consensus prior to commiting. If the client doesn't wait until the transaction is commited, a a client cannot 'proceed', as the transaction is not observable in any node. Giving this guarantee of consistency requires consensus among the nodes, and consensus introduces coordination, which introduces latency. That's the basic argument for PACELC. It is also totally possible, but rare in practice, to design a PC/EL system. The cannonical example by Prof. Abadi is PNUTS: https://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-…

I don’t know why the author is surprised that Postgres offers stronger guarantees than serializability in practice. Serializability per se allows anomalies that would be disastrous in practice: http://dbmsmusings.blogspot.com/2019/06/correctness-anomalies-under.ht…

Hello, author here. That's a good question and something I had a hard time sorting out as I worked on this. I think those fall into a different category confusingly sometimes called column-oriented databases. They're primarily used for analytic-focused tasks and get their name from storing data by column instead of by row (all data in a single column is stored to disk together). I didn't include those as a separate category here because they're basically relational databases with a different underlying storage strategy to allow for easier column-based aggregation and so forth. My colleague shared this article [1] with me, which definitely helped inform how I distinguished between the two in my head. [1] http://dbmsmusings.blogspot.com/2010/03/distinguishing-two-major-types…

For those wondering, since CockroachDB is not mentioned by name here, the post [1] from the same author on his own site does explicitly mention CockroachDB. "However, (unlike Google Spanner) CockroachDB does not wait for the maximum clock skew bound to pass before committing a transaction. Therefore, it is possible in CockroachDB for a transaction to commit, and a later transaction to come along (that writes data to a different partition), that was caused by the earlier one (that started after the earlier one finished), and still receive an earlier timestamp than the earlier transaction." [1] https://dbmsmusings.blogspot.com/2019/06/correctness-anomalies-under.h…

Yes, that's what I refer to as "unified consensus". Batching helps to overcome the scalability challenges (i.e. order batches rather than each individual xact). I talk about this briefly at: http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-fa…

Abadi has an interesting paper here about why so many NoSQL databases are missing transactions. http://dbmsmusings.blogspot.com/2015/10/why-mongodb-cassandra-hbase-dy… The fairness, isolation, and throughput (FIT) tradeoffs seem more interesting to me than CAP.

Cassandra really isn't a columnstore in the same sense that the word is used for OLAP systems. In the latter, efficient queries can be run across lots of values in a small number of columns across many rows. In Cassandra, queries spanning many rows are not particularly efficient. Sometimes wide-column tables are used for OLAP-type use cases in Cassandra, but that's something of a different approach. More info on that here: http://dbmsmusings.blogspot.com/2010/03/distinguishing-two-major-types…

I don't know the answer, but in this case does columnar store imply that it is a collection of arrays, perhaps for a scientific database, and a bit different than HBase? Here's someone else's blog post from 2010 on different categories of columnar store DBs: http://dbmsmusings.blogspot.com/2010/03/distinguishing-two-major-types…

> For a long time I was confused about why HBase was described as a columnar data store when access was still pretty row-based The term was overloaded by column-family stores, which were often referred to as just 'column stores', probably by people who were not aware of systems like Vertica and MonetDB. http://dbmsmusings.blogspot.co.uk/2010/03/distinguishing-two-major-typ…

> You cannot give up partition tolerance. It'd be more accurate to say you don't want to give it up all the time. You don't want CAP; you want PACELC. http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-l…

More information on what cperciva is saying (from Daniel Abadi) http://dbmsmusings.blogspot.com/2012/10/ieee-computer-issue-on-cap-the… Quoting from the post: "my past criticism of CAP not actually being about picking two of three out of C (consistency), A (availability), and P (partition tolerance) due to the fact that it does not make sense to reason about a system that is ‘CA’. (If there is no partition, any system can be both consistent and available --- the only question is what happens when there is a partition --- does consistency or availability get sacrificed?)"

Prof. Abadi's post on his blog about Calvin: http://dbmsmusings.blogspot.co.uk/2012/05/if-all-these-new-dbms-techno… Reddit comments on the blog post: http://www.reddit.com/r/programming/comments/trb7e/if_all_these_new_db…

Some of the code is based on Postgres, based on their own marketing materials: "Paraccel has leveraged Postgres for some of its parsing and planning functions". So the head node continues to have Postgres origins, but not the compute nodes. Also see this write-up on their paper discussing the links to Postgres: http://dbmsmusings.blogspot.com/2009/07/paraccel-and-their-puzzling-tp…

PACELC[1] helps distinguish just the kind of CAP sacrifice this makes. [1] http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-l…

I'm very curious about this too. I posted in another thread: "Doesn't that just sweep the latency tradeoff under the rug, or is flash making up the difference? What about the availability tradeoff? (I like the formulation here, consistency vs availability and consistency vs latency, as opposed to CAP which never made sense as a 3-way tradeoff: http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-an...) http://news.ycombinator.com/item?id=3480805

The author basically doesn't believe CAP represents a good model of the database tradeoffs; "Over the past few weeks, in my advanced database system implementation class I teach at Yale, I’ve been covering the CAP theorem, its implications, and various scalable NoSQL systems that would appear to be influenced in their design by the constraints of CAP. Over the course of my coverage of this topic, I am convinced that CAP falls far short of giving a complete picture of the engineering tradeoffs behind building scalable, distributed systems." http://dbmsmusings.blogspot.com/2010/04/problems-with-cap-and-yahoos-l…

A version which was rejected included the following (non-Apache-affiliated [afaik]) references: https://www.xenonstack.com/insights/what-is-apache-arrow/ https://link.springer.com/chapter/10.1007%2F978-1-4842-1311-7_5 https://www.biorxiv.org/content/biorxiv/early/2016/08/23/071092.full.p… http://delivery.acm.org/10.1145/3110000/3103003/p138-Maas.pdf https://www.theregister.co.uk/2016/02/17/apache_arrow_toplevel_project… https://www.cio.com/article/3034279/big-data-gets-a-new-open-source-pr… https://www.infoworld.com/article/3033446/hadoop/apache-arrow-aims-to-… https://sdtimes.com/apache/guest-view-first-release-apache-arrow/ https://www.infoq.com/news/2016/12/le-dem-apache-arrow/ http://dbmsmusings.blogspot.com/2017/10/apache-arrow-vs-parquet-and-or… https://dbmsmusings.blogspot.com/2017/10/apache-arrow-vs-parquet-and-o…

FaunaDB uses Calvin, a transaction protocol developed by Daniel Abadi. Their blog post explains it nicely, after a bit of a slow start: https://fauna.com/blog/consistency-without-clocks-faunadb-transaction-… But in summary: 1. A 'transaction' is a self-contained blob of code which reads input, does deterministic logic, and writes output (so not like a traditional RDBMS transaction, where the application opens a transaction and then interleaves its own logic between reads and writes) 2. When a transaction arrives, the receiving node runs it, and captures the inputs it read, and the outputs it wrote 3. The transaction, with its captured inputs and outputs, is written to a global stream of transactions - this is the only point of synchronisation between the nodes 4. Each node reads the global stream, and writes each transaction into its persistent state; to do that, it repeats all the reads that the transaction did, and checks that they match the captured input - if so, the outputs are committed, and it not, the transaction is aborted, and retried The key idea is that because the process is deterministic, the nodes can write transactions to disk independently without drifting out of sync. It's pretty neat. And it's exactly what Abadi wrote about a couple of months ago: http://dbmsmusings.blogspot.com/2019/01/its-time-to-move-on-from-two-p… This is also what VoltDB does (which Abadi worked on, along with Michael Stonebraker): As an operational store,…

You are correct that they serialize cross-partition transactions. So if your workload has a reasonable percentage of write/update/delete operations it is possible that you will bottleneck on the global coordinator. From this blog https://fauna.com/blog/distributed-acid-transaction-performance, you can get 3300 transactions/second. Daniel Abadi claims you can get to 500,000 trans/sec - http://dbmsmusings.blogspot.com/2018/09/newsql-database-systems-are-fa… In any case, there is an open-source in-memory database called NDB (MySQL Cluster) that can get to 100 million write transactions per second. The echo chamber of SV has ignored it, but it's real: http://mikaelronstrom.blogspot.com/2015/03/200m-reads-per-second-in-my… NDB has what i call lock-aware programming. When you read a row, you take either a read_lock or a write_lock, as it only supports READ_COMMITTED isolation level. But, it does not serialize cross-partition transactions. Transaction coordinators run independently, and it is up to the developer to ensure consistency by taking either read or write locks on rows.

That system works by giving up interactive transactions. But the foundationdb page claims to have interactive transactions, so it must be doing something else. From Abadi's blog: http://dbmsmusings.blogspot.com/2012/05/if-all-these-new-dbms-technolo… Calvin requires all transactions to be executed fully server-side and sacrifices the freedom to non-deterministically abort or reorder transactions on-the-fly during execution. In return, Calvin gets scalability, ACID-compliance, and extremely low-overhead multi-shard transactions over a shared-nothing architecture. The comments on that post are pretty interesting, too. And from the foundationdb features page: http://foundationdb.com/#features FoundationDB transactions are true interactive sessions, unlike distributed databases that require stored procedures. This means that client code can make an iterative series of reads and writes over the network to execute complex transactions.