PingCAP Blog

I do agree with you in that regard, however, that's also a dangerous line of thinking. There are attempts to provide horizontal scalability for RDBMSes in a transparent way, like TiDB https://pingcap.com/ (which is compatible with the MySQL 5.7 drivers), however, the list of functionality that's sacrificed to achieve easily extensible clusters is a long one: https://docs.pingcap.com/tidb/stable/mysql-compatibility There are other technologies, like MongoDB, which sometimes are more successful at a clustered configuration, however most of the traditional RDBMSes work best in a leader-follower type of replication scenario, because even those aforementioned systems oftentimes have data consistency issues that may eventually pop up. Essentially, my argument is that the lack of good horizontally scalable databases or other data storage solutions is easily explainable by the fact that the problem itself isn't solvable in any easy way, apart from adopting eventual consistency, which is probably going to create more problems than it will solve in case of any pre-existing code that makes assumptions about what ways it'll be able to access data and operate on it: https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing To that end, i'd perhaps like to suggest an alternative: use a single vertically scalable RDBMS instance when possible, with a hot standby if you have the resources for that. Let the architecture around it be horizontally scalable instead, and let it deal…

https://vitess.io/ https://www.cockroachlabs.com/ https://www.pingcap.com/

PingCAP | https://www.pingcap.com | Database Engineer, Product Manager, Developer Advocate and more | Remote in California | Full-time We work on a MySQL compatible distributed database called TiDB https://github.com/pingcap/tidb/ and key-value store called TiKV. TiDB is written in Go and TiKV is written in Rust. More roles and locations are available on https://www.pingcap.com/careers/

A relational database scalable for the billing data volume, should be a best answer, in term of realtime process and analysis，say complex query. Suggest an article of a big logistics company case here, not only billing, but also order, messaging, transshipment: https://en.pingcap.com/blog/how-to-reduce-cost-per-order-with-a-distri…

Compiling really is the Achilles heel of Rust I feel. This article really talks about the worst case. As others have noted, Haskell and other languages have this property too. Here’s another perspective on practical issues [1] that’s worth reading. Basically, Rust made some understandable but unfortunate design decisions that are hard to walk back or change now. [1]: https://pingcap.com/blog/rust-compilation-model-calamity

> It's not worth it, just learn C or C++. And your productivity on your team plummets as a result. Rust also has slow build times [0], but it didn't stop people from using it. [0] https://pingcap.com/blog/rust-compilation-model-calamity

The URL of this article was changed to the following (so the one linked on Hacker News as of when I posted this comment is now a 404). https://pingcap.com/blog/Rust-s-Huge-Compilation-Units

The following link shows how we do Chaos Engineering in TiDB, an open source distributed database: https://www.pingcap.com/blog/chaos-practice-in-tidb/ Regarding the Fault Injection tools we are using: - Kernel Fault Injection, the Fault Injection Framework included in Linux kernel, you can use to implement simple fault injections to test device drivers. - SystemTap, a scripting language and tool diagnose of a performance or functional problem. - Fail, gofail for go and fail-rs for Rust - Namazu: a programmable fuzzy scheduler to test a distributed system. We also built our own Automatic Chaos platform, Schrodinger, to automate all these tests to improve both efficiency and coverage

Morgan from the TiDB team here. Thank you for the feedback, and I agree with you. We actually took this line out from the same copy in the docs: https://pingcap.com/docs/ (We must have missed a spot, and I will follow up and make sure it is addressed). We try to be transparent about the differences from MySQL. On the compatibility page, there are a few cases described such as large transactions, small transactions and single threaded workloads: https://www.pingcap.com/docs/sql/mysql-compatibility/

NewSQL databases still may use consensus to point to a leader for data, its just that data is transparently partitioned and each partition can have a different leader. In TiDB for example each region (data partition) has its own leader and only that leader can serve writes (but the leader can change via a raft-based election). By default all servers are both leaders of some partitions and followers of others [1]. Putting all your data on one master stops scaling at some point when data is large enough. However, even before that point, even without high-availability requirements, you are making trade-offs which may be good or bad (e.g. throughput is limited on a single master, but latency is low). [1]: https://pingcap.github.io/docs/tikv/tikv-overview/

Also take a look at https://www.pingcap.com/ for sql scale-out and http://www.scylladb.com/ for cassandra-but-fast.

Brian Anderson wrote up his thoughts here, and it's a good intro to the topic: https://www.pingcap.com/blog/rust-compilation-model-calamity/ Let's dig into this bit of that, to give you some more color: > Split compiler/package manager — although it is normal for languages to have a package manager separate from the compiler, in Rust at least this results in both cargo and rustc having imperfect and redundant information about the overall compilation pipeline. As more parts of the pipeline are short-circuited for efficiency, more metadata needs to be transferred between instances of the compiler, mostly through the filesystem, which has overhead. > Per-compilation-unit code-generation — rustc generates machine code each time it compiles a crate, but it doesn’t need to — with most Rust projects being statically linked, the machine code isn’t needed until the final link step. There may be efficiencies to be achieved by completely separating analysis and code generation. Rust decided to go with the classic separate compilation model that languages like C use. Let's talk about that compared to Zig, since it was already brought up in this thread. So imagine we have a project, A, and it depends on B. B is a huge library, 200,000 lines of code, but we only use one function from it in A, and that function is ten lines. Yes, this is probably a bad project management decision, but we're using extremes here to make a point. Cargo will compile B first, and then A, and then link…

I do want to clarify a few points, on the project page it does provide the following information: > Distributed Transactions: TiDB uses a two-phase commit protocol to ensure ACID compliance, providing strong consistency. Transactions span multiple nodes, and TiDB's distributed nature ensures data correctness even in the presence of network partitions or node failures. > High Availability: Built-in Raft consensus protocol ensures reliability and automated failover. Data is stored in multiple replicas, and transactions are committed only after writing to the majority of replicas, guaranteeing strong consistency and availability, even if some replicas fail. Geographic placement of replicas can be configured for different disaster tolerance levels. See https://github.com/pingcap/tidb?tab=readme-ov-file#key-features Correctness has been a focus for a long time for TiDB, including working on passing Jepsen Tests back in 2019, see https://www.pingcap.com/blog/tidb-passes-jepsen-test-for-snapshot-isol… and https://jepsen.io/analyses/tidb-2.1.7 Disclosure: Employee of PingCAP the company behind TiDB

One of the founder of TiDB/TiKV here from PingCAP (https://pingcap.com) I have been thinking about this problem with my peers when I started to build TiDB (https://github.com/pingcap/tidb) seven years ago. At that time, nearly all of us were familiar with Go language, so we decided to use Go to build the SQL layer of TiDB. Thanks to Go, we could develop TiDB very quickly and released the first MVP in half a year. I remembered clearly the sense when we ran TPC-C successfully, although the TPMC was just 1 at that time, this was a good start for us. But Go had some problems, e.g. the GC was not good before, the fair scheduling might cause some latency problem, or data racing may happen sometimes. So when we decided to build a distributed storage (aha, TiKV (https://githbu.com/tikv/tikv), we wanted use another language to guarantee safety. I really admire our courage - we chose Rust which was just released 1.0 and missed lots of libraries at that time. Now it seems that this is an awesome choice, TiKV has been graduated from CNCF, and been used as building block not only for TiDB, but also for other distributed systems. Thanks Rust. When TiDB started being used in many companies, we found that our customer not only ran lots of online transactions in TiDB, but also they wanted to ran some realtime analytic queries directly because the data has been in TiDB already. So we decided to build a HTAP database, to introduce a column storage beside TiKV, this is TiFlash…

> So can someone please explain? Maybe what he actually wants is madvise or hugetlbfs ? I think you're getting warm. It's hard to know anything from this confused article but I'll assume they're correct that there was only one process, the pages were filled prior to the load test, and there were many minor page faults later. My best guess is that the page faults are due to the kernel's moving stuff to transparent huge pages in the background (and struggling to find/make unfragmented physical address space). The behavior change was due to the mmap size change they observed: before with the extra bytes the kernel didn't initially use huge pages. After, it did, so no background operation was necessary. (Or maybe MAP_SHARED helped because they had them backed by a non tmpfs filesystem, where huge issues aren't supported. Or maybe some behavior changes were just because they tested multiple times without rebooting and the physical page space got fragmented. Again, hard to know much from this article.) If I were them, I'd confirm this by "perf record" or eBPF and/or by disabling transparent huge page compaction. Then I'd switch to using dedicated huge pages, because huge pages really do speed things up by like 15% for a memory heavy workload; they're worth it if you can avoid THP's sometimes pathological behavior. Edit to add: Here is a much clearer article explaining the same problem. https://pingcap.com/blog/why-we-disable-linux-thp-feature-for-database… I don't agree with…

TiDB developer here. Yes, I think TiKV is an alternative to FDB. Compare to FDB Record Layer, TiKV aims to provide a more atomic primitive, just including Get/Set/Transaction in key-value layer, so users can build customized distributed system around it. The main differences between TiKV/TiDB and FDB are: 1. TiKV uses Multi-Raft architecture, I think Raft provides more HA. 2. TiKV's transaction model is inspired by Google Percolator, it's a classical optimistic 2PC transaction model with MVCC support. I'm not a expert of FDB, but I think different transaction models fit for different application scenarios, TiKV's transaction model is good when your workload is mainly small transactions and with a low conflict rate. 3. TiDB is a full-featured SQL layer on top of TiKV, aims to provide a MySQL compatible solution, you know, most of the TiDB users are migrated from the MySQL, so the focus of TiDB will be how to be compatible with these legacy MySQL-based applications. For example, how to read MySQL binlog and then replay on TiDB in real time, let TiDB become a MySQL active replica, or how to support complex SQL queries like distributed join or groupby, you know, building a full-featured SQL optimizer is a huge project. There are some case studies: https://pingcap.com/success-stories/ https://pingcap.com/success-stories/tidb-in-meituan-dianping/ There are some quick-start documents you can start…

Yes, and Linux. at least historically, has not used them without explicit program opt-in. Often advice is to disable transparent huge pages for performance reasons. Not sure about other operating systems. See, for example, https://www.pingcap.com/blog/transparent-huge-pages-why-we-disable-it-…

Isn't gettimeofday implemented with vDSO to avoid kernel context switching (and therefore, most of the overhead)? My understanding is that using tsc directly is tricky. The rate might not be constant, and the rate differs across cores. [1] [1]: https://www.pingcap.com/blog/how-we-trace-a-kv-database-with-less-than…

I'm a big fan of Rust but there are definitely warts that are going to be difficult to cure [1]. This is 5 years old now but I believe it's still largely relevant. It is a weird hill to die on for C/C++ devs though, given header files and templates creating massive compile-time issues that really can't be solved. Google is known for having infrastructure for compiling large projects. They use Blaze (open-sourced at Bazel) to define hermetic builds then use large systems to cache object graphs (for compilation units) and caching compiled objects because Google uses some significant monoliths that would take a significant amount of time to compile from scratch. I wonder what this kind of infrastructure can do for a large Rust project. [1]:https://www.pingcap.com/blog/rust-compilation-model-calamity/

Here's good companion article on the importance of Model Context Protocol (MCP) for AI agents: https://www.pingcap.com/blog/why-fresh-data-sql-and-unified-platforms-…

I don't share OP's hostility to VC-funded databases, but PingCAP is definitely VC-funded: https://www.pingcap.com/press-release/pingcap-the-company-behind-tidb-…

There are some other options as well, like this one: https://www.pingcap.com/tidb-serverless/

This is an example of a system built like you're describing at the end: https://www.pingcap.com/blog/how-an-open-source-distributed-newsql-dat… Also see https://tikv.org/deep-dive/distributed-transaction/timestamp-oracle/ Cockroachdb describes their approach here: https://www.cockroachlabs.com/blog/living-without-atomic-clocks/

https://www.pingcap.com/blog/rust-compilation-model-calamity/ is a good overview. In general it varies depending on the crate but we track the performance at https://perf.rust-lang.org/ - if you look at cargo, for example, over 60% of the time is spent in codegen through LLVM: https://perf.rust-lang.org/detailed-query.html?commit=222d1ff68d5bfe1d…

The home page of the Hacker News Insight site (https://hackernews-insight.vercel.app) gives a bit of background about the stack: it's a TiDB Cloud database (https://www.pingcap.com/tidb-cloud/) and Evidence front-end (https://evidence.dev)

according to https://en.pingcap.com/case-study/lesson-learned-from-queries-over-1-3… it can store 1.3 trillion rows in a single DB. And in one use case: "in this application, data increases by 8 billion records and 1.5 TB every day."

My source: https://pingcap.com/blog/rust-compilation-model-calamity#bootstrapping… > 7 femto-bunnies - rustboot building Rust prior to being retired > 49 kilo-hamsters - rustc building Rust immediately after rustboot's retirement > 188 giga-sloths - rustc building Rust in 2020

Often when building compilers you trade the speed of compilation against the speed of execution[1], and Rust has (traditionally) chosen the speed of executiong over making the compiler fast. This won't necessarily always be so, there's e.g. a new compiler backend called cranelift in the works, that makes compilations 30%ish faster, and in return the code isn't as optimized - which is good for a faster inner cycle. https://pingcap.com/blog/rust-compilation-model-calamity#rusts-designs…

Related posts: - https://pingcap.com/case-studies/how-we-use-a-scale-out-htap-database-… - https://pingcap.com/case-studies/horizontally-scaling-hive-metastore-d…

There is https://pingcap.com/blog/titan-storage-engine-design-and-implementatio… that splits keys from values.