SingleStore Blog

SingleStoreDB (formerly MemSQL) (https://singlestore.com) | India, Portugal | Full Time | Remote SingleStoreDB is a database focused on high performance and hybrid workloads (HTAP). Our customers include half of the top 10 US banks, 2 of the top 3 US telcos, and 12% of the Fortune 100. Our product is a distributed, relational database that handles both transactions and real-time analytics at scale. Querying is done through standard SQL drivers and syntax, leveraging a broad ecosystem of drivers and applications. We have been building SingleStore since 2011, and building a database-as-a-service product involves a bunch of interesting technical challenges such as: * Compilers and code generation * Distributed computing and fault tolerance * UI and User Experience * Integrations with connectors and third-party tools (e.g., Business Intelligence tools like Tableau) * Infrastructure that scales to dozens of thousands of database clusters across GCP, Azure and AWS We have the following positions open: * Partner Integration Engineer (Portugal, India) * Software Engineer, Managed Services (Portugal, India) Careers page with individual links for each open position: https://www.singlestore.com/careers/jobs/. If you want to learn more about the engineering work we do, check out https://www.singlestore.com/blog/category/engineering/. Feel free to email me directly via david at singlestore dot com.

The mixed column and row store HTAP approach is very similar to how single store works (https://www.singlestore.com/) which offers a mysql compatible interface with a few small differences.

Beyond being closed source, SingleStoreDB's License explicitly prohibits benchmarking: https://www.singlestore.com/assets/contracts/singlestore-free-tier-sof… 2. Restrictions. You acknowledge that the Software, and its structure, organization, and source code, constitute SingleStore’s and its suppliers’ valuable trade secrets, and the Software is subject to the following restrictions. Specifically, Customer shall not, and shall not ... conduct any competitive analysis, publish or share with any third party any results any results of any technical evaluation or benchmark tests performed on the Software, or disclose Software features, errors or bugs to a third party without SingleStore’s prior written consent (“Benchmarking”); or

> it's a bit difficult to wrap my mind around how Arrow will help in distributed systems Comparing with the role of Protobuf is perhaps easiest, there's a good FAQ entry [0] which concludes: "Arrow and Protobuf complement each other well. For example, Arrow Flight uses gRPC and Protobuf to serialize its commands, while data is serialized using the binary Arrow IPC protocol". This will be increasingly significant due to the hardware trends in network & memory (and ultimately storage too) compared with CPUs. I posted about that in a comment a few days ago [1], but it's worth sharing again: > here’s a chart comparing the throughputs of typical memory, I/O and networking technologies used in servers in 2020 against those technologies in 2023 > Everything got faster, but the relative ratios also completely flipped > memory located remotely across a network link can now be accessed with no penalty in throughput The graphs demonstrate it very clearly: https://blog.enfabrica.net/the-next-step-in-high-performance-distribut… > would be great to read a bit more about some optimizations that would help improving Postgres leaving out pure analytical use cases Unfortunately I don't have a good reference on that to hand but I'll take a look around and reply again soon. [0] https://arrow.apache.org/faq/#how-does-arrow-relate-to-protobuf [1] https://news.ycombinator.com/item?id=37365816 [2] https://www.singlestore.com/comparisons/postgresql/

Clickhouse is great a single table analytical queries (Group-by + aggregates + filters over a single table). It can match any top tier columnstore database at this, maybe even better at smaller scales, and if that is all you need it will do the job well Beyond this, it has a naive query optimizer and limited ability to run distributed joins. This is why you won't find well known analytical benchmark results like TPC-H and TPC-DS for clickhouse vs other SQL data warehouses. SinglestoreDB has been doing real time analytics much longer then clickhouse and has a much more mature feature set at this point[1][2]. Our go to market is more big enterprise driven so we are definitely much less well known among developers. Revenue share-wise I suspect we are the leader in real time analytics (see disclaimer below - I'm biased but our revenue is reaching thresholds for IPO readiness). Also, if you follow Jacks later posts you'll see he replaces DynamoDB and Redis with SinglestoreDB as well. Now were getting into places Clickhouse doesn't tread at all... Disclaimer: I'm one of the cofounders of MemSQL/SingleStoreDB (still working away on making it better all these years later...). [1] https://www.singlestore.com/blog/the-technical-capabilities-your-datab… [2] https://dl.acm.org/doi/abs/10.1145/3514221.3526055

When we talk about how SingleStore unifies transactions and analytics into a single database, and how fast it is at both, people understandably find it hard to believe at first - most database systems today do one or the other well, but not both. Here are numbers to back up our claims, on the industry-standard benchmarks TPC-H (analytics) and TPC-C (transactions). SingleStore delivers state-of-the-art performance on both, compared to leading cloud data warehouses and a leading cloud operational database - SingleStore matches the data warehouses on TPC-H and matches the operational dbs on TPC-C, and does it with excellent scale-out performance in the cloud. On the other hand, the data warehouses couldn't even run TPC-C, and the operational database didn't finish TPC-H within 24 hours. Not every workload needs the full spectrum of transactions and analytics, but modern applications increasingly need at least some of both. We think SingleStore is the general-purpose database system of the future, capable of running a breadth of workloads and unifying multiple disparate domain-specific database systems into one. Check out the results at https://www.singlestore.com/blog/tpc-benchmarking-results/ and if you're interested, try out SingleStore for yourself - we published the benchmark materials in our github repo https://github.com/memsql/benchmarks-tpc

(I used to work at SingleStore, and now work at Antithesis) SingleStore (f.k.a. MemSQL) used lock-free skiplists extensively as the backing storage of their rowstore tables and indexes. Adam Prout (ex CTO) wrote about it here: https://www.singlestore.com/blog/what-is-skiplist-why-skiplist-index-f… When SingleStore added a Columnar storage option (LSM tree), L0 was simply a rowstore table. Since rowstore was already a highly optimized, durable, and large-scale storage engine, it allowed L0 to absorb a highly concurrent transactional write workload. This capability was a key part of SingleStore's ability to handle HTAP workloads. If you want to learn more, take a look at this paper which documents the entire system end-to-end: https://dl.acm.org/doi/10.1145/3514221.3526055

I helped with the initial assessment for a migration from Postgres with Citus to SingleStore. https://www.singlestore.com/made-on/heap/

Singlestore has been doing json -> column expansion for a while as well. https://www.singlestore.com/blog/json-builtins-over-columnstore/ For a colstore database, dealing with json as strings is a big perf hit.

It is the right choice whenever you need linearizability. I can't believe you've never seen a (correct) production lock-free algorithm impl that used SeqCst. Many lock-free algorithms require SeqCst for correctness. Here's a trivial example: hazard pointers. Any thread publishing its hazard pointer must use a StoreLoad barrier (equivalent to SeqCst) to ensure any GC thread scanning the publication list sees its hazard pointer, before it deallocates pointers in the limbo list that didn't appear in the scan. MemSQL actually wrote a blog post on a nasty bug in their database arising from their use of AcqRel for this operation instead of SeqCst: https://www.singlestore.com/blog/common-pitfalls-in-writing-lock-free-…

I'm interested to see when SingleStore will support ARM. https://www.singlestore.com/forum/t/singlestore-for-arm64-m1-macs/2877…

I see, thanks for the context, it seems like a PITA. But given that each database system has its own flavor of SQL, vanilla TPC benchmarks may not work out of the box so one needs to tweak them a bit and this might be what actually disqualifies the published results from all of the clauses from above being applicable. I can also anticipate that combination of clause (2) and (3) is what some that publish the results are also taking advantage of. [1] https://www.oracle.com/mysql/heatwave/performance/ [2] https://www.singlestore.com/blog/tpc-benchmarking-results/ [3] https://docs.pingcap.com/tidb/v6.2/v5.4-performance-benchmarking-with-… [4] https://www.monetdb.org/blogs/learning-from-benchmarking/

there are the TPC benchmarks which try to cover a wide variety of use cases and scenarios and are designed independently from any one engine: https://www.tpc.org/information/benchmarks5.asp you post the results for your own product, others do the same, customers can compare: https://www.singlestore.com/blog/tpc-benchmarking-results/

Here is the database of your dreams: SingleStoreDB. https://www.singlestore.com/customers/fathom/

To add, they are using[2] the relatively recent VBMI2 instructions of AVX512. This article[1] talks about the advantages of VBMI on IceLake released 2021. [1] https://www.singlestore.com/blog/a-programmers-perspective/ comments https://news.ycombinator.com/item?id=28179111 [2] https://news.ycombinator.com/item?id=31522464

SingleStore (formerly MemSQL) | SF, Sunnyvale, Seattle, Lisbon, Kyiv, REMOTE | full-time | https://www.singlestore.co\ m/careers/engineering/ Our primary product is an advanced relational database, with unique performance that makes it indispensible to our customers. We're hiring to multiple roles. I personally work on the Managed Service team, responsible for making our database technology available in all major cloud environments. My team is looking for senior devs with Golang, K8s, and SQL skills. https://www.singlestore.com/careers/engineering/ or email me (profile)