Database Architects

Is there any indication that it’s possible to build subatomic size transistors? Last I checked the data, transistors are already only a few atoms in size (silicon and carbon atoms are somewhere in the 0.3nm range), and it was a widely held opinion that it would stop at that if not much sooner. That would keep Moore’s law alive for a bit longer at best but the end does seem in sight. Even considering all of that the economics seem to have already stagnated in cost for performance.[1] [1] http://databasearchitects.blogspot.com/2023/04/the-great-cpu-stagnatio…

SSDs work in an odd way: they have more capacity than they claim, and when you overwrite some data you're really just writing to some of that spare space while the old version is just marked as free [1, 2]. Admittedly, at that point it's going to be very hard to retrieve the old data (you'd need to bypass the remapping logic presented by the controller circuitry), but it's theoretically possible. If you're paranoid, it's best to rewrite the whole thing even if it's encrypted, ideally several times (at that point a little original data could technically be left over but it's unlikely and it's not going to be feasible to get any of it if it was encrypted). [1] https://databasearchitects.blogspot.com/2021/06/what-every-programmer-… [2] https://news.ycombinator.com/item?id=27572218

> None of the things people care about really get much out of "unified memory". GPUs need a lot of memory bandwidth, but CPUs generally don't and it's rare to find something which is memory bandwidth bound on a CPU that doesn't run better on a GPU to begin with. Not having to copy data between the CPU and GPU is nice on paper but again there isn't much in the way of workloads where that was a significant bottleneck. the bottleneck in lots of database workloads is memory bandwidth. for example, hash join performance with a build side table that doesn't fit in L2 cache. if you analyze this workload with perf, assuming you have a well written hash join implementation, you will see something like 0.1 instructions per cycle, and the memory bandwidth will be completely maxed out. similarly, while there have been some attempts at GPU accelerated databases, they have mostly failed exactly because the cost of moving data from the CPU to the GPU is too high to be worth it. i wish aws and the other cloud providers would offer arm servers with apple m-series levels of memory bandwidth per core, it would be a game changer for analytical databases. i also wish they would offer local NVMe drives with reasonable bandwidth - the current offerings are terrible (https://databasearchitects.blogspot.com/2024/02/ssds-have-become-ridic…

Nice blog. There is also a problem that generally cloud storage is "just unusually slow" (this has been noted by others before, but here is a nice summary of the problem http://databasearchitects.blogspot.com/2024/02/ssds-have-become-ridicu… Having recently added support for storing our incremental indexes in https://github.com/feldera/feldera on S3/object storage (we had NVMe for longer due to obvious performance advantages mentioned in the previous article), we'd be happy for someone to disrupt this space with a better offering ;).

If you would like some exposure therapy: https://databasearchitects.blogspot.com/2024/12/advent-of-code-2024-in… [0] [0] Recent discussion https://news.ycombinator.com/item?id=42577736

this only covers SSDs but I remember this from a previous hackernews thread. It's a very short read. https://databasearchitects.blogspot.com/2021/06/what-every-programmer-…

Why use learning when you can fit? http://databasearchitects.blogspot.com/2019/05/why-use-learning-when-y…

Even after watching these videos and reading lots of articles on the topic, I still find the full C++ memory model extremely hard to understand. However, on x86 there are actually only a couple of things that one needs to understand to write correct lock free code. This is laid out in a blog post: https://databasearchitects.blogspot.com/2020/10/c-concurrency-model-on…

check this blog - http://databasearchitects.blogspot.de/2014/05/trying-out-hyper.html

If you're interested in query compilation, check out the HyPer DBMS prototype at [1]. It efficiently compiles SQL (SQL92++) to super-fast LLVM code (faster than Vitesse). It's not open source, but you can download a binary [4] or try it out online using the "WebInterface" link and read about the query compiler here [2]. It also has lots of other goodies, like SIMD-powered CSV parsing (about 4x faster than Postgres) [3]. Rumor has it that the prototype will be commercialized soon ;) EDIT: It also has NUMA-aware multi-threaded query execution. [1] http://www.hyper-db.com [2] http://www.vldb.org/pvldb/vol4/p539-neumann.pdf [3] http://www.vldb.org/pvldb/vol6/p1702-muehlbauer.pdf [4] http://databasearchitects.blogspot.de/2014/05/trying-out-hyper.html

Concerning your critique of the in your opinion overly hard scientific review process. According to her blog the paper was rejected because: "Reviewers at conferences [...] have been very skeptical, and pressed us multiple times to produce some kind of comparison of basic block versioning against tracing."[1] I think this is a very valid concern. When you describe a new scientific approach (Basic Block Versioning) you should compare to the state of the art. Otherwise it is very hard for the reader to judge the merit of the new approach. However I agree with your sentiment that a new approach should not only be judged by performance numbers (there is actually a nice article going deeper on this topic on databasearchitects[2]). But there should at least be a more thorough theoretical discussion than only 3-4 sentences. Benchmarks comparing to existing approaches in this case can help to show pathological cases which might indicate weaknesses of the approach or to empirically demonstrate the feasibility of the new approach. [1] http://pointersgonewild.wordpress.com/2014/11/14/the-fastest-for-in-lo… [2] http://databasearchitects.blogspot.de/2014/09/experiments-hurt-review-… EDIT: Many conferences also allow publishing papers without a deep comparison to exisiting research in the industrial session. This allows demonstrations of interesting implementation variants or system choices EDIT 2: The review critisim in older blog posts "Conference…