Adam Drake

The "bash is 235x faster than Hadoop" article pops up on HN every so often and this is another great time to link it here: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

Congratulations on learning about distributed electronic computers. (This is worth tinkering with. This is how people actually get good at HPC.) Pay attention to your SRAM (L3 unified cache), DRAM and swap space tilings. [Snark] In practice: With memory access latency depending on both the square root of the memory size and the physical lengths of the wires in your cluster this sounds like a case for Adam Drake: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

Apt time to mention the classic "Command-line Tools can be 235x Faster than your Hadoop Cluster", for those who may have not yet read it. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

I was once interviewed by Lyft and I told them that a 64 GB Mac Book Pro is sufficient to run the core business. The interviewer didn't believe me. I did the calculations in front of him. This was the time, I just learned Rust. I did the full analysis and showed him that QPS is great. Of course, my design didn't include 500 microservices, 50 deployments of Hadoop, Kafka, Elastic, Mongo DB, and 500 slightly different dashboards that different teams in Lyft probably use. Most modern tooling in ZIRP phenomenen. Someone's Hadoop experience - https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

> Pipe this into that and through seven pieces of software that sound like glibgcd, add 8 arcane flags and in the end you have some kind of textfile that would've just as easily been made in some handmade program? What exactly is the selling point for devs? No, that's exactly the selling point. Yes, you could write some hand made program. And piping software together IS a kind of hand made program. It's just going to be far faster to write that pipeline, than to write a custom program, deploy, and run it. And the pipeline might be much much faster. A classic example: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

The GNU coreutils never fail to impress me with how insanely fast they go. One of my favorite blog posts of all time [1] really opened my eyes to how insanely optimized Unix really is. [1] https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

I was interviewing for software jobs recently, and while I was studying up on the "system design" portion I kept circling around the same insight that Dan Luu writes about so well here. I would sit down at an interview and try to create these "proper" system designs with boxes and arrows and failovers and caches and well tuned databases. But in the back of my mind I kept thinking, "didn't Facebook scale to a billion users with PHP, MySQL, and Memcache?" It reminds me of "Command-line Tools can be 235x Faster than your Hadoop Cluster" at https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-… , and the occasional post by https://rachelbythebay.com/w/ where she builds a box that's just fast and with very basic tooling (and a lot of know-how).

See the classic of the genre, "Command-line Tools can be 235x Faster than your Hadoop Cluster" https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

This is great. It reminds of that Adam Drakes’ 2014 article “Command-line Tools can be 235x Faster than your Hadoop Cluster” [1] was the wake up call I needed. 1. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

I run into the microservices obsession very often in the high-growth startups I advise, so often that I wrote Enough With The Microservices (https://adamdrake.com/enough-with-the-microservices.html). I typically see people try to push microservices as a "best practice" for improving the delivery rate of software teams. They carve the codebase into separate services, but still leave all of the persistence in place so that all services are communicating with the same DB(s). The result is a tangled web of interdependent services, plus additional tooling like Docker, Kubernetes, etc. that actually makes the teams even slower than they were before. I'm glad to see us as a community getting more pragmatic about these topics, and realizing that _best practices_ are highly context dependent!

Ha, I was explaining this just yesterday to a few people at $WORK (I'm not at TB, though we do use a bunch of Zig in prod). "Multiprocessing" (multiple CPU cores independently executing and only able to coordinate via some message-passing system -- a definition which encompasses both multi-core CPUs and horizontally scaled distributed systems, contrasting slightly with the normal definition) is challenging for a few reasons. Firstly, the details are an open math question, but I'll blindly state that some problems aren't amenable to parallelization. I.e., no algorithm can meaningfully improve performance via parallelization no matter the implementation. Think through how you would more quickly compute hash(hash(hash(hash(...)))) for example. The serial dependency makes things challenging. That isn't too dissimilar to the problem TB faces. Secondly, message passing is expensive. If the only way two CPU cores can coordinate is through a multi-level cache, at best you're incurring ~tens of nanoseconds of latency per message. Contrast that with a base rate of 512 bytes processed per nanosecond with enough attention to detail on typical modern server hardware (4 pipelined AVX512 instructions at 2GHz). Messages are several orders of magnitude worse than your normal work, so if you need very many of them then you're hosed from a performance perspective (worse with longer delays, like networked computers). Even very parallelizable problems at an abstract level can suffer…

>I mean if you're doing data science the data is not always organized and of course you would want multi-processing Not necessarily - I might not want it or need it. It's a few TB, it can be on a fast HD, on an even faster SSD, or even in memory. I can crunch them quite fast even with basic linear scripts/tools. And organized could just mean some massaging or just having them in csv format. This is already the same rushed notions about "needing this" and "must have that" that the OP describes people jumping to, that leads them to suggest huge setups, distributed processing, multi-machine infrastructure, for use cases and data sizes that could fit on a single server with redundancy and be done it. DHH has often written about this for their Basecamp needs (scalling vertically where others scale horizontally having worked for them for most of their operation), there's also this classic post: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-… >1 TB of memory is like 5 grand from a quick Google search then you probably need specialized motherboards. Not that specialized, I've work with server deployments (HP) with 1, 1.5 and 2TB RAM (and > 100 cores), it's trivial to get. And 5 or even 30 grand would still be cheaper (and more effective and simpler) than the "big data" setups some of those candidates have in mind.

I missed the edit window on this but: I don't think you're going to get computers to not be messy and full of huge time sinks. That's just the nature of humanity being reflected in a silicon mirror. I like the idea of focusing first and foremost on debuggability, on manipulating program execution histories as first-class objects. There are a huge number of interesting tradeoffs there with performance. What I mean by "power" is that Unix makes it easy enough to do things like "interactive text chat in three lines of code" (untested, and quality at best "usable prototype" rather than "production ready") #!/bin/bash ch="$ {2?Usage: $0 /tmp/channelfile}" touch "$ch"; chmod 666 "$ch"; tail -f "$ch" & # chat of the beast while read -e line; do echo "<$USER> $line"; done >> "$ch" or "which other English words might be Greek neuters, plural and all, like automaton/automata" grep "on$" /usr/share/dict/words | sed 's/..$/a/' | sort | join - <(sort /usr/share/dict/words) or "which directory is growing in my home directory and filling up the fucking disk" du -k ~ > tmp.du.1 du -k ~ > tmp.du.2 diff -u tmp.du.[12] or "build the thumbnails for any images I haven't thumbnailed yet", at least as long as none of them have spaces in the names, using GNU make (untested, but adapted from real code): thumbnails: $(patsubst %.jpg,thumbs/%.jpg,$(wildcard *.jpg)) thumbs/%.jpg: %.jpg convert -geometry 80x160 $< $@ or…

> syslog-ng supports the systemd journal natively so you'd never see a difference from before if you don't want to. We use rsyslog with systemd and it works too, however it's just another level of abstraction so, I'm not fond of piling stuff over and over. Binary logs' and journals' usefulness can be also debated. It of course brings some advantages to the table but, it doesn't enable anything groundbreaking for me. > The principle is dogma that has held back Unix imo. It's useful for simple CLIs but that is about it. By your definition the kernel itself goes against the principle. Actually I don't consider it dogma, because it has some very important results and corollaries. This principle reduces secondary complexity (glue logic, complexity required to do many things with one code base, etc.) greatly hence the software can be relentlessly optimized. grep, find, sed, awk, cut, head, tail, less all are extremely performant tools. I develop high performance scientific applications and making a tool accomplish more than a couple different functions either reduce performance (generalized algorithms) or increase size a lot (specialized functions for everything). Its refactoring, verification, maintenance gets complicated and expensive fast. UNIX pipes and small optimized tools are much more useful beyond simple CLI commands and small scripts. Since the tools are fast, low memory footprint performance behemoths, you can build very fast and reliable machinery with them. You…

All the new shiny. From experience I can tell you a) most companies deal with small amounts of data. Small can mean dozens of megabytes to dozens or hundreds of gigabytes. A single well provisioned server will typically be able to handle that very well. Also an SQL database can do a great deal if you know what you're doing. b) inappropriately used big data frameworks are expensive performance killers. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-… for example. c) Good quality programming, as in understanding the machine, memory layout and why it matters, and a good understanding of algorithms (and a hefty dose of common sense), will often yield you more speedup than buying almost any number of new machines. c) Hiring is often driven by fads and companies often don't like being told 'you don't need this roomful of servers', they like to waste money, so maybe do learn them (the profligacy with money is likely to be coming to an end with the economic damage of covid). Takeaway: brainpower will get you much further than horsepower

This reminds me a bit of "Command-line Tools can be 235x Faster than your Hadoop Cluster" from 2014 https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-… The constraint I, and I bet many here, have is just how much data there is. 3GB like in the 2014 article is one .pdf Enterprise level data store is measured in hundreds of GB for a single customer, and you'll get murdered on data egress costs if you try to search an entire corpus, if you can even get through it all before the request times out or the customer decides after 5 minutes that enough is enough. You'd need a true distributed filesystem to even start attempting what the authors suggest at any scale outside of your local machine.

made me remember this article <https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-… Command-line Tools can be 235x Faster than your Hadoop Cluster (2014) Conclusion: Hopefully this has illustrated some points about using and abusing tools like Hadoop for data processing tasks that can better be accomplished on a single machine with simple shell commands and tools.

Here are clickable links: https://youtu.be/ygr8yvIouZk?t=1307, https://adamdrake.com/from-enterprise-decentralization-to-tokenization…

650GB? Your data is small, fits on my phone. Dump the hyped tooling and just use gnu tools. Here's an oldie on the topic: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

Awk? https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

I think what the OP is asking for is an article _like this one_ about where you go in-depth into what you tried, where the system went, and more specifically what went wrong (even if it's just a list of "undifferentiated issues"). Because "we tried a thing. It didn't work. We bailed out." doesn't show off the rough edges of the tool in a way that helps people understand "the shape of the elephant". Or, in the vein of https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-… - "here's a place I wouldn't use an AI tool because _other thing_ is far better"

This has the same energy of this article named "Command-line Tools can be 235x Faster than your Hadoop Cluster" [1] [1] - https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

The comparison posted may be true, but on some level it doesn't make sense. It's like this old awk-vs-hadoop post https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-… Yes, a very specific implementation will be faster than a generic system which includes network delays and ensures you can handle things larger than your memory in a multi-client system. But the result is meaningless - perl or awk will also be faster here. If you need a database system, you're not going to replace it with K, if you need super fast in-memory calculations, you're not going to use a database system. Apples and oranges.

I wouldn't underestimate how much a modern machine with a bunch of RAM and SSDs can do vs HDFS. This post[1] is now 10 years old and has find + awk running an analysis in 12 seconds (at speed roughly equal to his hard drive) vs Hadoop taking 26 minutes. I've had similar experiences with much bigger datasets at work (think years of per-second manufacturing data across 10ks of sensors). I get that that post is only on 3.5GB, but, consumer SSDs are now much faster at 7.5GB/s vs 270MB/s HDD back when the article was written. Even with only mildly optimised solutions, people are churning through the 1 billion rows (±12GB) challenge in seconds as well. And, if you have the data in memory (not impossible) your bottlenecks won't even be reading speed. [1]: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

Having not read the book I don't know if it delves into the speedups that can come from the fact that pipelines have processes running in parallel. A nice article about how much faster it can be to process data on the command line vs something like Hadoop: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-… Clearly, doesn't work in all cases but often enough, it's not only simpler to set up but runs much faster besides.

Sounds like my experience when tinkering as well. https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

There's a lot of overhead and inefficiencies in how large companies do anything. It's hard to organize thousands of people well. It's the social equivalent of the old 'Command-line Tools can be 235x Faster than your Hadoop Cluster'[1] conclusion. You put a hundred developers on a project and you'll generously get twice the meaningful output of ten developers (but 50 times the LOC; as demanded by conway's law). The upside is that this makes it all the more feasible for a small group or even single individual to be surprisingly competitive. Peak productivity per person is probably is probably found in a self-organized 1-3 person team. [1] https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

There was also the "Hadoop vs. unix pipeline running on a laptop"-story a few years back, a more modern take: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-…

No, not crazy: https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-… ~10 years ago I parsed 15GB of Windows binary logs with xargs, grep and strings(1) in minutes. Minutes. Less than 1/4 of an hour.