Aaron Drake

Amen right back at ya! (I love the O'Reilly book cover.) I highly recommend people read your blog post. And there's also this classic: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-ha… I'll also take this opportunity to plug Make and Drake for manipulating data in a replicable way: https://bost.ocks.org/mike/make/ https://github.com/Factual/drake If you're processing data using tools that cannot trace their ancestry directly to some time before 1985, you're probably wasting your own and your colleagues' time.

Also GNU utils http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had… The above experiment (which has an interesting github repo) is somewhat over (and real world unusable), but still is eye opening. Hadoop and Spark bring so much complexity that looking for simpler solutions is something worth considering.

Previous comments (2013): https://news.ycombinator.com/item?id=6398650 Also worth a read, "Command-line tools can be 235x faster than your Hadoop cluster (2014)": http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had… | https://news.ycombinator.com/item?id=8908462

In this order, I recommend: 1. Bash. Can you get the answer you need with awk, sed, sort, etc.? Linux commands behave nicely with CPU and RAM, and "Command-line Tools can be 235x Faster than your Hadoop Cluster" (https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-ha… 2. SQLite. The maximum database size is 140 terabytes, and SQLite can join different database files together. First I would recommend using bash to cull your data down before import, if possible (https://www.sqlite.org/limits.html). 3. PostgreSQL. Again, I would recommend using bash to cull your data down before import, if possible (https://www.postgresql.org/about/). Maximum Database Size Unlimited Maximum Table Size 32 TB Maximum Row Size 1.6 TB Maximum Field Size 1 GB Maximum Rows per Table Unlimited Maximum Columns per Table 250 - 1600 depending on column types Maximum Indexes per Table Unlimited More: "Bane's Law (Going Deep)" (https://news.ycombinator.com/item?id=8902739)

Actually, this has been false or at least highly questionable for quite some time. "Command-line tools can be faster than your Hadoop cluster" http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had… https://news.ycombinator.com/item?id=8908462

250GB data is tiny. There are laptops with 128GB RAM, let alone servers. Obligatory read: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-ha…

Hi HN! I was working on a project some time ago where I created a web page (http://tinysite.aadrake.com) deliverable in a single IP datagram and needed a way to see the size of compressed data I was using for testing. I created a utility for this, and now it has a frontend (not for mobile) and lives at https://compresstest.com. The utility is a custom TCP server written in Go, for those who are curious, and the more detailed writeup is available on my website (https://aadrake.com/posts/2017-10-29-the-biggest-smallest-website.html… Hope it's helpful!

Well, if you are taking many hours to process said data on a single processor machine (say eight cores) then throwing it into an environment where you can crunch it with many cores (say 1440) then this does make some kind of sense Yes but https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-ha…

>It's that in some scenarios python can't be made fast. Can you give some examples of this? I mean, obviously with enough effort you can "make python fast" since it has good C bindings, and can just be a thin wrapper around fast stuff. Similar to how command line tools can be ridiculously fast[^1] despite, ostensibly, running in bash. So I'm a bit confused about what you're claiming. Organizational issues, it's difficult to get management on board with an optimization pass? [^1]: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-ha…

Related: https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-ha…

Your Shell and Awk example is funny since simple, single-CPU tools are often faster than the Big Data tools the industry is standardizing on: http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had… http://www.frankmcsherry.org/assets/COST.pdf It's true, though, that you should use a more complex tool if it's necessary to get the job done. It's just not clear that the complexity users or developers are dealing with are intrinsic to the problems being solved. It seems more social than anything. :)

There's also this [0] on how to outperform Hadoop with command line tools. Obviously, you should look at the role I/O costs can play. A lot of RAM and some SSD drives might be a better idea than Hadoop at this scale. [0] http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had…

Yes, I sent him this to read, http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had… and he changed his mind :-)

At what point is using Storm or any other alternative inevitable? I am always reminded of [1] in these cases. Have you found instances where an implementation of the core logic in C/C++ (maybe with thread parallelism) is fast enough? 1: http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had…

> Or maybe there is a totally simple way and this could take me 5 minutes! At that paragraph I immediately thought of http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had…

I think you're talking about this: http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had… I quite enjoyed it as well.

If there's not so many data, awk and sed will do the job. Taking the story to extreme, modern day developer should take a Hadoop (despite the fact it would be 235x time slower than awk - http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had… and old timer should take some APL descendant like Q with kdb+.

Actually, piping data between standard Unix tools can be extremely efficient. This example is working on gigabytes rather than terabytes of data, but memory utilization is basically limited to just buffers, so you could definitely scale that to terabytes without a problem. http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had…

Depends on your familiarity with the unix toolset vs use cases, an amusing (if somewhat contrived) example of this: http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had… Most CSV analysis is static, not something ongoing with inserts/updates/etc, so using a combination of unix tools to handle the data processing seems like a sufficient solution for many scenarios.

I would not use Apache Spark when the amount of data is this little. I have myself done this mistake when processing only a few gigabytes. Take a look at this: http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-had… IMHO I think people abuse Spark and I would be truly impressed if anybody could write a Spark program faster then just a regular Scala program for processing this.

It feels like that dev time cost is instead being crowdfunded by energy, disk, CPU and RAM that every user has to pay with and eventually that cost is paid by Earth itself. An app might be cheap or free but in the long time cheaper for the planet and each user would be an app costing a dollar or two more per install. I was using a 2011 laptop with only an HDD, 2 gigs of RAM and no discrete GPU until it actually went and finally broke last year so it irks me especially when I see this handwaving of "computers are cheap", especially from Westerner millenials or developers from SF. The fact that well off people who change machines every few years can even dare to call less well off people with older and shittier hardware "entitled" for wanting performant snappy software just like they got 5-10 years ago when they bought their machine is baffling. Not everyone needs a crazy and new machine, writers, reviewers, sales people, admins, etc. Case in point - G.R.R. Martin uses a DOS machine to write - https://www.youtube.com/watch?v=X5REM-3nWHg . I.e. Slack and Atom got absolutely lambasted for performance, sluggishness and resource use (while VS Code was applauded, so it's clearly not an Electron specific thing) despite being made by companies valued in billions and based in the most expensive region of the world, one of them even being a paid product. Or a game with pixel art (I do like it and I understand that particular indie dev optimizing for time with such a niche product so…

Thanks for writing this. I think many of your observations are correct (and agree with them). I'm not as sold on your solutions though. Here's some thoughts: Abstractions are a correct way to do things when we don't know what "correct" is or need to deal with lots of unknowns in a general way. And then when you need to aggregate lots of different abstractions together, it's often easier to sit yet another abstraction on top of that. However, in many cases we have enough experience to know what we really need. There's no shame in capturing this knowledge and then "specializing" again to optimize things. In the grand 'ol days, this also meant that the hardware was a true partner of the software rather than an IP restricted piece of black magic sealed behind 20 layers of firewalled software. (At first this wasn't entirely true, some vendors like Atari were almost allergic to letting developers know what was going on, but the trend reversed for a while). Did you want to write to a pixel on the screen? Just copy a few bytes to the RAM addresses that contained the screen data and the next go around on drawing the screen it would show up. Sometime in the late 90s the pendulum started to swing back and now it feels like we're almost at full tilt the wrong way again despite all the demonstrations to the contrary that it was the wrong way to do things, paradoxically this seemed to happen after the open source revolution transformed software. In the meanwhile, layers and layers…

While awk is indeed under-appreciated, there are many instances where using grep to pre-filter your input is helpful because the grep family (GNU grep, agrep, etc.) can match strings much much faster than awk's regex engine. For example: GNU grep uses a heavily optimized implementation of the Boyer-Moore string search algorithm [1] which (it is claimed) requires just 3 (x86) cycles per byte of input. Boyer-Moore only searches for literal strings, but grep will extract the longest literal from the search regex (e.g. the string "foo" from the regex /foo(bar|baz)+/) and use Boyer-Moore to find candidate matches that are then verified using the regex engine. So if you have a large corpus of text to search, grep is your friend. It can easily saturate your I/O bandwidth and perform an order of magnitude faster than typical "big data" platforms. [2] [1] https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310… [2] https://aadrake.com/command-line-tools-can-be-235x-faster-than-your-ha…