L
Live Free or Dichotomize
livefreeordichotomize.com · Personal blog · on HN since 2017
A data scientist writes about R, statistics, causal inference, and practical data processing, including the well-known 25TB awk and R post.
What HN saysHN cites the 25TB post as evidence that awk and Unix tools can process very large datasets efficiently, as an alternative to Spark.
17HN people shared it
4linked it in comments
13stories submitted
468story points
2 years agolast shared
2017people per year2026
Most-shared posts
- 1Using AWK and R to parse 25TB7 people · 327 points · first shared 2019 · 103 comments on HN
- 2Using Awk and R to parse 25tb (2019)3 people · 2 points · first shared 2022 · discussion on HN
- 3Using Awk and R to parse 25tb (2019)2 people · 12 points · first shared 2023 · discussion on HN
- 4Hill for the data scientist: an xkcd story1 people · 122 points · first shared 2017 · 11 comments on HN
- 5Show HN: Building a data-driven CV with R1 people · 2 points · first shared 2019 · discussion on HN
- 6Understanding propensity score weighting1 people · 1 points · first shared 2019 · discussion on HN
- 7Understanding Propensity Score Weighting1 people · 1 points · first shared 2024 · discussion on HN
- 8Why you maybe shouldn't care about that p-value1 people · 1 points · first shared 2017 · discussion on HN
What HN people wrote when they linked it
wodenokoto · 2022-10-16
I love Unix pipes and I love the immense computational power you can achieve through streaming pipelines. You hear stories of people processing terabytes of data faster and cheaper than a distributed cloud solution, by the virtue of Unix pies[1].
But I don’t really see them in any other environment. I can’t think of anything that uses something similar except for big, distributed data processing pipelines like Apache Beam.
Where’s the Python with streaming pipelines?
[1] https://livefreeordichotomize.com/posts/2019-06-04-using-awk-and-r-to-…
allarm · 2023-09-08
AWK/Sed are actually very efficient. See this post for example, it describes parsing 25Tb of data with awk: https://livefreeordichotomize.com/posts/2019-06-04-using-awk-and-r-to-…
Chris2048 · 2020-03-23
before you go down the spark route, consider perl/unix-tools may do this kind of thing quite well: https://livefreeordichotomize.com/2019/06/04/using_awk_and_r_to_parse_…
coldtea · 2024-05-29
>How would running awk for analysis on 6TB of data work quickly and efficiently?
In that 6TB is not that huge of an amount
That's their total dataset, and there's no "real time" requirement.
They can start a batch process, process the data, and be done with it.
Here's an example of someone using awk (read further down for the relevant section):
https://livefreeordichotomize.com/posts/2019-06-04-using-awk-and-r-to-…
"I was now able to process a whole 5 terabyte batch in just a few hours."
>They say it would go into memory but its not clear to me how that would work as would still have paging and thrashing issues if the data didnt have often used sections of the data
There's no need to have paging and thrashing issues if you can fit all (or even most) of your data in memory. And you can always also split, process, and aggregate partial results.
>am I overthinking it and they were they just referring to buying a big ass Ram machine?
Yeah, they said one can buy a machine with several TB of memory.