Matthew Rocklin
matthewrocklin.com · Personal blog · on HN since 2013
Matthew Rocklin, creator of Dask, writes about Python data tooling, benchmarks, software practice, and AI.
What HN saysCommenters like the blog and cite his posts on biased benchmarks, writing dumb code, and Dask compared with Spark.
53HN people shared it
12linked it in comments
52stories submitted
758story points
9 months agolast shared
2013people per year2026
Most-shared posts
- 1Biased Benchmarks: honesty is hard5 people · 10 points · first shared 2017 · discussion on HN
- 2AI Zealotry3 people · 4 points · first shared 2026 · 1 comments on HN
- 3GitHub taught me to micromanage2 people · 74 points · first shared 2024 · 61 comments on HN
- 4Pickle isn't slow, it's a protocol2 people · 11 points · first shared 2018 · discussion on HN
- 5Streaming Pandas Dataframes2 people · 8 points · first shared 2017 · discussion on HN
- 6Credit Models in Python with Dask2 people · 7 points · first shared 2018 · discussion on HN
- 7Pandas on HDFS with Dask Dataframes2 people · 6 points · first shared 2016 · 1 comments on HN
- 8Craft Minimal Bug Reports (2018)2 people · 3 points · first shared 2018 · discussion on HN
- 9Pangeo: JupyterHub, Dask, and XArray on the Cloud (2018)2 people · 2 points · first shared 2019 · discussion on HN
- 10Write Dumb Code2 people · 2 points · first shared 2018 · discussion on HN
- 11Claude Chic2 people · 2 points · first shared 2026 · discussion on HN
- 12Dask distributed2 people · 1 points · first shared 2016 · discussion on HN
- 13Complex Graphs2 people · first shared 2015
- 14Avoid Indirection in Code1 people · 263 points · first shared 2019 · 216 comments on HN
- 15Write Dumb Code (2018)1 people · 113 points · first shared 2024 · 76 comments on HN
What HN people wrote when they linked it
kndjckt · 2020-02-08
I love Matthew Rocklin’s (the maintainer of Dask) blog https://matthewrocklin.com/
shoyer · 2021-03-16
Hi Peter, good to run into you again :)
I agree, formulating open benchmarks is great work, and there's nothing suspicious about performing well on benchmarks you write. It's just worth keeping in mind: https://matthewrocklin.com/blog/work/2017/03/09/biased-benchmarks
My initial remarks were a little careless and overly provocative! I don't doubt that there are cases where a true "array database" provides value. I do think the use-cases are less clear for arrays than they are for tabular data, because the users of arrays tend to be more sophisticated.
mrocklin · 2026-01-10
If you read on in the post you might be interested in the section titled
Drop Python: Use Rust and Typescript
https://matthewrocklin.com/ai-zealotry/#big-idea-drop-python-use-rust-…
IanOzsvald · 2023-01-08
This https://docs.dask.org/en/stable/spark.html notes
"However, Dask is able to easily represent far more complex algorithms and expose the creation of these algorithms to normal users [compared to spark]" linking to:
http://matthewrocklin.com/blog/work/2015/06/26/Complex-Graphs
MrPowers · 2021-12-10
I work on Dask now and the creator summarizes his preferred code style nicely in this post: https://matthewrocklin.com/blog/work/2018/01/27/write-dumb-code. I like working on simple codebases like this.
The only Typelevel library I've used is frameless, which I blogged about here: https://mungingdata.com/apache-spark/frameless-typed-datasets/. Don't think the lib is really usable. Typelevel stuff is not what I'm referring to when I say better Python. I'll try to make another blog to clarify.
I think Scala is a better language than Python. But for some reason it just seems like Scala is full of mind bending code that I can't understand. Full disclosure: I am way less smart than lots of the Scala ppl I've worked with.
makmanalp · 2021-07-31
FWIW the scientific computing community (who often deal with petabytes of geodata) has been thinking of ideas like this for a while, e.g. techniques around file formats that are easy to lazily and partially parse, (ab)using FUSE to do partial reads using http RANGE requests, some combination thereof, etc:
http://matthewrocklin.com/blog/work/2018/02/06/hdf-in-the-cloud
pwang · 2016-04-21
BUT WHY
Your performance is going to be complete and utter crap because you're paying for serialization on every single data element.
Dask is higher performance and more pythonic: http://matthewrocklin.com/blog/work/2016/02/22/dask-distributed-part-2
pixelmonkey · 2016-04-12
Very promising IMO. Lots of new stuff going on in their 'distributed' project:
- https://github.com/dask/distributed
- http://matthewrocklin.com/blog/work/2016/02/17/dask-distributed-part1
math_and_stuff · 2015-12-29
Distributed [1] is very new and seems to have similar core architectural goals as TensorFlow. But perhaps I'm being too politically correct: both make use of very course-grain parallelism relative to a typical distributed-memory linear algebra library (e.g., the current communication mechanisms of both are likely to be too course-grain to efficiently support distributed dense matrix inversion or eigensolvers; not that this is likely to be a design goal of either).
[1] http://matthewrocklin.com/blog/work/2015/06/23/Distributed/
Nrpf · 2015-11-15
Thanks for your honesty re the web stuff.
I agree multi dispatch is awesome, and python has some implementations of it as well.
http://matthewrocklin.com/blog/work/2014/02/25/Multiple-Dispatch/
mathnode · 2015-09-22
Yup. Cytoolz. Some notes on performance of Cytoolz:
http://matthewrocklin.com/blog/work/2014/05/01/Introducing-CyToolz/
Lofkin · 2015-06-30
You forgot python, with package: (similar in that R needs parallel packages): http://matthewrocklin.com/blog/work/2015/06/26/Complex-Graphs/