Itamar Turner-Trauring
pythonspeed.com · Personal blog · on HN since 2019
Itamar Turner-Trauring writes about Python performance, Docker packaging, and data processing, aimed at developers shipping Python in production.
What HN saysHN commenters call it an excellent resource for Python in containers and data science, and cite its posts on Alpine being slow for Python images.
112HN people shared it
42linked it in comments
349stories submitted
4.4Kstory points
2 months agolast shared
2019people per year2026
Most-shared posts
- 1Alpine makes Python Docker builds slower, and images larger13 people · 377 points · first shared 2020 · 148 comments on HN
- 2When Python can’t thread: a deep-dive into the GIL’s impact8 people · 21 points · first shared 2022 · 1 comments on HN
- 3The Performance Overhead of Python C Extensions6 people · 17 points · first shared 2021 · 2 comments on HN
- 4All Pythons are slow, but some are faster than others6 people · 15 points · first shared 2021 · discussion on HN
- 5Production-Ready Docker Images6 people · 14 points · first shared 2019 · 2 comments on HN
- 6The fastest way to read a CSV in Pandas5 people · 10 points · first shared 2022 · discussion on HN
- 7CI for performance: Reliable benchmarking in noisy environments5 people · 5 points · first shared 2020 · discussion on HN
- 8Why your multiprocessing Pool is stuck (it’s full of sharks)5 people · 3 points · first shared 2020 · discussion on HN
- 9The security scanner that cried wolf4 people · 106 points · first shared 2021 · 65 comments on HN
- 10Should you use uv's managed Python in production?4 people · 24 points · first shared 2024 · 2 comments on HN
- 11Python⇒Speed: Speeding Up Cython with SIMD4 people · 15 points · first shared 2023 · discussion on HN
- 12Faster hardware is a bad first solution to slow software4 people · 8 points · first shared 2022 · 1 comments on HN
- 13Cython, Rust, and more: choosing a language for Python extensions4 people · 8 points · first shared 2021 · discussion on HN
- 14Some Reasons to Avoid Cython4 people · 7 points · first shared 2023 · discussion on HN
- 15Python’s Multiprocessing Performance Problem4 people · 7 points · first shared 2023 · discussion on HN
What HN people wrote when they linked it
sterlinm · 2023-06-22
I’ve found the Python Speed [1] site an excellent resource for the quirks of working with Python in containers for data science use cases.
[1] https://www.pythonspeed.com
pella · 2024-06-12
imho:
A false positive CVE list is an issue elsewhere as well, and it's important to understand that there's not much a Docker Postgres maintainer can do if the problem lies in the Debian or Ubuntu package and isn't getting fixed for some reason.
https://github.com/docker-library/faq#why-does-my-security-scanner-sho…
It's also advisable not to use the default settings:
https://pythonspeed.com/articles/docker-security-scanner/ "trivy --ignore-unfixed <image>"
Of course, it is advisable for the image maintainer to rebuild the Docker image weekly or bi-weekly to ensure all recent patches are included.
However, for those who prioritize security, it is best to build the image themselves to guarantee up-to-date packages.
jackbravo · 2024-04-26
I remember there were some performance-related posts a while ago, about alpine running in docker, recommending people to use debian/ubuntu instead:
Slow alpine posts: - https://pythonspeed.com/articles/alpine-docker-python/ - https://superuser.com/questions/1219609/why-is-the-alpine-docker-image…
Pro alpine post: - https://nickjanetakis.com/blog/benchmarking-debian-vs-alpine-as-a-base…
I wonder if this still holds water.
adr1an · 2024-04-10
This depends a lot on the usecase. For example, data scientists prefer using libraries like dask and polars for faster analyses. What you are asking is, in a way, looking under the hood of such libraries. Oftentimes, you will find that under the hood there's C or Rust code.. anyway, check these articles: https://pythonspeed.com/
IshKebab · 2023-12-26
1. Sounds like you have written very few shell scripts if you don't know the issues and think you can just "don't make mistakes" (a classic fallacy). You can Google it and there are a ton of articles. Here's the first one I found which is decent but doesn't even mention some big issues like quoting: https://pythonspeed.com/articles/shell-scripts/
2. Try putting a space in your home directory and let me know how that goes...
3. Most programs don't resort to reading /proc or /sys because it is such a pain! I bet there's a ton of undiscovered vulnerabilities in programs that do.
davidktr · 2023-07-25
And for this you cannot use Python's multiprocessing because ... ? Sure, moving data between processes is slow because of pickling [0]. However, I'm using parallel processing for the things you suggested, and for these it works great.
If I really had the use case and needed threads, I'd much rather use C++ bindings in a Python package than rebuilding the whole thing. Guess it depends on the scale we are talking about.
[0] https://pythonspeed.com/articles/faster-multiprocessing-pickle/
pablogsal · 2022-04-20
Nice! Fil looks like an awesome profiler and is fantastic that works in other platforms. I am super excited to see more cool features and https://pythonspeed.com/sciagraph/ looks fantastic :)
calpaterson · 2021-06-03
Using alpine with Python in production is usually a mistake. Python is considerably slower with musl libc than with glibc and on top of that no one is providing built wheels for musl-based platforms so you need to drag in the entire buildchain to package most projects. Who wants to build numpy from scratch as part of their build pipeline? And having a C compiler present is not great as it allows exploits to escalate their seriousness considerably.
Anyway most of these security scanners are bunkum:
https://pythonspeed.com/articles/docker-security-scanner/
There are not, in fact, a huge number of known and unpatched vulnerabilities in debian stable. If your tool is finding numerous problems, it is time to look more closely at your tool.
hn_acc_2 · 2021-01-14
Another "I want to love Alpine but can't" user here.
Their use of musl libc makes it a very poor choice for running python in a container, because it forces a lot of manual rebuilding since the PyPi wheels don't work on Alpine, among other issues [1]
[1] https://pythonspeed.com/articles/alpine-docker-python/
fantod · 2020-12-27
Itamar Turner-Trauring's blog has a number of good articles on Docker best practices:
https://pythonspeed.com/docker/
amanzi · 2020-02-07
If you're interested in Docker best practices for Python specifically, I highly recommend the following site: https://pythonspeed.com/docker/
I listened to the author on The Python Podcast and learned lots from the discussion, in particular the differences between Alpine and Debian for Python images.
itamarst · 2019-12-10
Multi-GB images still seem... big. Though to be fair in many situations those 60 seconds are meaningless, so might not be worth the time to optimize.
Often images that large are the result of including build toolchains, which you can omit with multi-stage builds (e.g. for Python https://pythonspeed.com/articles/multi-stage-docker-python/.
ainch · 2026-08-20
I think the point is that, from a compiler's perspective, it's not obvious how much you should be allowed to optimise code at the cost of changing the outcomes of floating points maths - do you allow 1e-10, or 1e-6, or 1e-4 level changes? Does your compiler have to run some test calcs to bound the scale of the change introduced by rewriting fp maths? Some compilers will let you opt in to rewriting floating point maths, but that's opt in so users understand that their numeric outputs might change between optimisation levels.
For more, there's a good post on this kind of flag in Rust: https://pythonspeed.com/articles/faster-float-math-rust/
Twirrim · 2025-08-06
The serialisation cost of translating data representations between python and C (or whatever compiled language you're using) is notable. Instead of having the compiled code sit in the centre of a hot loop, it's significantly better to have the loop in the compiled code and call it once
https://pythonspeed.com/articles/python-extension-performance/
jtmcivor · 2025-05-23
IIUC:
* You still need to load all the bytes into memory before passing to msgspec decoding
* You can decode a subset of fields, which is really helpful
* Reusing msgspec decoders saves some cpu cycles https://jcristharif.com/msgspec/perf-tips.html#reuse-encoders-decoders
Slides 17, 18, 19 have an example of the first two points https://pythonspeed.com/pycon2025/slides/#17
user070223 · 2024-10-07
When should you upgrade to Python 3.13? https://pythonspeed.com/articles/upgrade-python-3.13/
Python libraries support https://pyreadiness.org/3.13/
thomasjudge · 2024-07-13
"In Python 3.14, the default will be changed to either “spawn” or “forkserver” (a mostly safer alternative to “fork”)."
- https://pythonspeed.com/articles/python-multiprocessing/
enriquto · 2024-01-12
> offering almost a Python like experience
If you do scientific computing, the fortran experience is much better than python's. You never feel out of place. Multi-dimensional arrays of floats are a native type, loops are fast, and the system is extremely stable.
What does python offer you? Lists, dictionaries, strings (none of which is of any use to scientific computing) and a plethora of slightly incompatible external libraries for dealing with matrices. Worse, there's no hope [0] that your algorithms based in numpy/scipy/numba/tensorflow/torch/jax will run unchanged in a decade, not to say in 50 years.
[0] https://pythonspeed.com/articles/numpy-2/
ftaghn · 2023-07-26
https://pythonspeed.com/articles/alpine-docker-python/
>Using Alpine can make Python Docker builds 50× slower
Musl is a dire mistake.
wenc · 2023-04-03
I don’t know if it’s still true but Wes McKinney’s (Pandas author) rule of thumb for Pandas was “have 5 to 10 times as much as RAM as your dataset”. But he wrote this in 2017 so things may have changed.
That said, this is a 2023 comparison of Pandas and Polars memory usage.
https://pythonspeed.com/articles/polars-memory-pandas/
noloblo · 2023-03-05
what about GIL? https://wiki.python.org/moin/GlobalInterpreterLock
why python can't thread? impact of GIL on python https://pythonspeed.com/articles/python-gil/
JosephRedfern · 2023-03-01
This post links to another post by the same author which goes into this in a little more detail: https://pythonspeed.com/articles/python-multiprocessing/
mikepurvis · 2023-01-13
Just to clarify for anyone who isn't aware, the "compiling issues", at least historically, have been that that Alpine uses musl, and PyPI's manylinux wheels are built against old glibc versions. So stuff like numpy that would trivially and quickly install from whl on glibc distros (like a bare-bones Ubuntu image) trigger compilations and the installation of build-only dependencies on Alpine.
That said, it looks like as of late-2021, at least some projects are offering musllinux wheels as well, per the discussion here: https://github.com/pypa/manylinux/issues/37 (not numpy, though: https://pypi.org/project/numpy/#files)
Here's a longer version of the above, originally published in 2020, but with an update regarding PEP 656: https://pythonspeed.com/articles/alpine-docker-python/
strangattractor · 2022-08-14
Sounds like a dumb interview. Are you sure you want to work there? How can you do any of this if you have never used the libraries. Else it must be so simple and obvious (at least to the interviewer) that they think it is a no brainer. Likely they want you to unwind some loops and use some of Pandas Vectorized routines. But as this article states https://pythonspeed.com/articles/pandas-vectorization/ that is not always easy to do. It takes a lot of work to get good performance. Improve SQL performance - which one? Mysql, Postgres, etc. Sounds like an interview designed by an engineer.
mukundesh · 2022-07-27
Using Cachegrind to get hardware independent performance numbers (https://pythonspeed.com/articles/consistent-benchmarking-in-ci/)
Also used by SQLite in their performance measurement workflow(https://sqlite.org/cpu.html#performance_measurement)
minimaxir · 2022-04-30
The fact that most data analysis/ETL tutorials on the internet have converged on the same CSV/pandas tactics over the past decade is disappointing when newer tools demonstrated here such as DuckDB/Arrow have practical advantages without much code complexity overhead.
This post also links to another discussion about the Parquet data format (https://pythonspeed.com/articles/pandas-read-csv-fast/), also supported by Arrow, which is also extremely useful but I never see anyone talking about it. Granted, Parquet data can't natively be imported into Excel which is likely the main cause.
philshem · 2022-04-30
As of Pandas 1.4, you can use the pyarrow engine for reading a csv
df = pd.read_csv("large.csv", engine="pyarrow")
https://pythonspeed.com/articles/pandas-read-csv-fast/
nikolay · 2022-03-11
Also, isn't Python problematic with musl-libc [0]?
[0]: https://pythonspeed.com/articles/alpine-docker-python/
steppi · 2022-02-19
It's easy to miss, but the second bullet point about "coding in a low-level language directly" links to another of the author's articles entitled Cython, Rust, and more: choosing a language for Python extensions [1] which discusses the pros and cons of using Cython.
I've done a pretty extensive amount of development in Cython. I agree, it's very nice to be able to incrementally type Python code but I've found that using the language to its full potential requires a good understanding of C programming, especially for debugging.
[1] https://pythonspeed.com/articles/rust-cython-python-extensions/
btilly · 2021-12-07
Sorry, but we did all of that 20 years ago, and those approaches remain a lot simpler for basic use cases. You know, like serving 100k dynamic pages/hour database backed site. (That's what I was doing 20 years ago, how about you?)
See https://pythonspeed.com/articles/dont-need-kubernetes/ for some of the reasons NOT to use Kubernetes for this.
With, of course, a giant exception for two cases. The first is, as with the author, if the purpose of using Kubernetes is to learn how to use Kubernetes. The second is if you are aware of the tradeoffs and have specific reason to say that Kubernetes really is appropriate for your circumstance.
And if you think that Kubernetes is always the right approach, then you clearly are NOT aware of the tradeoffs.