Gregory Szorc

My experience personally with python3 (and repeated interactions with about a dozen python programmers, including core contributors) is that python3 does not let you trivially work with streams of bytes, esp if you need to do character set conversions, since a tiny python2 script that I have used for decades for conversion of character streams in terminals has proved to be repeated unportable to python3. The last attempt was much larger, still failed, and they thought they could probably do it, but it would require far more code and was not worth their effort. I'll probably just use rust for that script if python2 ever gets dropped by my distro. Reminds me of https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journey-to-and-…

uv is great, but downloading and installing base python interpreter is not a good feature as it doesn’t fetch that from PSF but from a project on GitHub, that very same project says this is compiled for portability over performance, see https://gregoryszorc.com/docs/python-build-standalone/main/

On the compression side ZSTD is good all around. It'll give similar results at the ultra high compression side as LZMA in terms of ratio and speed but also scales to be a fantastic choice if you just want to pipe something through at a few hundred mbps and still get decent compression. ZSTD is also much cleaner in its multithreading scaling in terms of memory usage and the like. On the decompression side ZSTD is absolutely blazing in comparison to LZMA. Decompression requirements also don't scale based in compression settings (i.e. doing some ultra super duper compression of a big file places no more memory requirement on the client doing the decompression later). In short, ZSTD is great because it can be your reliable go-to regardless of your particularly use case. For more detail check out https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstan…

If you want to codesign and notarize your binary from linux, you can also use rcodesign[0]! I've been using it in production for about a year, and it's been working great so far. [0]: https://gregoryszorc.com/docs/apple-codesign/stable/

Would not recommend using this in 2022, use zstandard or xzip instead. zstandard is faster and slightly better compression at speed selection settings that are equivalent to gzip, in addition to having the ability to compress stuff at a much greater ratio, optionally, if you allow it to take more time and cpu resources. https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstan…

https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-complex-and-… is the best article about CI/CD that I’ve ever read. Gregory gets it.

It seems like in your case the easiest path might be to learn how the language works and see it for yourself. Otherwise, I think these posts are roughly in the genre you want: * http://dtrace.org/blogs/bmc/2018/09/18/falling-in-love-with-rust/ * http://dtrace.org/blogs/bmc/2020/10/11/rust-after-the-honeymoon/ * https://gregoryszorc.com/blog/2021/04/13/rust-is-for-professionals/

Obviously not? But there are trustworthy people working with and integrating that code, there's a good chance they'll notice a hinky commit, and they're very close to having completely reproducible builds—which means that there can be verification that the shipped binary matches the inspected source. https://gregoryszorc.com/blog/2018/06/20/deterministic-firefox-builds/

Because after nearly 10 years this mediocre list is the best they can offer as consolidation for breaking your code, and for much of that time they didn't have everything it does now. If py3k shipped with something like https://gregoryszorc.com/blog/2017/03/13/from-__past__-import-bytes_li… there probably would have been more uptake.

Phabricator is a brilliant piece of software. Very well written code, an emphasis on security, nice UI, tons of features. Once you get used to it, the Differential code review process feels very natural (better than pull requests, even). Further reading: http://gregoryszorc.com/blog/2013/10/14/phabricator-is-awesome/ http://cramer.io/2014/05/03/on-pull-requests/

I blogged about this the other day and would love to hear about your experience! http://gregoryszorc.com/blog/2014/10/13/deterministic-and-minimal-dock…

This is a great blog post about Mercurial and git: http://gregoryszorc.com/blog/2013/05/12/thoughts-on-mercurial-%28and-g… tl;dr: "If you take anything away from this post it should be that Mercurial is a flexible and agile tool."

> The filesystem layer apparently made some really poor life choices NTFS is plenty fast, even for thousands of small files; it is the Windows Defender file system filter driver that slows things down. Specifically, it slows down CloseHandle[1]. > NT's core tech I'm not just talking about NT. I think most of the user mode is great, too. Office blows the pants off most other 'office' suites. D3D is generally very forward-looking, and many extensions are released on D3D first, sometimes years before they're ported to Vulkan (ray-tracing, mesh shaders, descriptor heaps, etc). Windows has had a superb low-latency audio subsystem in WASAPI since Vista, which is something like a decade before Linux got Pipewire. There are many other examples of random cool stuff in Windows that Linux 'rediscovered independently' but Windows got there much earlier just because of the sheer install base and surface area. [1]: https://gregoryszorc.com/blog/2021/04/06/surprisingly-slow/

The thing that put me off of Rye is that it strongly encourages you to use non-standard Python builds. From their philosophy page: https://rye.astral.sh/philosophy/ > No Python Binary Distributions: CPython builds from python.org are completely inadequate. On some platforms you only get an .msi installer, on some you literally only get tarballs. The various Python distributions that became popular over the years are diverging greatly and cause all kinds of nonsense downstream. This is why this Project uses the indygreg standalone builds. I hope that with time someone will start distributing well maintained and reliable Python builds to replace the mess we are dealing with today. And here is info about those particular indygreg builds. https://gregoryszorc.com/docs/python-build-standalone/main/ It is, however, possible to choose a different Python. https://rye.astral.sh/guide/toolchains/ I've never really experienced the problem they are describing. Any official Python build I've gotten from python.org has worked. Every normal old Python I've gotten from my distribution's package manager has worked. Every Python included with an official Docker container image has worked. I'm sure their special builds will also work just fine, but their non-standardness gives me pause. There's even a list of behavior quirks. Why are we adding quirks to something that has no problems? And the fact that the rye philosophy seems so intent on using them, turns me off from it compared to…

There is one thing you can't with most algorithms: prallelize decompression. That's because most compression algorithms use sliding windows to remove repetitive sections. And decompression speed also drops as compression ratio increases. If you transfer over say a 1GBit link then transfer speed is likely the bottleneck as zstd decompression can reach >200MB/s. However if you have a 10GBit link then you are CPU bound on decompression. See e.g. decompression speed at [1]. Bzip2 is not window but block based (level 1 == 100kb blocks, 9 == 900kb blocks iirc). This means that, given enough cores, both compression and decompression can parallelize. At something like 10-20MB/s per core. So somwhere >10 cores you will start to outperform zstd. Granted, that's a very very corner case. But one you might hit with servers. That's how I learned about it. But so far I've converged on zstd for everything. It is usually not worth the hassle to squeeze these last performance bits out. [1] https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstan…

The Python 3 upgrade process for many projects was incredibly painful. "Mercurial’s journey to and reflections on Python 3" should be required reading for anybody with rose-tinted glasses of the migration. https://gregoryszorc.com/blog/2020/01/13/mercurial%27s-journey-to-and-… There was, of course, a Hacker News thread discussing the article, and a fair few people decided to blame the Mercurial developers for handling the migration inelegantly. Because that's how you win over an audience of developers - reassuring them that if Python has a backwards-compatibility break, Python fans will go out of your way to try and blame you for writing bad code. And not, perhaps, the fact that Python was missing things like a u string prefix and % bytestring formatting until 3.3 (2012) and 3.5 (2015!!!) respectively. If I sound peeved, I really loved Python in the 2.x days, and the way the 3.x transition was handled broke my heart and prevented me from using the language for pretty much an entire decade. There are lessons to be learned from the transition, but not if we ignore the real problems that the transition caused. More importantly, we need to recognize that Python is not the Python we know today because of how "well" the transition was handled, but because Numpy and Matplotlib swooped in and gave Python new niches to excel in at just the right time.

Another great article on this theme: https://gregoryszorc.com/blog/2021/04/07/modern-ci-is-too-complex-and-… It is frustrating that we seem to be evolving towards the nesting of one build system inside another, such as "go build" inside "docker build". (One of the projects I work on uses janky, docker compose, docker build, Go build, Make, Rake, and Bundle.) This approach is opaque, coarse-grained, inefficient, unnecessarily sequential, and doesn't materialize "the" build graph as a single entity from which we can identify our software artifacts and their relationships and derive and automate all our workflows. Bazel addresses these problems with a single, canonical, first-class graph, and many projects that use Bazel achieve close-to-ideal build efficiency, scalability, and reproducibility, as well as a good foundation for other workflows. But switching to Bazel is a large undertaking for a project that has already become entangled in the kind of situation described above. Maintaining hand-written or generated BUILD files for dependencies can be a burden. (I led the design of Blaze in 2006, at which time Google was already using BUILD files to declare software artifacts and dependencies in its more primitive predecessor system; we simply could not have achieved Blaze even at Google's then scale otherwise.) I often dream of a future world in which it is expected that compiler writers will provide a formal definition of the toolchain's workflow in a form that…

As the sibling comment says, horizontal scrolling would be an improvement over the status quo. Pinch to zoom would be another nice win. > Can you tell me more about what you use this feed for? You're talking about the one on the github.com home page? Yes, that's the one I'm talking about. I use it to see who's starring repos I'm involved in, who's following me, and what the developers I follow are starring (the feed also has stuff about what people are pushing to repos I watch, but that is not interesting to me because I usually keep track of it by following PRs or issues). Just keeping track of what's going on in the world. Also, now that I have your attention, please please read https://gregoryszorc.com/blog/2020/01/07/problems-with-pull-requests-a… and make life better for those experienced engineers who do per-commit reviews. Finally, I found the feedback handling from the betas somewhat discouraging. At least when I tried it, it just emailed off and there was no follow-up at all. If that process was a bit more conducive, I might have given you this feedback sooner.

This is great. But please note, some of the graphs presented use a common statistics cheat of logarithmic scale along one or more of the axis. Many times, lz4 level 1 isn't just 25% faster [0] in Compression vs. say zlib level 1 (as it appears visually), it's actually 6 times (6x, or 600%) faster in terms of MB/s (~600 MB/s vs. ~100 MB/s)! And it's not just 5% or so faster than zstd in Decompression (visually) [1], it's about twice as fast (~2000 MB/s vs. ~1000 MB/s). Yes. It does not compress as well. But ~3.2:1 compression ratio best case for lz4 (listed as ~3.2 in the chart) compresses almost as quickly as the 6:1 zstd [0] but will then decompress twice as fast as zstd at any level [1]. Having a high-compression is a case I could (almost) not care about. If you're youtube/google or facebook/instagram, you might care (streaming, photos, static assets all at enormous scales). For me, however, the above results mean far less CPU burden on the client with a twice the burden on the network I/O. If you're concerned about battery, loading screens, or download dialogs, I'd still pick lz4. Just turn the AC up a half notch in my AWS DC. [0] http://gregoryszorc.com/images/compression-bundle-common.png [1] http://gregoryszorc.com/images/decompression-bundle-output.png

You are correct. I just hadn't noticed: - Project Status Update: https://github.com/indygreg/PyOxidizer/discussions/740 - https://gregoryszorc.com/blog/2024/03/17/my-shifting-open-source-prior…

This got me curious and it seems that there is known APFS behaviors slowing things down on parallel I/O on same folders [1] I would suspect that this is related to to the fact that reading/writing dirs have no atomic semantics in POSIX fs, and same directories are easily read/written simultaneously by unrelated processes, but surely is cause to edge cases bugs, especially in a world with backup system based on fs snapshots. This is a sad state of affairs. https://gregoryszorc.com/blog/2018/10/29/global-kernel-locks-in-apfs/

First time I used absorb was in Mercurial back in the day: https://gregoryszorc.com/blog/2018/11/05/absorbing-commit-changes-in-m…

Yes. Uv can install specific Python versions from python build standalone (they have taken over maintenance for it https://astral.sh/blog/python-build-standalone ) instead of from python.org because python doesn't release relocatable binaries (and doesn't release binaries for Linux I think). It works well but does have a few minor https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html

Details here: - https://gregoryszorc.com/blog/2024/12/03/transferring-python-build-sta… - https://astral.sh/blog/python-build-standalone

Not anymore: PEX now supports including an interpreter in the package: https://docs.pex-tool.org/scie.html And unfortunately, PyOxidizer went into maintenance mode: https://gregoryszorc.com/blog/2024/03/17/my-shifting-open-source-prior…

Because Linux is a kernel and doesn't have a standardized userland to link against. Check out IndyGreg's portable Python builds. They're used by Rye and uv. https://gregoryszorc.com/docs/python-build-standalone/main/

Zstd is generally known for markedly faster decompression (as well as compression) than zlib. https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstan…

The reason is due to file system filters, of which Windows Defender is always there. There is a significant delay from Defender when performing CloseFile()[0]. > As I was looking at the raw system calls related to I/O, something immediately popped out: CloseFile() operations were frequently taking 1-5 milliseconds whereas other operations like opening, reading, and writing files only took 1-5 microseconds. That's a 1000x difference! This is why DevDrive was introduced[1]. You can either have Defender operate in async mode (default) or remove it entirely from the volume at your own risk. The performance issue isn't related to sync or async I/O. [0] https://gregoryszorc.com/blog/2015/10/22/append-i/o-performance-on-win… [1] https://devblogs.microsoft.com/visualstudio/devdrive/

Replacing lz4 with zstd will give you half the speed, with just better compression. https://gregoryszorc.com/blog/2017/03/07/better-compression-with-zstan…

> pip does this for you though, I'm not sure why most end users need to know "build" or "install" exist. Because at least at one point in time "build" was marketed as the future, it even appeared in a setuptools warning message. This developer spends several paragraphs on the issue: https://gregoryszorc.com/blog/2023/10/30/my-user-experience-porting-of… If it is no longer the future, so be it. Pip seemed to force isolated installs by default for a while and then backed off again. As for the volunteer question: All OSS package managers are maintained by volunteers and I do not know of any other ecosystem that prompted an xkcd comic about the packaging situation.