Hynek Schlawack

also a great guide for building production-ready containers using uv: https://hynek.me/articles/docker-uv/

This has been useful advice for many years, although restricting to AEAD is best when possible. https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/

Those pages are managed by the "PyPA" group, not technically python.org itself, although they were given a subdomain there. Start here and follow the links for more info: https://hynek.me/articles/python-app-deps-2018/

Solid Snakes has a excellent page with the referenced links: https://hynek.me/talks/reliability/ I liked the presentation, but, probably due to already knowing almost all, didn't find it great. Better to read the link above first.

I actually just built my first python package two weeks ago. It was pure-python and I found it fairly straightforward, starting with the bare-bones LPTHW [1] template, and adding things from the Sharing Your Labor of Love [2] guide. It basically 'just worked' across Python 2, Python 3, Linux, OSX and Windows. There's a lot of outdated examples out there, and it might be more troublesome if I had a more complicated project, but... well, it was ok for me. [1] http://learnpythonthehardway.org/book/ex46.html [2] https://hynek.me/articles/sharing-your-labor-of-love-pypi-quick-and-di…

Shipping python projects isn't easy. Even if you have a prefect package on pypi, that specifies its dependencies, it may conflict with other tools using other versions of those libraries. Compiling python does not feel right. Using native packages is better[1], but you have to plan it out. You can package virtualenvs into native packages[2] - and this is probably the most solid way, but you'll need packaging setups for various package managers. In short, "written in Go" for me means with a high probability, just run "go get x" - or download a binary. Since we are at it, I'd love to hear about better shipping strategies for Python projects. [1] https://hynek.me/talks/python-deployments/ [2] https://labs.spotify.com/2013/10/10/packaging-in-your-packaging-dh-vir…

Breaking changes don't break everyone. For many projects, only a small fraction of users are broken any given time. Firefox is on version 139 (similarly Chrome and other web browsers); how many times have you had to reinstall your plugins and extensions? For that matter, have you seen any Python unit tests written before the Pytest 8 release that were broken by it? I think even ones that I wrote in the 6.x era would still run. For that matter, the Python 3.x bytecode changes with every minor revision and things get removed from the standard library following a deprecation schedule, etc., and there's a tendency in the ecosystem to drop support for EOL Python versions, just to not have to think about it - but tons of (non-async) new code would likely work as far back as 3.6. It's not hard to avoid the := operator or the match statement (f-strings are definitely more endemic than that). On the flip side, you can never really be sure what will break someone. Semver is an ideal, not reality (https://hynek.me/articles/semver-will-not-save-you). And lots of projects are on calver anyway.

The form you propose runs `command`, which may have undesired side-effects. I always thought `which` to be standard, but TIL `command` (sh builtin) is[0]. [0]: https://hynek.me/til/which-not-posix/

Saw this tweet yesterday: https://twitter.com/hynek/status/1609160276888141824 From https://hynek.me/til/copyright-years/ >Every January 1st, an army of open source developers rushes out to update their copyright attributions in licenses and documentation. Why? Because we’ve always done it that way. >I’ve stopped participating after I learned that copyright statements need only the year of the first publication and no lawyer that I asked contradicted >Now, I’m not a lawyer, so don’t take this as legal advice from me. All I’m saying is that if it’s good enough for Google’s, Microsoft’s, and Netflix’s lawyers, it’s good enough for me

Sure but that has nothing to do with monorepo-vs-manyrepo. Those are good things to do in any case. And I suspect you're going to find that inside of Google these are 10+ year long projects which were not designed well enough for that (because honestly nobody does that). And the breaking changes are most likely largely all well intentioned. Without at time machine its just work that needs to get done in one way or another. SemVer also has its problems, and is not a panacea, for example: https://hynek.me/articles/semver-will-not-save-you/ And I'm positive Google is aware of SemVer.

Hynek, a CPython committer, had written a blog post[1] the state of Python application dependencies in 2018, updated in 2019 (no change in 2020, I asked). It was also surfaced on HN 3 times but did not get much attention[2] [1]: https://hynek.me/articles/python-app-deps-2018/#petry [2]: https://hn.algolia.com/?q=python-app-deps-2018

I came to the comments to say exactly this. There's a decent summary of why someone might still prefer pip-tools even in a world where pipenv and poetry exist here: https://hynek.me/articles/python-app-deps-2018/ For my purposes, the primary downside of this approach is that adding dependencies takes slightly more effort, because you have to edit a file and then execute a shell command, rather than just executing a shell command. But managing dependencies takes up about 0.001% of my time, so this is not an area where I have much to gain by micro-optimizing my workflow. I do, on the other hand, have a lot to lose by switching to something that's newer and shinier and less stable.

The src directory is becoming a popular option. See https://blog.ionelmc.ro/2014/05/25/python-packaging/#the-structure for an enumeration of the reasons. Another resource might be https://hynek.me/articles/testing-packaging/

There was an interesting article 2 months ago that was trending on HN [0] with a good discussion [1]. - [0] https://hynek.me/articles/speaking/ - [1] https://news.ycombinator.com/item?id=14450905

David Beazley : http://www.dabeaz.com/ Kenneth Reitz : https://www.kennethreitz.org/ Armin Ronacher : http://lucumr.pocoo.org/ Julien Danjou : https://julien.danjou.info/ Hynek Schlawack : https://hynek.me/ Donald Stufft : https://caremad.io/

If you are making SSL requests with Python you will want to use service_identity[0] to perform certificate validation. Hynek has some good information on his blog[1]. [0]: https://github.com/pyca/service_identity [1]: https://hynek.me/talks/tls/

Using a binary package with pip requires "a compiler and all of the development libraries", which GP said he doesn't want to do. Personally I gave up a while ago and now I do "apt-get build-essential python-dev" on every new server, but it's not ideal. I really like the approach outlined in https://hynek.me/articles/python-app-deployment-with-native-packages/ , but haven't gotten around to implementing it.

Take a look at http://hynek.me/articles/python-app-deployment-with-native-packages/

> If you are mocking anything in the standard library your code is probably structured poorly. I like Hynek Schlawak's 'Don’t Mock What You Don’t Own' [1] phrasing, and while I'm not a fan of adding too many layers of abstraction to an application that hasn't proved that it needs them, the one structure I find consistently useful is to add a very thin layer over parts that do I/O, converting to/from types that you own to whatever is needed for the actual thing. These layers should be boring and narrow (for example, never mock past validation you depend upon), doing as little conversion as possible. You can also rephrase the general purpose open()-type usage into application/purpose-specific usages of that. Then you can either unittest.mock.patch these or provide alternate stub implementations for tests in a different way, with this this approach also translating easily to other languages that don't have the (double-edged sword) flexibility of Python's own unittest.mock. [1] https://hynek.me/articles/what-to-mock-in-5-mins/

Shouldn't some AI be able to clean that up for you? This seems something LLMs should be well-suited for. --- FWIW, I'm the speaker and let me be honest with you: I'm super unmotivated to write nowadays. In the past, my usual MO was writing a bunch of blog posts and submit the ones that resonated to CfPs (e.g. <https://hynek.me/articles/python-subclassing-redux/> → <https://hynek.me/talks/subclassing/>). However, nowadays thanks to the recent-ish changes in Twitter and Google, my only chance to have my stuff read by a nontrivial amount of people is hitting HN frontage which is a lottery. It's so bad I even got into YouTubing to get a roll at the algorithm wheel. It takes (me) a lot of work to crystallize and compress my thoughts like this. Giving it as a talk at a big conference, at least opens the door to interesting IRL interactions which are important (to me), because I'm an introvert. I can't stress enough how we're currently eating the seed corn by killing the public web.

Building Docker images is a tricky design problem, as with many ops tasks. The problem is due to 3 issues: 1. A Docker image is the intersection of literally 50 years of technology, starting from Unix in the 1970s, none of which are perfect. Decisions from back then still have an impact, e.g. you need to get signal handling right if you want shutdown to work (https://hynek.me/articles/docker-signals/). 2. Getting it right is all about _details_. Lots and lots of details, all of which need to be correct. Signal handling, where logs go, security updates, on and on. 3. Every organization does things slightly differently. So there are multiple approaches: == Build an abstracted tool == A tool will support certain conventions and ways of doing things... but as soon as you want to diverge too much you'll start having issues, because it only supports its particular way of doing things. Pros: Easy for users. Cons: Only so long as they don't want something the tool can't handle. The problem is that it's quite difficult to build a tool that works for _all_ organizations. If you've built a custom buildpack for your company, you've built a custom tool, which a pretty good solution, but still every organization has to build their own tool. == Build a configuration language == In order to support all the edge cases, you build a configuration language... and pretty soon it's super complicated because there are so many different things people want to do. In fact, you end up with…

Well my team does that for one. We use Python packaging ecosystem, specify Python dependencies using standard tools like setup.py, requirements.txt and pip. All Python dependencies are baked into a fat Python package using PEX format[1]. Also tried Facebook's xar format[2], without success yet. What matters is to have a statically linked dependencies packaged in one executable file. Like a Windows exe file. Then you proceed with bundling in higher level OS dependencies, because each app is not just a Python package but also a collection of shell scripts, configs, configuration of paths for caches, output directories, system variables, etc. For this we throw everything into one directory tree and run FPM [3] command on it which turns the directory tree into a RPM package. We use a fpm parameter to specify installation location of that tree to /opt or /usr elsewhere. The way to bundle it properly is to actually use two rpm packages linked together by rpm dependency. One for the app and the other for the deployment configuration. The reason is you only have one executable, but many possible versions of deploying it. Ie. depending on environment (dev, staging, prod) or you just simply want to run the same app in parallel with different configuration. eg. one rpm package for the app executable and statix files my_app_exe-2.0-1.noarch.rpm and many othe related config rpms my_app_dev-1.2-1.noarch.rpm (depends on my_app_exe > 2.0) my_app_prod-3.0-1.noarch.rpm (depends on…

I think that for the majority of use cases when it comes to packaging and shipping Python programs, we really want to ship OS packages. We don't want users (that includes sysadmins/operations teams in this case) to deal with having pip available, understanding what virtualenv is, and so on. We just want them to be able to use apt, yum, or whatever they experience as most common on their system. Yet at the same time, we don't want our application to take in regard ancient versions of OS packages to depend on. I think in the end we want a self contained OS package, to offer a sane install method for any target OS. Why would users have to make a difference between e.g. a C program that they install through apt, or a Python program that all of a sudden requires a completely different way of installing? Here are some interesting thoughts on it: https://hynek.me/articles/python-app-deployment-with-native-packages/ And here's a tool that I wrote to be able to wrap up a virtualenv into an OS package, while specifying a precise list of OS dependencies. It's not perfect by far, and admittedly a bit crude, but it already does the job well for some real life "production" stuff. It's called vdist (virtualenv distribution). https://github.com/objectified/vdist Documentation is here: https://vdist.readthedocs.org/en/latest/

Ciphers to avoid DSS, MD5/RC4, SHA-1. Ciphers to prefer ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256 A pretty good source/guide: https://hynek.me/articles/hardening-your-web-servers-ssl-ciphers/ You'll need apache 2.4+[I think], or nginx. And possibly fresh certs to use DHE/EC. A quick rundown of a fairly secure setup: Cipher Priority list: ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:!ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:!RSA+3DES:!aNULL:!MD5:!DSS:!SHA:AEAD ========================================================== Generate the cert and private key: openssl req -x509 -sha256 -nodes -days 3650 -newkey rsa:4096 -keyout serverkey.pem -out servercert.pem ========================================================== Generate the DH parameters: openssl dhparam -out dh2048.pem -outform PEM -2 2048 ========================================================== How to List Elliptic Curves: openssl ecparam -list_curves =========================================================== Note: Generating DH parameters is gonna take a while. If you are implementing this on a slowish machine like a Raspberry Pi, you might want to use a faster machine to do the DH step, then copy file the key over.

I think we should look back to the rant of Python Core Committer Hynek Schlawack. (https://hynek.me/articles/python-app-deployment-with-native-packages/) In short build your system and it's dependencies once and once only then pass them around through test into live. We have three competing needs: a reliable deployment process that can move a binary-like blob across multiple test environments A need for exactly reproducible environments but without dd/ghosting A desire to keep things simple Isolation is good - whether through the mostly isolated approach of venv, the almost total isolation of jails/LXC or the vagrant approach. But they focus almost entirely on binary builds - how does one pass around a python environment without rebuilding it and it's dependencies each time ala pip? Well by taking the running built python environments and passing them into a package manager like apt and calling that a binary. That might mean tar balling a venv or tar balling /use/local/python but in the end it matters that we pass around the same basic bits. I am working this out in pyholodeck.mikadosoftware.com and in my head - when I have a good answer I will shout

https://github.com/mschultheiss/Charme/blob/7634cd89febbe28825d6d33c01… Please see: https://paragonie.com/blog/2015/07/how-safely-generate-random-strings-… -------------- https://github.com/mschultheiss/Charme/blob/c8eb501f1ac0c607d953aa60bc… https://github.com/mschultheiss/Charme/blob/7634cd89febbe28825d6d33c01… please see: http://codahale.com/how-to-safely-store-a-password/ https://hynek.me/articles/storing-passwords/