Caremad

That blog post provides nothing of value. It complains that there isn´t package signing and that´s it. It doesn´t propose anything. In the comments the author claims removing the signing is "beyond logic" even while linking to a reddit thread in which the logic is laid out clearly. For example: https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ https://mail.python.org/pipermail/distutils-sig/2018-March/032066.html https://old.reddit.com/r/Python/comments/8r9qby/pypi_has_removed_pgp_s… Read those links if you are curious, but I would recommend skipping the post linked by el_programmador. Read

> https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ Thanks, that's actually a great article that explains very well why you can't just throw signatures at the problem and claim that fixes everything. As other commenters have pointed out, the reason Maven central doesn't have this problem has nothing to do with signatures, and everything to do with the fact that all new packages must undergo manual review, which is unfortunately a solution that doesn't scale. (See the "Linux Has Packaging Signing, Let’s Steal Theirs" section from the article you linked.)

Depends on the use. For a library, it doesn't need to be distributed at all, since the application that's using the library will eventually dictate the actual versions of dependencies (so it can play nicely with other libraries). For an application, yes: you commit it along-side the pipfile, and re-generate the lock file when you upgrade things. https://caremad.io/posts/2013/07/setup-vs-requirement/ covers it pretty well. Or https://medium.com/@sdboyer/so-you-want-to-write-a-package-manager-4ae… for a fantastic read and a LOT more context.

A very insightful look at package signing, and why it wouldn't actually improve security for PyPI, by Python packaging guru Donald Stufft: https://caremad.io/2013/07/packaging-signing-not-holy-grail/

Yes and no. There are different types of dependencies, and there are different rules for them, but here's an overview of the best practices: 1. For applications, scripts and services (i.e. "executable code"), during development, pin your direct dependencies; ideally to the current major or minor version, depending how much you trust their their authors to follow SemVer. Also make sure you regularly update and retest the versions to make sure you don't miss any critical updates. You should not explicitly pin your transitive dependencies, i.e. the dependencies of your direct dependencies -- at least unless you know specifically that certain versions will break your app (and even then it is better to provide a range than a single version). 2. For production builds of the above, lock each of your dependencies (including the transitive ones) to specific version and source. It is not really viable to do it by hand, but most packaging tools -- pip, Poetry, PDM, uv... -- will happily do that automatically for you. Unfortunately, Python still doesn't have a standard lock format, so most tools provide their own lock file; the closest thing to a standard we have at the moment is pip's requirements file [0]. Besides pinned versions, a lock file will also include the source where the packages are to be retrieved from (pip's requirements file may omit it, but it's then implicitly assumed to be PyPI); it can (and should, really) also provide hashes for the given packages,…

Which you do not describe, but set that aside: A post that honestly said "we do not like PGP, but here is our alternate plan" would be great. On an actual better solution, I don't think anyone has proposed a good one. Here is the closest I've seen from PyPi (or at least linked from this post as describing their thinking), from 10 years ago: "Everything is Terrible So What Do We Do? Bluntly put, I don’t know for sure. This isn’t an already solved problem nor is it an easy to solve one." https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ What I'll say on PGP is the perfect is the enemy of the good. It's not a tech anyone has much fun using, but in a group setting, used regularly, I have found it can fade into the background at least. I don't want to go any further down the "is PGP good or bad" rabbit hole than that. But if you have a better solution for package security, please do describe it here.

The vast majority of python codebases are still in python 2, even though many of the popular framework have been ported to 3, and there's no reason for most of them to bother with the amount of work it can be to port to py3 when they're happy with py2. Whether it be devs like Dropbox or open source software like NVDA, they will stay in the world of python 2 for the years to come. The amount of customers they could get by supporting python 3 is definitely not worth the effort unless they get paid explicitly for it. In terms of libraries, python 3 is "mostly" ready (though there are still issues with highly popular ones like wxwidgets. Or amazing tools like Pyjamas, which replicated GWT with Python instead of Java. Or Google AppEngine, which still shows no sign of moving to 3.) but large codebases outside of highly worked on web frameworks haven't moved on and there's not enough gain to justify the effort it takes to port them. The Python 3 debacle caused a lot of devs to also look at other languages for new projects. Old python 2 codebases will stay python 2, but new projects might not necessarily be python at all. The anger with Python 3 is real. See also: https://caremad.io/2015/04/a-year-of-pypi-downloads/ https://www.reddit.com/r/Python/comments/45sm94/what_are_the_most_rece… "Looking at these graphs, we can see that in the past year Python 3.x has grown from roughly 2% of the total downloads from PyPI to roughly 5-6%" I expect things…

From here: https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ which is linked to the article since PyPI has so many packages and that everyone can sign up to add a package it would be extremely unmanageable.

There are 2 ways to specify requirements in a python project as well. Manually constructed list of dependencies in setup.py, and requirements.txt generated via `pip freeze` (with some possible tweaks on top of that if necessary). See also https://caremad.io/posts/2013/07/setup-vs-requirement/

Signing packages works well for a linux distro. It's much less clear that it would help for a language package manager: https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/.

FWIW, here is a blog post by dstufft which might help contextualize this behaviour: https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/

You should read this: https://caremad.io/posts/2013/07/setup-vs-requirement/, it specifically references this "feature" of pbr.

David Beazley : http://www.dabeaz.com/ Kenneth Reitz : https://www.kennethreitz.org/ Armin Ronacher : http://lucumr.pocoo.org/ Julien Danjou : https://julien.danjou.info/ Hynek Schlawack : https://hynek.me/ Donald Stufft : https://caremad.io/

For people who want to do it right without using an additional tool read this: setup.py vs. requirements.txt by Donald Stufft https://caremad.io/posts/2013/07/setup-vs-requirement/

I'm afraid I never saw what was wrong with specifying dependencies in setup.py. For me, having requirements.txt and setup.py is confusing. Can't we just stick with setup.py? (and yes, I've read Donald Stufft's post https://caremad.io/posts/2013/07/setup-vs-requirement/ and remain unconvinced).

Maybe I am missing the point, but "loose" dependencies go in setup.py, and "exact versions" go in requirements.txt. https://caremad.io/posts/2013/07/setup-vs-requirement/

There is https://caremad.io/2015/04/a-year-of-pypi-downloads/ which is also based on PyPI downloads and is more recent.

From this (last year), Python 2.6 is bigger than 3: https://caremad.io/2015/04/a-year-of-pypi-downloads/ >I'm guessing that most people who use Python 3 are using recent versions of pip that have caching built in, so that may skew the results a bit. Since pip updates itself too, why wouldn't people using Python 2 also have newer versions?

GPG isn't the holy grail. See https://caremad.io/2013/07/packaging-signing-not-holy-grail/ Proxying via a local dependency is a good practice, but that can be set up independently of npm.

This is a nice idea in theory, but setup.py and requirements.txt are not the same, and you cannot particularly generate one from the other: https://caremad.io/2013/07/setup-vs-requirement/ Also, unfortunately pip's parser for requirements.txt isn't public, like much of its internals at the moment, which means AIUI it's likely that the parsing code here is going to be brittle if it uses it, or miss edge cases if it doesn't. (I haven't read this carefully yet though).

I've bookmarked Donald Stufft's opinion on this for a couple years, which contrasts interestingly with Kenneth's and your own (and which makes a lot of sense): https://caremad.io/2013/07/setup-vs-requirement/ I like the concrete-vs-abstract, library-vs-application ideas there. Most opinions on this overlook the distinction.

6% were 3.x in April 2014 https://caremad.io/2015/04/a-year-of-pypi-downloads/

Some packages are ported but that does not mean people use Python 3. If you look at the download numbers it's pretty clear that they are not indicate actual use: https://caremad.io/2015/04/a-year-of-pypi-downloads/

we're good friends with the original owner "dstufft" and took over the domain after he took over an official pypi position (more than a year ago). read his own blog post: https://caremad.io/2014/03/crate-io-new-ownership/

Ideally, setup.py is for generic minimum-version-known-to-work dependencies of a package. Pip requirements files are for very specific known-good combinations of dependencies of an application. https://caremad.io/blog/setup-vs-requirement/

I was only asking why you say it comes crashing down, when the link cites multiple cases of successful parallel version maintenance with a version that is at least partially calendar-based. As far as usability, time is a dimension no developer can escape. We develop software on a schedule and we deprecate it on a schedule. CalVer is tremendously semantic at that higher level. The semantics SemVer clings to suffer even under ecosystems which have mandated their use (go, node). The effect has been an over-granularization of APIs themselves. I'm fine with agreeing to disagree for now. I'm also fine with the case that CalVer is not for everyone and everything. I'll leave it with this, a post seems to articulate a lot of your concerns: https://caremad.io/posts/2016/02/versioning-software/ It was written in 2016 by Donald Stufft, a maintainer of pip. Today, pip has been CalVer for a while and I'm not sure anyone has looked back. Try it sometime, and you might find the same. :)

When I read things like, https://caremad.io/posts/2013/07/packaging-signing-not-holy-grail/ I get the impression they don't care about security at all. They seem like children plugging their ears and shouting "nah nah nah" while putting repo users at risk. They've obviously done nothing since that post was made four years ago. In contrast, Maven central requires signing. Unsurprisingly, Maven central doesn't have typosquatting problems. That's not a coincidence. That's also a strong reason why Java still dominates the enterprise. If you (the reader) are a PyPi/NPM user, I challenge you to watch this, https://www.youtube.com/watch?v=pBJafU0p_Nk and tell me why you shouldn't use a repository manager like Sonatype Nexus which validates package signatures, checks licenses, and does vulnerability scanning. By itself, package signatures aren't the holy grail? That's not the point. Security is achieved in layers. Too bad they aren't mature enough to understand that.

As I mentioned the last time this came up, I recently converted a medium-sized (about 30,000 lines) system from Python 2 to Python 3. The problem is not the language changes. Those are minor, and most of the new forms worked in Python 2.7.9. It's the state of the external modules. Important old Python 2 modules have been discontinued, and their replacements are not used enough or tested well enough to use reliably. I kept finding bugs that should have been found in the first months of heavy use of a module.[1] It took an extra month of work to find and work around all the external module bugs. These aren't obscure modules; they're ones most large server-side applications would need. Porting was at least possible. Two years ago, I took a look at doing this, and there weren't enough modules ready to even attempt it. Now there are. There is progress. Slow progress. The download statistics tell the story: "We can see that in the past year Python 3.x has grown from roughly 2% of the total downloads from PyPI to roughly 5-6%."[2] That's why the external modules don't work reliably on Python 3. 95% of the package use is on Python 2. Python 2 use is probably higher than that, since those are download statistics; PyPI doesn't see packages downloaded long ago and still running. There's extensive denial about this among Python fans. That's a big part of the problem. Another problem is that PyPI is just a link farm - unlike CPAN, it doesn't centralize code hosting, bug…