Victor Stinner

For anyone interested in learning why Python 3 works this way I highly recommend the blog of Victor Stinner[0]. As for the article, this is nothing new. The problem is similar to the issues raised by Armin Ronacher[1]. These problems are well known and Python developers address them one at a time. Issues around these egde cases have improved since the initial release of Python 3.0. [0] http://vstinner.github.io [1] http://lucumr.pocoo.org/2014/5/12/everything-about-unicode/

Funnily, because of the 2/3 transition, Python became a language obsessed with encoding correctness. Hence the team spent a great deal improving the situation, from version to version during the last 10 years. For a fantastic read (from Victor Stinner himself) on all the work done and get how twisted this get when you want to be cross plateform and abstract it: https://vstinner.github.io/python37-new-utf8-mode.html Windows and Linux FS encoding, of course, are at the center of the challenge. It's also the reason why we now have a __fspath__ protocol that allows any object to be converted to a file system path instead of having pathlib.Path inheriting from string.

So, my favorite thought process from the Python 2 to 3 transition comes from staring deeply into the hubris of the Python 3000 effort itself as wanting to insist "everything is Unicode"; on the one hand they were trying to shame developers using Python 2 for using it incorrectly (even though it was totally possible to handle Unicode correctly with that stack), yet on the other they were pumping out ridiculous unavoidable security SNAFUs in their API surface by assuming that filenames on disk are UTF-8 :/. To tell another story of the importance of reading the--in this case, not even long!--specification: on Windows, filenames are stored as UCS-2 (as it all predates UTF-16), and is most likely most correctly modeled as an arbitrary sequence of non-0 16-bit values; meanwhile, on Unix systems, a filename is pretty much just an arbitrary sequence of non-0 8-bit values (with some weirdness around the slash character... the API surface doesn't support it really but sometimes the filesystem does). The premise that filenames have a display encoding mostly comes from the software rendering the strings and is then defined in terms of a bunch of hacks involving "locales", and isn't a property of the underlying API or storage semantics. You can't assume in either case that they are canonical Unicode representations, and so storing them as "strings" in your program is super wrong and can lead to anything from data storage mistakes to security vulnerabilities. The Python 3 API…

But isn't 'surrogateescape' supposed to address this? (no expert) https://vstinner.github.io/pep-383.html

Thanks, yeah that's basically what I thought, but it's nice to know it was the same year! If only UTF-8 had been invented a little earlier, we could have avoided so much pain :-( The idea of global varables like LANG= and LC_TYPE= in C is utterly incoherent. Python's notion of "default file system encoding" is likewise incoherent. You can obviously process strings with two different encodings in the same program !!! Encodings are metadata, and metadata should be attached to data. Encodings shouldn't be global variables! Python 3 made things worse in many ways, largely due to adherence to Windows legacy, and then finally introduced UTF-8 mode: https://vstinner.github.io/painful-history-python-filesystem-encoding.…

Found some history: https://vstinner.github.io/python30-listdir-undecodable-filenames.html Includes "Modify os.listdir(str) to ignore silently undecodable filenames, instead of returning them as bytes", but not later work where apparently this was changed to use surrogates.

Victor Stinner (python dev) wrote a good series on trying to get stable benchmarks in python: https://vstinner.github.io/journey-to-stable-benchmark-system.html https://vstinner.github.io/journey-to-stable-benchmark-deadcode.html https://vstinner.github.io/journey-to-stable-benchmark-average.html