Alexander Rushter

https://rushter.com Most of my posts are about Python's internals and some security stuff

If you want to go really deep, I would recommend reading some blogs about Python internals: https://tenthousandmeters.com/tag/cpython/ https://rushter.com/blog/tags/cpython/ https://eli.thegreenplace.net/tag/python-internals https://realpython.com/cpython-source-code-guide/ I'm not aware of any good books about Python's internals.

For starters, just adding Cython and/or Numba to the code should introduce great improvements. https://rushter.com/blog/numba-cython-python-optimization/

Apparently, Python split the difference and uses reference counting alongside garbage collection: https://rushter.com/blog/python-garbage-collector/ However: > I'd say that Borrow Checker is still manual memory management, except the compiler writes the free() for you. And: > #4: purely declarative languages and such is going to be garbage collected in practice. My point is to catalog the different ways programmers have to think about resource usage, not implementation details. Python is a GC'd language, and unlike C++ with smart pointers, because it doesn't make the programmer manually manage memory in order to avoid a leak, and regular expressions and SQL are high-level in that the languages don't even offer a way to express the concept of resource management.

I'm not sure about the article, but the blazingly-fast IP scanner sounds a lot like Masscan: https://github.com/robertdavidgraham/masscan It can scan the entire Internet in 5 minutes and has received a lot of press. https://rushter.com/blog/how-masscan-works/ is one of many articles about how it works.

Just as a note, the way that Python stores the items isn't always 4 bytes per char, it depends on the actual string contents. I think that https://rushter.com/blog/python-strings-and-memory/ is a nice reference on that.

The design decisions of the Python language (arbitrary length integers for example) lead Python to just be SLOW. It's much slower than Perl. Don't believe me? Look at how numbers are stored: https://rushter.com/blog/python-integer-implementation/ But it doesn't really matter. Computers increase in speed over time. Libraries allow speed sensitive sections to use more sane data types. As you pointed out, the Perl language's design decisions cause much worse problems -- hard to identify bugs.

In my experience selectolax is about 10x faster than lxml, and keeps the familiar CSS selector API: https://rushter.com/blog/python-fast-html-parser/

> what sort of "everything" do you have in mind? I’ll answer in two parts. First, in my original claim: > not everything is necessarily backed by the same kind of heap-allocated memory object. the emphasized part is important. Not all PyObjects are equivalent. Most things in Python are indeed PyObjects of some stripe, but they act very, very differently in some common and performance-sensitive situations. The optimizations discussed in the article are one example: allocations can be re-used, and some more complex data structures are interned/cached (like the short-tuple/short-list optimization, which is similar but not the same to how floats/numbers are reused via allocator pools: https://rushter.com/blog/python-lists-and-tuples/). As you point out, immortal objects and fully preallocated objects are also often special cases vis-a-vis the performance of creating/discarding objects. Second, you’re wrong. PyObjects do not come into play in a wide variety of cases: - In some (non-default even today, I believe) interpreter configurations, tagged pointers are used for some numbers, removing pointer chasing entirely: https://rushter.com/blog/python-lists-and-tuples/ - Bound method objects can be cached or bypassed, meaning that, while there’s a PyObject for the object being method-called on and the method, there’s sometimes one less PyObject for the binding: https://github.com/python/cpython/issues/70298, with additional/somewhat related info in PEP-580 and PEP-590. PyPy…

> Then I guess using the id of the tuples to show that they are using the same object doesn't exactly prove the point Without the `del a`, it does, because they both have active references. If they were unique objects, we'd see a unique ID for `b` as long as a reference to `a` is active. > Strings seem to behave like tuples. When I delete a string and create a new one it creates a new object with a new address.... unless the strings are of the same length. String _objects_ (as opposed to variables referring to them) are immutable in Python. They tend to be allocated anew, but for optimization reasons you can end up with cases where the string objects have the same ID. Like here: >>> a = 'asdf' >>> id(a) 4389881424 >>> a = 'qwerty' >>> id(a) 4395015672 >>> a = 'asdf' >>> id(a) 4389881424 'asdf' has the same ID with no deling involved because its object wasn't GC'd yet. Below are some relevant links if you want to knock yourself out (some do) but I write an awful lot of Python and even for me this is well into the realm of "what happens when..." after a few beers or job interview trivia. https://rushter.com/blog/python-lists-and-tuples/ http://guilload.com/python-string-interning/