Glyph Lefkowitz

Anytime this comes up I plug the excellent "Unyielding" (https://glyph.twistedmatrix.com/2014/02/unyielding.html) and "Notes on structured concurrency" (https://vorpus.org/blog/notes-on-structured-concurrency-or-go-statemen… as the counterpoint to "What color is your function". Being able to see structurally what effects your invocation of a function will result in is very helpful in reasoning about the effects of concurrency in a complicated domain.

It's worth reading that excellent article alongside Glyph's excellent article from the opposite perspective: https://glyph.twistedmatrix.com/2014/02/unyielding.html A few languages avoid the pitfalls of threads, like Rust where its ownership system protects you.

A tangent... Glyph† pointed out that, in Python, as an OOP language, we should objectify all the things, e.g. like what Pathlib does for file/path manipulation. (Pathlib was inspired or extracted from Twisted, IIRC.) A Pythonic dependency/package manager library would be soooo cooool... † https://en.wikipedia.org/wiki/Glyph_Lefkowitz https://glyph.twistedmatrix.com/

I still use and greatly prefer gevent to this day. They are indeed quite similar to goroutines. Asyncio is a different model, and more irritating for me to work with. (I'm sure this isn't the case for everyone. I'm just more productive with gevent, personally.) Performance is pretty close for both. I was disappointed to not see Python get an official green thread implementation. The counter-argument I commonly see cited is https://glyph.twistedmatrix.com/2014/02/unyielding.html. I personally don't find it to be a very convincing argument.

Here is a great article that argues pretty well for why e-mails aren't tasks: https://glyph.twistedmatrix.com/2016/04/email-isnt-the-problem.html One of the arguments that directly comes to mind when I look at dragapp's screenshots is that parsing messages to find out what the underlying task is just takes too much time. Look at the image and you'll see that there is no way to see what part of "Chrome extension for Gmail" is in to-do, progress or complete. You have to read each e-mail to find out. Backup screenshot: http://imgur.com/a/VtyyO

The python distribution story is far behind Go. This article goes into more details: https://glyph.twistedmatrix.com/2015/09/software-you-can-use.html PS: I'm the original author of PyInstaller. I obviously love it but it's a gross hack around the fact that Python doesn't have a serious distribution story.

Also on the subject of being reckless with strings, I recommend this blog post by Glyph Lefkowitz: https://glyph.twistedmatrix.com/2008/06/data-in-garbage-out.html

That they are not portable to the web and vice-versa in Python, first. Second, I can not do justice... so I will give you a great blog post that sums my thoughts nicely: https://glyph.twistedmatrix.com/2015/09/software-you-can-use.html

https://glyph.twistedmatrix.com/2014/02/unyielding.html is a good read - there are reasons why explicit sync-async "coloring" (i.e. await/yield) is better than green threads/coroutines which author admires.

> Thank you, I've read the links, but without an example of Haskell/Scala code the benefits for any particular real problem are not clear to me. Do you have cases where you've used AOP/reflection/metaclasses/decorators/macros or similar "magic"? (Exceptions or mutable variables sort of work as examples, but most people don't have experience of working on codebases that don't have them). It's hard to give simple examples because these things exist to solve complex problems - one should never reach for a monad where a plain function will do. > Specifically, speaking of the second link, what happens if you need to get rid of two types of 'hell' at the same time? There are several possibilities. First, it's worth noticing that most forms of "hell" are fine on their own, it's the interaction between two that tends to be surprising - e.g. https://glyph.twistedmatrix.com/2014/02/unyielding.html talks about how important it is to explicitly manage async, but it's making an underlying assumption that your language will have mutable variables. Unmanaged, pervasive mutation in a language without async is ok; equally, unmanaged, pervasive async is ok in a language without mutation. So often you can get away with managing one effect and letting the other happen naturally according to the program control flow. (This is less true in Haskell because laziness is built into the language, so Haskell has kind of "spent its freebie" there - but it can be a very effective style in…

Goroutines are, for all practical purposes, threads. They're cheaper than real threads but they still have all the subtle complexity that comes with shared-memory preemptive threading (everything shared needs to be appropriately locked, scheduling is non-deterministic and can happen any time, etc). Async/await is different; context switches can only happen at an `await` expression. This greatly reduces the number of possibilities that must be considered and simplifies the task of concurrent programming (for more, see https://glyph.twistedmatrix.com/2014/02/unyielding.html). In Python, the closest equivalent to `go foo()` is `threading.Thread(target=foo).start()`. Or you can use gevent to get goroutine-like lightweight threads, but I find this is works better in go than in python because the entire language is designed around it. In python it's common to find libraries that are not compatible with gevent's monkey-patching and so you lose concurrency in a way that is difficult to detect or guard against.

One of the comments on the original post claims open source apps don't count because they "are not subject to many of the challenges the average developer faces in a corporate environment" so it appears TDD fans can't win either way. But since you're looking for open source success stories there was some very positive noises about TDD from a Twisted dev recently: "I would strongly recommend at least practicing some TDD, too. Learning how to write tests was useful for me, but practicing TDD really expanded my design skills in a bunch of surprising directions that were almost unrelated to testing. I originally felt as you do, that I wasn't disciplined enough to do it; now I feel like I'm too undisciplined not to do it ;-)." From: http://glyph.twistedmatrix.com/2009/09/diesel-case-study-in-that-thing… Though he specifically states that Twisted didn't start out TDD, which he regrets, but maybe that disqualifies it. Regarding Rubinius (and JRuby that was listed in the original post) I'm not sure trying to create a spec for an existing language implementation and then try to meet it would count as "true" TDD if people care about such differences. But maybe they do TDD for other bits lower in the stack, I'm not sure.

I think the promise-based model is the ideal model. After reading [1], I'm convinced that it's very close to realizing the underlying state machine ([2]) of the program, and I find that valuable. I agree that fibers are easier to use, but not simpler. I understand that my hard-line approach is not practical everywhere, but at least the async model used by Rust is not just a concession to the C ABI and whatnot, but an important model in its own right. So I think that it is quite sensible, if not practical for most programmers to make sense of :). Add fiber abstractions on top if appropriate, of course. [1] https://glyph.twistedmatrix.com/2014/02/unyielding.html [2] https://lamport.azurewebsites.net/pubs/state-machine.pdf (read the introduction)

One thing I don't fully understand is how async-without-yield is different from the "bad" green threads as described in e.g. https://glyph.twistedmatrix.com/2014/02/unyielding.html Are these two fundamentally opposing viewpoints? Or can these two sets of ideas be reconciled somehow? The article that I posted specifically calls out Java as an example of why you need explicit await, yet the article in the OP praises Java for not having it!

Actually, this precursor blog post focusing on typing.Protocols is even clearer: https://glyph.twistedmatrix.com/2020/07/new-duck.html

I was just reading this last week talking about zope.interface https://glyph.twistedmatrix.com/2021/03/interfaces-and-protocols.html

Are you aware of this security issue? There are also similar issues with DLLs on Windows. https://glyph.twistedmatrix.com/2020/08/never-run-python-in-your-downl…

In Python I use async instead of threads for reasons unrelated to performance. https://glyph.twistedmatrix.com/2014/02/unyielding.html

Do you ever install packages from Emacs? If so, you definitely want that to be at least TLS protected. https://glyph.twistedmatrix.com/2015/11/editor-malware.html is a post describing the problem and some solutions. I haven't checked to see how many of the steps are still necessary on Emacs 26, but I bet it's still non-zero.

That was true years ago. It's really not the case anymore. Related reading: https://glyph.twistedmatrix.com/2016/08/python-packaging.html More to the point, even if what you said were true, the tooling is still absurdly complicated in the JS world. Your comment is a mix of of a red herring and whataboutism.

there's a very popular argument often made that if you dont use an explicit async-on-evented-IO approach for concurrency in general, and instead use threads or even an event-IO approach that conceals the IO wait just like a thread scheduler, then your program is impossible to reason about and will have bugs forever. Glyphs "Unyielding" at https://glyph.twistedmatrix.com/2014/02/unyielding.html is one well known blog post that describes this concept. For a lot of developers, the original reasons for event-based IO (e.g. many stateful network connections) is lost. The popularity of Javascript, which in recent years has exploded as a server side platform as well, is a key driver in this trend.

An article about the "attrs" library was posted here a couple weeks ago. Really highlighted the tedium of Python objects while offering a neat solution. https://glyph.twistedmatrix.com/2016/08/attrs.html Regarding protobuf, I'm a bit disappointed with the direction of version 3. Fields can no longer be marked as required - everything is optional; i.e. almost every protobuf needs to be wrapped with some sort of validator to ensure that necessary fields are present. I understand the arguments, but I did enjoy letting protobuf do the bulk of the work making sure fields were present.

Thanks for your considered reply. HN decided my reply was too long to post here, so instead it's on my blog: https://glyph.twistedmatrix.com/2016/01/stop-working-so-hard.html

Here’s a more carefully elaborated version of the criticism of Graham’s sloppy thinking and lazy writing from his inequality essay: https://glyph.twistedmatrix.com/2016/01/premises.html Since life is short, that’s it for me for this thread.

I'm not sure from what perspective you were looking to compare asyncio to green threads (performance, readability, etc.) but there is an interesting blog post [0] by the lead architect of Twisted about the fundamental difference in programming model between explicit yielding (e.g. asyncio) and implicit yielding (e.g. green threads, regular threads). It may give you some answers. PEP 492, from what I understood, is more about streamlining the syntax, and the major differences between asyncio (pre this proposed syntax) and green threads in terms of programming model still apply. [0] https://glyph.twistedmatrix.com/2014/02/unyielding.html

Well, thanks to operator overloading, comparisons or assigning a variable (to a property of an object) can cause a switch. If you really need to get a mental model of what your code is doing, or could do under sufficiently unusual circumstances, gevent doesn't really provide any guarantees beyond raw threading. Glyph (of Twisted fame) wrote a good blog-post about it: https://glyph.twistedmatrix.com/2014/02/unyielding.html

I used that at some point, but I tend to prefer the explicit yield of asyncio to the implicit context change of greenlets. (This blog post is a good explanation of that: https://glyph.twistedmatrix.com/2014/02/unyielding.html)

See "if it used Twisted for the event loop and none of the HTTP stuff:" at http://glyph.twistedmatrix.com/2009/09/what-i-wish-tornado-were.html (Some of these things might have been fixed, I haven't checked)

Original link: http://glyph.twistedmatrix.com/2009/10/unboxing-you-won-see-on-gizmodo…