Joe Duffy

http://joeduffyblog.com is great, albeit it's far from being daily. It has long posts about operating system and programming language design

It's worth mentioning Microsoft's Midori, a follow-on to Singularity: https://en.wikipedia.org/wiki/Midori_(operating_system) Joe Duffy, who worked on Midori, wrote some on his blog about it: https://joeduffyblog.com/2015/11/03/blogging-about-midori/

I couldn't give a better reply than this author: https://joeduffyblog.com/2010/01/03/a-brief-retrospective-on-transacti… Also, a phenomenal writing (as are his other posts) on the whole concurrency landscape, see: > A wondrous property of concurrent programming is the sheer number and diversity of programming models developed over the years. Actors, message-passing, data parallel, auto-vectorization, …; the titles roll off the tongue, and yet none dominates and pervades. In fact, concurrent programming is a multi-dimensional space with a vast number of worthy points along its many axes.

An older yet still excellent take on this topic is https://joeduffyblog.com/2016/02/07/the-error-model/

Project Midori was really good read, a lot of good stuff came from it https://joeduffyblog.com/2015/11/03/blogging-about-midori/

I am not familiar with any details, but the haskellians seem to still be working with STM. At least there are a bunch on talks online about it. I had spent most of the 90s working with transactional systems and while they are useful and powerful, they aren't magic and require awareness to keep from making errors. Joe Duffy wasn't wrong when he says those problems are well known. The solutions normally are either Don't Do That Then, or managing the policy details and logic yourself, which wasn't giving them the easy concurrency they were trying for. Edit: He has a followup post at http://joeduffyblog.com/2010/05/16/more-thoughts-on-transactional-memo…

The more I research Singularity, the more I like it. I deep dived into all the design docs years ago and the amount of rethinking existing OS infrastructure is astounding. Joe Duffy has some great blog posts on Midori (OS based on Singularity) here: http://joeduffyblog.com/2015/11/03/blogging-about-midori/

In any case, these arguments have been hashed many times. I always recommend reading Joe Duffy's post on errors for the curious. http://joeduffyblog.com/2016/02/07/the-error-model/

http://joeduffyblog.com/2015/11/03/blogging-about-midori/ There is also a really great blog about Singularity’s “rebirth” experimental OS, Midori, that continued in its footsteps.

The follow up MS project Midori has one of my favourite series of blog posts written about it http://joeduffyblog.com/2015/11/03/blogging-about-midori/

Joe Duffy has blogged about it, excellent read. http://joeduffyblog.com/2015/11/03/blogging-about-midori/

I believe the idea behind could be the same for Mindori project inside Microsoft years ago: "Midori was a research/incubation project to explore ways of innovating throughout Microsoft’s software stack. This spanned all aspects, including the programming language, compilers, OS, its services, applications, and the overall programming models. We had a heavy bias towards cloud, concurrency, and safety. The project included novel “cultural” approaches too, being 100% developers and very code-focused, looking more like the Microsoft of today and hopefully tomorrow, than it did the Microsoft of 8 years ago when the project began." [1] Later some of this ideas became core features on Windows. For a great overview about Mindori project: [1] http://joeduffyblog.com/2015/11/03/blogging-about-midori/

I want to be sold on Fuchsia/want it to succeed because of security. If it succeeds the UI will change a bunch of times, and there will be many UIs. Very happy to be corrected, but it seems to be the best hope since Midori was canned, and Fuchsia has the great advantage of being open source. By the way, I've casually looked for but not yet found a comparison of Midori and Fuchsia approaches based on what is known about each, maybe following structure of posts in http://joeduffyblog.com/2015/11/03/blogging-about-midori/ ... I would love to read that!

> It is almost never okay for a library to abort the process. Using Joe Duffy's distinction [1], bugs aren't recoverable errors. Error handling admits the necessity of error recovery, but you generally don't have such a fine-grained policy for bugs. A dual-use of exceptions as a means to signal bug conditions is unfortunate, and I believe Mundane's policy is clearly following this distinction. Also, panicking in Rust can be caught in the same thread [2]. So it is actually much more flexible than process abort---if you need a general, umbrella protection against bugs, here you have a way. [1] http://joeduffyblog.com/2016/02/07/the-error-model/ [2] https://doc.rust-lang.org/stable/std/panic/fn.catch_unwind.html

Joe Duffy's series on the Midori system goes into great detail about the problems in various error-handling scenarios and how they went about dealing with them: http://joeduffyblog.com/2016/02/07/the-error-model/

That's a relatively minor syntax issue, fixable with something like Kotlin. All runtimes need to erase types at some level. Otherwise ArrayList<Foo> and ArrayList<Bar> would end up compiling identical versions if both Foo and Bar are reference-only types, which just wastes memory. At some level the compiler and runtime need to merge duplications - in C++ that feature is called COMDAT folding, or used to be. .NET has had serious problems with code duplication in the past. Here's an excellent blog post by a Microsoft engineer on it: http://joeduffyblog.com/2011/10/23/on-generics-and-some-of-the-associa…

Haven't you heard of Singularity or Midori? http://joeduffyblog.com/ https://www.infoq.com/presentations/csharp-systems-programming There are quite a few OS projects written in system programming languages with GC, all the way back to Mesa/Cedar at Xerox PARC. If you want to have an idea, check the Project Oberon book. http://people.inf.ethz.ch/wirth/ProjectOberon/index.html This is the 2013 re-edition, it was originally published in 1992.

The most annoying thing about all this is that the central argument of this article "Separate recoverable errors from bugs" never made it to a widely used imperative language. C# had the opportunity but blew it. Java mixed the two kinds of exceptions up completely and checked exceptions just added insult to that injury. The best implementation I have seen for an imperative language is in Midori (The language used in Microsofts research OS with the same name). http://joeduffyblog.com/2016/02/07/the-error-model/#bugs-arent-recover… It's basically "C# done right". The blog post is well worth reading.

Assorted pointers: http://danluu.com/clwb-pcommit/ on the speed of modern persistent storage and the need to get the OS out of the way for best performance http://blog.erratasec.com/2013/02/custom-stack-it-goes-to-11.html on the speed of modern networking and the need to get the OS out of the way for best performance http://joeduffyblog.com/2015/12/19/safe-native-code/ and the rest of that blog on Microsoft Research's Midori research OS, based on C# and type safety rather than traditional process-based (i.e. Memory Mapping Unit-based) security https://sel4.systems/ for a proven-secure (with many asterisks) native-code microkernel MirageOS (OCaml) and similar (often Rust-based) unikernels ... and of course, the rise of the heavyweight microkernel under the name "hypervisor" Each of these ideas has very serious downsides, but each also has earnest proponents.

I don't know, either, but, for what it's worth, Joe has said that Midori began around ~2008. Edit: 2003 according to above comment. I got my year from "The project included [...] looking more like the Microsoft of today [...] than the Microsoft of 8 years ago when the project began" - dated 2015. http://joeduffyblog.com/2015/11/03/blogging-about-midori/

Learning a bit of static typed language will definitely benefit you. I would start with Rust and then switch to M# when it gets released (MS has a strong track record of delivering brilliant languages - C#, F#): http://joeduffyblog.com/2013/12/27/csharp-for-systems-programming

I haven't heard anyone anywhere say that "typed exceptions are an antipattern" just "typed exceptions _as implemented by Java_ are an antipattern". The general problem is well set forth by Anders Hejlsberg, Bill Venners and Bruce Eckel in _The Trouble With Checked Exceptions_ [1]. Briefly, typed exceptions in Java are not genericizable. Everything must be known at compile time. That is to say, you cannot write: interface ThrowingRunnable<E extends Throwable> { void run() throws E; } Instead, the best you could do is manual multiplication of interfaces: interface AThrowingRunnable { void run() throws ExceptionA; } interface BThrowingRunnable { void run() throws ExceptionB; } // ... snip thousands more ... Or give up and erase all the type information by saying `throws Exception` or `throws Throwable`. Genericizing throws in particular was tried in Midori [2] and worked out really well (by report). In addition, several less-than-completely-obscure languages are starting to experiment with the algebra of effects in general (as opposed to error handling in particular). Pony [3], OCaml [4], and others are experimenting with bringing what Koka [5] (among others) to the masses. [1]: https://www.artima.com/articles/the-trouble-with-checked-exceptions [2]: https://joeduffyblog.com/2016/02/07/the-error-model/ [3]: https://www.ponylang.io/ [4]: https://ocaml.org/manual/5.2/effects.html [5]:…

> you end up needing to propagate a bunch of extra stuff everywhere. Declaring a function as non-IO is a contract to your callers that you don't do IO. You don't need to do it. You can write everything in IO if you choose. You can also call into the wonderful ecosystem of libraries, because IO functions can call other IO functions, as well as non-IO functions. There is only ever friction if you declare your function to be IO-free. If you declare your function to be IO-free, but call IO from inside it, it's a compile error because of course it is. So why bother declaring anything IO-free? If you do arbitrary IO in a Parser, you can't back-track. If you do arbitrary IO in Parallel code, you invite race conditions. Transactions is my favourite example, though: .NET [1] > Disillusionment Part I: the I/O Problem > It wasn’t long before we realized another sizeable, and more fundamental, challenge with unbounded transactions [...] What do we do with atomic blocks that do not simply consist of pure memory reads and writes? (In other words, the majority of blocks of code written today.) This was not just a pesky question of how to compile a piece of code, but rather struck right at the heart of the TM model. Scala: [2] > ScalaSTM does not have the goal of running arbitrary existing code, which is where most of their problems arose. Java/Akka: [3] > STM is considered as a failed experiment Clojure: [4] > Very simply the side-effects will happen again.…

I'm sorry about the code quality, I never got around to refactoring it, I was just trying to get it to work safely and reliably. Without any money destruction or creation. Checkout MVCC.java and TransactionC.java for multiversion concurrency control. It's far easier to understand. It takes less code to implement multiversion concurrency control. Software transactional memory is great and I like it but it can have scalability problems. I really enjoyed Joe Duffy's blog series on Midori, the Microsoft .NET software transactional memory implementation. http://joeduffyblog.com/2015/11/03/blogging-about-midori/ STM essentially can get transpiled into a for loop modifying a log area and then a CAS instruction to the actual memory. STM can be thought of as multiversion concurrency control because the operations occur on a log. But the problem is that most things are not transactional. How do you reverse an API call?

Once I got used to writing in Rust, I found it really hard to mess up. I sort of like it for applications, although some idioms can be a bit verbose and hard to parse visually and mentally. First, thanks to default immutability of variables and the borrow checker, I trust that I'm far less likely to accidentally mutate something and cause an action-at-a-distance bug. Second, I absolutely love idiomatic Rust error handling of returning a Result (especially combined with the question-mark operator). It forces me to deal with error conditions. Unlike C# or Python or Java or C++, I don't have to worry that any code could throw an exception at any time. (Rust code can panic, but panics generally represent unrecoverable errors — see http://joeduffyblog.com/2016/02/07/the-error-model/#bugs-arent-recover… ) And unlike Go or C, I don't have to remember to check the error every single time. I still have to write functionality tests for Rust code, but once I've done that I feel like the code is mostly locked down. It's harder to get to that sense of security in other languages.

It's good to see so much focus on errors. They are essential when trying to build resilient systems. But our approaches are still very immature. First, to make it clear, this article appropriately points out that exceptions are still necessary and relevant. I disagree with some of the use-cases given, but it's important to recognize that exceptions should still exist in programming languages. Joe Duffy's article about Midori's error model [0] is in my opinion the best reference to actually understand the difference between exceptions and errors and why it's so important to get right. It's a very good article, and it has been posted here before; if you are interested in error handling it's a must read. Now, about errors as values. Treating errors as values is practical, and in modern programming languages, relatively ergonomic. That said, we already have some other comments in the thread pointing out how sticking to just "errors as values" is often not enough (btw, the article uses Rust, not Go, but anyway...). And it's also important to clarify this: errors are such an integral part of our programs, and have so much to do with flow control, that I don't believe thinking about errors just as values is enough. Sure, they might be "just values" under the hood, but in all programming languages we see either optional results or syntax sugar to be able to handle errors more gracefully. And in most cases, we still feel it's not enough (or it's enough to be practical, but not to…

I follow Joe Duffy in recognising that we should make a distinction between "errors" and "bugs" [0]. IOException is an error; out of bounds access, division by zero, assertion failures, and things like ClassCastException and NullPointerException in Java, are bugs. For errors, you should do error handling, which means you want the compiler to make sure you don't miss them. For bugs, you want to let the program bomb out, either aborting entirely or stopping at some high-level boundary from which it's possible to sanely continue. In Java, unchecked exceptions are a mostly adequate mechanism for that. Rust's catch_unwind is better, because it makes stronger guarantees [1]. Erlang's approach of terminating the thread and throwing away its heap is also very good, if you can apply that. [0] http://joeduffyblog.com/2016/02/07/the-error-model/#bugs-arent-recover… [1] https://doc.rust-lang.org/std/panic/fn.catch_unwind.html

I highly recommend reading The Little Typer if you want a great book that bridges math and code. It starts out describing Pi, a Lisp with some interesting restrictions (limited recursion, types can have values, etc.). They build up some cool stuff like vectors with the size encoded in the type. All of a sudden, they explain that equality is a type, and any value of said type is a proof! Turns out you can think of many proofs as manipulating data structures to get a value of a certain type. I wonder how long until we get a somewhat mainstream language with pi types. I know Rust considered adding them. And I recently learned that Rust does allow for quantification over lifetimes^[1]. I could certainly see a language that implements dependently typed arrays. Midori for instance looked into eliding bounds checks with compiler proofs^[2]. [1]: https://doc.rust-lang.org/beta/nomicon/hrtb.html [2]: http://joeduffyblog.com/2016/02/07/the-error-model/

What we do with the spin lock classes that we use in our products is a SwitchToThread call: https://msdn.microsoft.com/en-us/library/windows/desktop/ms686352(v=vs… If the SwitchToThread call returns False, then the code reverts to a Sleep(0) call instead. Credit to StackOverflow and Joe Duffy's excellent article on this: https://stackoverflow.com/questions/1383943/switchtothread-vs-sleep1 http://joeduffyblog.com/2006/08/22/priorityinduced-starvation-why-slee… However, as mentioned in one of the SO replies, you still need to make sure that you don't use locks that use loops on calls like this when those locks will have to wait on anything long-running - you're just going to burn CPU/battery needlessly. So, only use such locks when you know that the protected code doesn't involve any long wait states, or modify the code to back out even further into a proper kernel wait after looping X times. Personally, I agree with second SO answer and don't like such constructs, and would personally say just go with a straight-up kernel wait if there's any question about whether such conditions could exist now or in the future.

The brilliant thing about capability-based security is that it turns classic object-oriented programming patterns into security patterns. Access control is implemented by encapsulating private members behind a restricted public interface. Polymorphism allows you to add wrappers around an object implementing arbitrary security policies. E.g. want to revoke access to this object after some time? Add a wrapper where the methods forward to the real object until revoked, and then throw exceptions instead. What's interesting is that these are patterns we are doing anyway, in real code, not for the purpose of security but for the purpose of correctness and maintainability. Because we were doing these things anyway, our programming languages provide really good tools for making them work naturally, and we already know how to think about them. On the other hand, the competing model -- access control lists (ACLs), and other types of externally-specified policy -- are not things we were doing anyway. They're external concepts bolted into the system awkwardly. Because of this, they tend to be painful to maintain, which in turn means people often don't bother. One of the clearest examples of how ACLs go against the grain: An ACL is a list of entities who are allowed to access a resource. These are effectively pointers pointing in the opposite direction of all other pointers in the system: ACLs point from the callee to the caller. This creates all kinds of problem. Just try to…