Fred Hébert

Speaking of awesome, check out other stuff Fred Hebert wrote. Learn You Some Erlang For Great Good: http://learnyousomeerlang.com/ His blog posts are great too: http://ferd.ca/ And his new (free) book, Erlang In Anger: http://www.erlang-in-anger.com/

Or in one of my favorite coinages: "Moral Crumple Zones". "Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions." See https://ferd.ca/notes/paper-moral-crumple-zones.html

I recommend https://ferd.ca/the-zen-of-erlang.html starting from "if my configuration file is corrupted, restarting won't fix anything". The tl;dr is it helps with transient bugs.

Have you considered the opposite? Reflected on your own biases? I’m listening to my own experience. Just today I gave it another fair shot. GitHub Copilot agent mode with GPT-4.1. Still unimpressed. This is a really insightful look at why people perceive the usefulness of these models differently. It is fair to both sides without being dismissive as one side just not “getting it” or how we should be “so far” past debate: https://ferd.ca/the-gap-through-which-we-praise-the-machine.html

WDT patterns are highly underrated, even in pure software there's value in degrading/recovering gracefully vs systems that have to be "perfect" 100% of the time and then force user intervention when they go wrong. One of my favorite blogs on the topic https://ferd.ca/the-zen-of-erlang.html that does a great job of covering how Erlang approached the topic, lots of learnings that can be applied more broadly.

Also I had forgotten but here is a more recent follow-up to Fred Hebert's post: https://ferd.ca/a-bridge-over-a-river-never-crossed.html

Elixir. It has the very same concepts (actually it IS a dialect of Erlang - it's just a different front-end for the same compiler), but with a saner syntax. After a while you do Elixir, you'll start grokking Erlang. (Of course, if you need to deep dive into some Erlang code tomorrow, "Learn you some Erlang" or Joe's book. But do your Elixir.) Also, https://ferd.ca/ , the guy behind Erlang in Anger and Learn you some Erlang, has a lot of very well thought-of contents.

Two years after writing A Pipeline Made of Airbags, I ended up prototyping a minimal way to do hot code loading from kubernetes instances by using generic images and using a sidecar to load pre-built software releases from a manifest in a way that worked both for cold restarts and for hot code loading: https://ferd.ca/my-favorite-erlang-container.html It's more or less as close to a middle-ground as I could imagine at the time.

I highly recommend awk because you can learn it in 20 minutes [1] So you can start by spending 20 minutes to learn awk, and then spend 20 years to learn Perl (and use awk in the mean time) [1]: https://ferd.ca/awk-in-20-minutes.html

I believe you’re oversimplifying. The approach that seems most common with serious Erlang systems is to identify the common errors and handle them without crashing. Crashing on less common, or impossible to reproduce, errors allows you to gradually improve your error handling over time if it’s worthwhile, or just allow it to continue to crash into a saner state. See the “bohrbug” vs “heisenbug” discussion: https://ferd.ca/the-zen-of-erlang.html

Agreed on all parts. The Erlang and OTP manuals were very nicely written, and I was able to reason about most aspects of the system pretty well from reading them. I did a bit more research after writing up my comment (my mind got a bit too focused on it to let it go) and found this great resource about handling various system load scenarios: https://ferd.ca/handling-overload.html I'll +1 your pragmatic comment on not adopting tools just because they're there. Again I no longer work in Erlang, but I find the systems, architecture, and problem solving particularly interest piquing. Now I'm off to look up production use-cases where Mnesia was the most pragmatic solution.

Here are mine: - https://franz.hamburg - https://www.project-daily.com Some blogs I follow via RSS (using the awesome NetNewsWire app): - https://brendangregg.com - https://brandur.org/ - https://ferd.ca/

IMO Erlang/Elixir does it best with the "let it crash" mentality. Once you internalize and truly understand how it works, you won't want to do error handling in any other way. The tldr; handle errors you can handle, let everything else crash and rely on the runtime to bring you back online. The zen of erlang (https://ferd.ca/the-zen-of-erlang.html) goes more in depth if you're curious.

Knowing how to use OTP is definitely something worthwhile if you're going to stick with the BEAM platform: https://ferd.ca/ten-years-of-erlang.html

I love having a variety of these types of sites available for a new language. Different learners will react well to different styles. My personal favorite is AWK in 20 Minutes: http://ferd.ca/awk-in-20-minutes.html The AWK Programming Language book is also great. Nice work.

Here's a great transcript from which I learned "why restarts help" (somewhere in the middle, the rest is well worth reading as well): https://ferd.ca/the-zen-of-erlang.html

My favorite Awk reference is this: https://ferd.ca/awk-in-20-minutes.html Also, a handy trick is to combine awk and cut. For example I had a log line that had a variable amount of columns just in one field, but immediately after the field was a comma. I cut based on the comma: cut -d, -f1,2 and then awk'd the last column: cut -d, -f1,2 | awk '{ print $2" "$5" "$NF }' So, sometimes awk and cut can help each other.

Great video! I don't suppose there is an online version of your presentation for Tout est terrible[1] is there? I looked but couldn't find one. Ca va si c'est suelement en francais. [1]: https://ferd.ca/tout-est-terrible.html

Erlang/OTP and Elixir on top of that, brings more than 30 years of experience of building Reliable, Scalable and High Available Systems. It's difficult to fight this. Somebody new to Erlang can get a feel of what Systems Architecture in Erlang really means from a great article by Fred Hebert: "The Hitchhiker's Guide to the Unexpected" https://ferd.ca/the-hitchhiker-s-guide-to-the-unexpected.html

ferd did a great write-up on how Erlang approaches reliability: https://ferd.ca/the-zen-of-erlang.html It's more about assuming things will fail by default with good ways of handling it built-in at the language level. A lot of people also find its inventor's thesis enlightening. Here it is: http://ftp.nsysu.edu.tw/FreeBSD/ports/distfiles/erlang/armstrong_thesi… And for those concerned, there's also at least one project to write the native functions in a safer language to reduce their bugs a bit: https://github.com/hansihe/Rustler

I loved the heisenbug/bohrbug distinction in the zen of Erlang [1]: > Basically, a bohrbug is a bug that is solid, observable, and easily repeatable. They tend to be fairly simple to reason about. Heisenbugs by contrast, have unreliable behaviour that manifests itself under certain conditions... It continues talking that for some kind of heisenbugs, one smart strategy is just to restart to a known state, and forget about this one in a million failures (let it crash). [1] https://ferd.ca/the-zen-of-erlang.html

I'm the author of this article. If you find it interesting, here are some links to a couple other articles I've really enjoyed on the topic: # Queues don't fix overload http://ferd.ca/queues-don-t-fix-overload.html # There is no happy path in programming http://multithreaded.stitchfix.com/blog/2016/01/29/no-happy-path-in-pr… Also, if you're looking for a whole book on systems thinking, albeit in a more general context, it's worth checking out "Thinking in Systems: A Primer" by Donella H. Meadows. Once you start thinking about this stuff it's impossible to not see it everywhere. But for programmers, ops people, etc. that's a good thing... because our jobs depend on it.

If you view the comma and semicolon as separators instead of terminators it works much better. It is also what they really are. A semicolon separates two clauses, either two if/case/receive clauses or two function clauses and means or. Either choose the first clause or the clause following the semicolon. An if/case/receive is terminated with an 'end' while a function is terminated with a dot. Fred Hebert, the author of Learn You Some Erlang, wrote this on syntax: http://ferd.ca/on-erlang-s-syntax.html

I couldn’t agree more. Security is almost always user hostile (speaking from a UX perspective). I am NOT advocating that we remove security for obvious reasons (a hacked app is also user hostile). HOWEVER - if we can just acknowledge that security is antithetical to an easy to use, user friendly app then we can make appropriate decisions moving forward. One of my favorite sayings is “if you are not careful, you are going to secure yourself right out of business”. Ease of use is a real thing, and if you don’t figure out a way to make a secure app that is also very easy to use, someone else will and your pleas of “but it is not secure” are going to fall on deaf ears. Honestly, baseline security is part of “essential complexity” https://ferd.ca/complexity-has-to-live-somewhere.html. Essential complexity can’t be removed, but it CAN be moved around. Right now we are talking about the essential complexity of managing security through managing a user’s session length. The advocated solution is to make sessions short so that they expire quickly. This seems to remove all “accidental complexity” so that we are only dealing with the essential complexity. But this is misleading. There is still complexity in juggling those short sessions. As a designer you think this solution is simple, but what you have done is moved that essential complexity over to your users. THEY must now manage the impact of short sessions. The complexity does not go away, you just moved it to your users, hence…

I'm all for bending your text editor to your will, but a few of these tips gloss over the core of vim: the shortcut language it offers and the composability of its parts of speech [1]. For example, remapping the paste key to paste + move to end is cool, but `ppppp` to paste 5 times circumvents vim's killer repetition. Want to paste 100 copies? (hopefully you do this rarely). Just type `100p` and you're done. I won't bother typing out the obvious sequence of repeated `p` presses, but you get the drift. Also, remapping v to progressively select larger surrounding text objects will keep you from learning that you can perform any action on the 3rd parent curly braces by following the action with the text object `3a{` or `3a}`. Select it: `v3a{`, delete it: `d3a{`, comment it out `gc3a{` ([2]), whatever. If there's an action in vim you can perform it on a text object which can usually start with some count. I really do mean language, too. When I'm using vim, I feel a lot more like I'm communicating with my computer than trying to figure out how to do what I want to do. The actions are there: delete, change, select. The direct objects are there: braces, this line, line number N, end of this word, the next character C, this paragraph. The prepositions are there: inside, outside, up to (inclusive, exclusive). Heck, the sensible defaults feel a whole lot like the "understood" words that you can leave out of sentences. Instead of "navigate to line 30", you can just use the…

I write it fully acknowledging that programming language flamewars are pointless, but this article just shows that you don't even have to try hard to create a biased comparison. Here's the essential difference between Go and Erlang: Go gets most of the things right, Erlang gets way too much wrong. So what does Go gets right but Erlang doesn't: * Go is fast. Erlang isn't * Go has a non-surprising, mainstream syntax. You can pick it up in hours. Erlang - not so much. * Go has a great, consistent, modern, bug-free standard library. Erlang - not so much. * Go is good at strings. Erlang - not so much. * Go has the obvious data structures: structs and hash tables. Erlang - no. * Go is a general purpose language. Erlang was designed for a specific notion of fault tolerance - one that isn't actually needed or useful for 90% of the software but every program has to pay the costs * Go has shared memory. Yes, that's a feature. It allows things to go fast. Purity of not sharing state between threads sounds good in theory until you need concurrency and get bitten by the cost of awkwardness of having to copy values between concurrent processes So sure, if you ignore all the major faults of Erlang (http://damienkatz.net/2008/03/what_sucks_abou.html, http://www.unlimitednovelty.com/2011/07/trouble-with-erlang-or-erlang-… http://ferd.ca/an-open-letter-to-the-erlang-beginner-or-onlooker.html, http://sacharya.com/tag/erlang-sucks/) it compares very favorably to Go. You…

The idea sounds related to the distinction between essential and accidental complexity. https://ferd.ca/complexity-has-to-live-somewhere.html I really like the article above in that complexity does have to live somewhere. If you have a hard problem to solve, there is a minimum level of complexity (essential complexity) that can't be further reduced, only moved from one place to another. And yet we often add accidental complexity by making bad choices (thinking of you, microservices). If your choice is bad enough, you get all the disadvantages (challenges with distributed systems, complicated debugging), but without any advantages (a distributed monolith doesn't get you any deployment or team independence). Telling one from the other, well, is something I'd like to learn and that seems to require both experience and some sort of intuition.

This write up on a paper about business process redesign is such a fine read https://ferd.ca/notes/paper-moving-off-the-map.html

There's certainly many dimensions along these lines that if you don't think about them in relation to the type of software development you are doing, and what you have done, you will take a long time to improve.. But this being the model for all software development (such that all others are bad) strikes me as a missing chapter from printf. https://ferd.ca/the-little-printf.html

Sure, you can shut down and restart your entire application. You could do that back in 1990 without containers, too. The thing is that Erlang does hot reload at a per-function (or -according to Hebert- sometimes more-fine-grained) level, so nuking the entire program and paying the cost to start it up again is not at all the same thing as -say- using a not-absurdly-priced AWS Lambda [0] or similar to get per-function hot reloading. By the way, have you read "A Pipeline Made of Airbags"? If not, you should give it a read: <https://ferd.ca/a-pipeline-made-of-airbags.html>. It might be old news to you, maybe, but maybe not. [0] Good luck finding one, though.