Curt Clifton

There are many sources of complexity, and avoiding it is the hardest problem in software engineering. I recommend the first half of the Out Of The Tarpit paper. https://curtclifton.net/papers/MoseleyMarks06a.pdf

Not crazy at all; just uncommon. It would be an implementation of the basic recommendation from the excellent paper, "Out of the Tar Pit". https://curtclifton.net/papers/MoseleyMarks06a.pdf

Out of the tar pit is the best paper on this I’ve ever read: http://curtclifton.net/papers/MoseleyMarks06a.pdf

A long, but thoroughly worthwhile read on this topic is Out of the Tar Pit [1]. [1]: http://curtclifton.net/papers/MoseleyMarks06a.pdf

The model described in the parent comment is essentially using the database (in this case PostgreSQL, but any RDMS would do) as the hexagonal "core" in which adapters plug in to. This is a powerful pattern that works very well when you use the full features of the RDMS like constraints, triggers, views, etc. This does require "coupling" to the RDMS-specific features, which makes migrating to an alternative system difficult, but in practice this rarely happens if you choose a strong RDMS from the beginning. You can certainly use the database as a "dumb storage" tool in the hexagonal architecture, that is, as just another adapter. But most of the time you'll end up re-creating RDMS features in poorly written/documented application code that has to interact with the database anyways. Why not just do it all in the database? With a RDMS core, hexagonal adapters can be pure functional components, making them much easier to reason about and maintain. For more on this idea, and how to avoid pitfalls with the hexagonal pattern, I recommend reading Out of the Tar Pit [1]. It's a short but highly influential paper on "functional relational programming". [1]: https://curtclifton.net/papers/MoseleyMarks06a.pdf

I always find myself sitting down to read Out of the Tar Pit[0] at least a couple times per year. It has been—and continues to be—one of the most seminal texts in my career. I still remember the first time I read the following passage on complexity, and how it just turned on all the mental light bulbs: >> Essential Complexity is inherent in, and the essence of, the problem (as seen by the users). >> Accidental Complexity is all the rest — complexity with which the development team would not have to deal in the ideal world (e.g. complexity arising from performance issues and from suboptimal language and infrastructure). >> Note that the definition of essential is deliberately more strict than common usage. Specifically when we use the term essential we will mean strictly essential to the users’ problem (as opposed to — perhaps — essential to some specific, implemented, system, or even — essential to software in general). The best skill I've learned, and continued to practice and improve, is the ability to strip how we talk about problems we want to solve with software down to what's truly essential to the problem. Making a habit of doing so helps clarify the contours of the problem itself, and improves discussions around solving because the boundaries of what's truly essential become clear—and then everyone involved knows that every choice we make from that point is additional, accidental complexity we are adding to the problem ourselves. Far too often I have seen even…

You're being condescending, telling me to "learn the theory". We've clearly had different experiences and work with different kinds of systems. You're automatically discounting what I say by assuming I must be a newbie because I disagree with you. No. We both have things to learn from each other and you telling me "go RTFM, newbie" is shutting down that opportunity. > Virtually all of the systems I write have at least two back-ends to maintain that flexibility. It sounds like that's a worthwhile tradeoff for you, but it is a tradeoff, giving up some of the unique power of each individual database in order to support both. Realize that most people don't do this. > A good object model is NOT a relational model, and a good relational model is NOT an object model. Maybe this is our biggest disagreement. I believe there is a model that can get the biggest benefits of both. I'm not alone in this; papers in Software Engineering, "Out of the Tar Pit", prescribes this as a possible solution to a lot of woes in software engineering: "The classical ways to approach the difficulty of state include object oriented programming which tightly couples state together with related behaviour, and functional programming which — in its pure form — eschews state and side-effects all together ... We argue that it is possible to take useful ideas from both and that — when combined with some ideas from the relational database world - this approach offers significant potential for simplifying…

I find similar arguments around SQL. So much time & frustration expended simply to avoid typing out the magic database commands... And the constant ego trips attempting to outperform 30+ year old query planner codebases on 7-way+ joins by using baby's first ORM. > the tar pit If you find yourself stuck in one of these, I strongly recommend giving this a shot: https://curtclifton.net/papers/MoseleyMarks06a.pdf "but it won't scale" We are in the era of hyperscale SQL engines. Database engines that are spread out across multiple servers, racks and buildings. Engines so vast & complex the compute & storage responsibilities have to be separated into different stacks. But, they (the good ones) still work just like the old school approach from an application perspective. The workload necessary to actually saturate one of these databases would be incredible. I some days wonder if Twitter could be rewritten on top of one without much suffering. And, if you aren't trying to go big and bold or spend a bunch of money, there's always SQLite. It also supports basically all the same damn things. It can run entirely in memory. It has FTS indexing. Your CTE-enabled queries will work just fine on it. If you find SQLite doesn't scale with you, swapping to a different engine really isn't that big of a deal either. You will have some dialect conflicts but it's generally very workable, especially if you use some thin layer like Dapper between your code and the actual connection instances.

> Wouldn't it be great if the man page of getaddrinfo mentioned those (mine only mentions gai.conf) That's a huge part of the problem. Was looking for a monograph to pass on my successor that wasn't so familiar with linux - modern linux - as in systemd, docker, nssswitch, pam_homed, network-manager is penetrable but more often than not I'm just looking at the c source in github and I am in my 30ies, starting in my teens running linux nonstop and I'm still throwing up my hands every other day... Add time constraints and pressure to deliver (don't waste your time understanding this, just do xyz) and here we are - on the other hand it's not okay that you have to devote years of trail and error to get a comprehensive undestanding of the system. I guess a nuclear power plant has at least a training plan and complete reference book - that still needs to be read and grokked and trained upon but modern linux is often just undestandable by reading the source if you hit a problem - for some projects like things in the freedesktop ecosystem and partly systemd even that is kind of difficult because i.e. this bug https://github.com/systemd/systemd/issues/19118 explains the problem - there is no overview documentation, no clear way to look up what's happening, not even a good way to introspect and it's several components that interact with each other fail subtly. I've chosen this one because I've also hit it, not because I want to blame systemd, which is not so bad there are much worse…

You keep saying what they don't want to, but remember that most users of Excel are not programmers in the traditional sense. I'd wager that most of them aren't even aware of the things you say they don't want. It's not that you don't need to follow modern software practices. It's that they don't know about them to follow them or not. Further, Excel is almost pure thought stuff. The distance between the user's idea and their implementation is almost as small as we can get without investing in a lot of educational outreach. And then they end up going crazy with macros because they don't know there are other or better ways to do it. Also, "modern" Excel (not sure which version, maybe 2007 or 2010?) has largely obviated the need for macros with the addition of tables and functions for interacting with them. It turns Excel into a kind of relational database permitting something close to functional relational programming as described in "Out of the Tar Pit" [0]. [0] http://curtclifton.net/papers/MoseleyMarks06a.pdf

>rather than merely apply mathema^H^H^H^^H^H^H^H logical operations within constraints. This is what all programming and software engineering is - logical operations within constraints. Types make those constraints more explicit, and make them work for you rather than against you. They make refactoring of large-scale systems more tractable and less painful, which is invaluable when requirements are shifting. And they can eliminate entire classes of errors, bugs, and security vulnerabilities at compile time, which is invaluable in a day and age when everything is getting hacked. As type systems become increasingly sophisticated, those benefits become even more true. And when you take into account all the man hours spent on a system, not just in building the quick and dirty initial prototype, but in debugging it for production, and then maintaining it over a long period of time with eventually 100% turnover of the initial dev team, then types reduce time and effort spent too. You should read the famous paper Out of the Tarpit. Type systems are one of the best solutions to the tarpit of complexity that large systems inevitably incur. http://curtclifton.net/papers/MoseleyMarks06a.pdf

ClojureScript is awesome. There are several tools available for live-coding (REPL-driven development) such as more recent shadowcljs or the older Figwheel. Rum is a powerful wrapper over React.js and you can live code and see instantaneous results in the browser (using shadowcljs or figwheel). ClojureScript's strengths largely stem from Clojure -- collections as fundamental types, REPL, macros, immutable datastructures* and lovely syntax. As far as cons go, it takes some time to start up a REPL and making release builds is an extra step, but should work just fine unless you're a newbie like me and you leave strange unnecessary lines in your code. Javascript and js/CSS interop is there, but it's not super obvious and takes some getting used to what it looks like -- luckily there are enough online writeups to make that process easier. The addage "why clojurescript? Because clojure rocks and javascript reaches" sums up nicely the motivation. You can have largely isomorphic code on the front-end and back-end, which lends nicely to creating server-side rendered versions of SPAs for search engine indexing and fast load times. However, the server-side copy and the client rendered copy must match exactly for this to work, which can be a bit of a "thread the needle" kind of problem. * immutable datastructures are awesome, contrast with mutable datastructures and "location oriented programming" -- any time you "edit" a value you're actually creating a new datum. This is…

As others point out, more detail about what you're looking for. I found Out of the Tar Pit[1] somewhat useful. I thought the back half of the paper was disappointing (sorry, functional is not the cure to all problems, and state is something inherent and we must deal with it), but the definition of "essential complexity" and "inessential complexity" from that paper are invaluable, and too often I see people/devs/PMs going "simpler is better" where "simpler" would not address essential complexity: i.e., their simpler == broken, for the use case at hand. But once you have that, then when you see a fragile system, you can start looking it through a more productive lens of "okay, what of this must I keep, and what complexity can I dispense with?" [1]: https://curtclifton.net/papers/MoseleyMarks06a.pdf

> Immutability is not as useful in a single-threaded event-loop like nodejs. Concurrency is not the only, or even primary, benefit of immutability in software. State and state management is the number one driver of complexity in software, and complexity itself is the number one problem that makes building software hard. [0] > For me, there was little value in using it over nodejs for single server CRUD apps with postgres. You can use clojure on nodejs with the value that clojure is a much, much better language than javascript. [0] http://curtclifton.net/papers/MoseleyMarks06a.pdf

Is it more complicated though ? I've been developing for over 30 years, and I don't feel like its got any worse, I think it's the nature of the medium. These days I fight with npm or nuget dependencies problems, 20 years ago I was trying to write Delphi code to do simple REST requests, the complexity moves around but is still there. Out of the Tar Pit (http://curtclifton.net/papers/MoseleyMarks06a.pdf) was written 15 years ago, and it's all about curbing complexity

> the "tar pit" paper that has been discussed a lot here on HN Did you mean Mosley and Marks' "Out of the Tar Pit" http://curtclifton.net/papers/MoseleyMarks06a.pdf?

> A full definition of "simplicity", as with "complexity", lies somewhat in the eye of the beholder. Perhaps "somewhat", but not entirely or arguably even mostly: http://curtclifton.net/papers/MoseleyMarks06a.pdf

Came here to add Out of the Tar Pit. Here’s my clipboard: http://curtclifton.net/papers/MoseleyMarks06a.pdf

Paper link: http://curtclifton.net/papers/MoseleyMarks06a.pdf

Relying primarily on relational modeling reminds me of Out of the Tar Pit[0]. The well known paper suggests a combination of functional programming and relational data modeling. [0] http://curtclifton.net/papers/MoseleyMarks06a.pdf

I've never understood why people like this paper. Over the years I've evolved a bite-sized rebuttal, if anybody wants to fight me: The authors' "ideal world" is one where computation has no cost, but social structures remain unchanged, with "users" having "requirements". But the users are all mathematical enough to want formal requirements. The authors don’t seem to notice that the arrow in "Informal requirements -> Formal requirements" (pg 23) may indicate that formal requirements are themselves accidental complexity. All this seems to illuminate the biases of the authors more than the problem. Link to the paper, for convenience: http://curtclifton.net/papers/MoseleyMarks06a.pdf. I think it's a siren and takes away attention that could be spent on better papers.

Art of the propagator It is like a discovering a new programming paradigm. Something new and different, not a Lisp, not a Datalog, weirdly distributed, with interesting properties, but powerful. There are adjacent things, BLOOM project and Eve language seems to have discovered something simmilar, and Edward Kmett working on his new language is using propagators quite heavily. https://pdfs.semanticscholar.org/755c/48fd10aa303497ef849977c36529c0bb… Sci-fy to go with it: probably Ted Chiangs Stories of your Life (or its movie adaptation Arrival) Out of the tarpit I thing this is a paper that might have lead to Clojure, React, Elm, e.t.c ... but not really, a weird parallel reality where we really like reactive relational databases. http://curtclifton.net/papers/MoseleyMarks06a.pdf

"Out of the Tar Pit" (Ben Moseley and Peter Marks) http://www.curtclifton.net/storage/papers/MoseleyMarks06a.pdf Software complexity related to mutable state.

The main problem with ORM is the "object" part. I feel object-oriented programming had its chance and it failed. Both event sourcing and relational models (="table modelling") are better models for programming systems. So by using an ORM, you are (at least in the name, and as traditionally done) taking a better programming model (relational models) and mapping it to an inferior one (OOP) -- also getting a ton of leaky abstractions involved too. I wish there were more "ORMs" that were made with the assumption that Object-Oriented Programming is BAD, and started out with assumptions of wanting to program either in an event sourcing model and/or a relational model, and see where that would take things. Related: Fred Brooks, The Mythical Man Month (1975): "Show me your flowchart and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won't usually need your flowchart; it'll be obvious." Out of the tar pit paper: https://curtclifton.net/papers/MoseleyMarks06a.pdf -- My take on the software industry today is that OOP failed to produce good and maintainable code, but instead of fixing that, one is adding the idea of micro-services to try to contain the damage done by OOP. If more people moved to event sourcing, functional code and focusing on data over code, there would be much less need for micro-services etc (or, they just become less relevant).

Cool. I've noticed that general program complexity seems to be a topic that everyone acknowledges, but nobody tries to frame objectively. Except for a few people at the fringes. I'm not sure that I understand and/or agree with this link. However, I've written and talked about this topic in the past and the response that I've gotten is usually some variation of, "Cool, but I'm not sure that I get it." It feels kind of like the blind men describing the elephant. People are grasping at something that's probably real, but it's so big that it's going to take a long time before we finally manage to pin down anything concrete. The effort has to start somewhere though. Other interesting occurrences: http://curtclifton.net/papers/MoseleyMarks06a.pdf (out of the tar pit) https://www.sonarsource.com/docs/CognitiveComplexity.pdf (semantic compression) https://www.progsbase.com/blog/flow-charts-of-programming-language-con… I'm not sure any attempt I've seen covers everything, but I think it's interesting that people keep trying to describe something that looks suspiciously similar.