Gregor Hohpe

I recommend you to buy some proper books to get familiar with well written design patterns. Classics are "Design Patterns" from the Gang of Four (GoF), "Pattern-Oriented Software Architecture" by Buschmann et al. or "Enterprise Integration Patterns" by Hophe and Woolf which patterns are partly available on the website https://www.enterpriseintegrationpatterns.com/. The first two books I consider almost a must read. Actually it's not about reading them but learning the patterns. More about this here: https://www.industriallogic.com/papers/learning.html

This isn't exactly what you want, but you might find both of the following books helpful. With respect to using queues/messaging: http://www.enterpriseintegrationpatterns.com/ And with respect to understanding this stuff in general: http://dataintensive.net/

>> i can't tell the difference > i did see differences. Spot the difference. ;-) > but i can do the same in pike if i tried, and i expect in lisp and other languages too. Absolutely you can! You can even do it in C: write yourself a message-passing library in C. You might want to call it Objective-C. Or do a VM for Smalltalk. > where then is the great revelation that comes from smalltalk? The claim was not that the great revelation came from Smalltalk, but that it came from message passing: >>> Message passing belongs up there with lisp, forth and pure functional programming as paradigms that are worth learning for "the profound enlightenment experience you will have when you finally get it." Smalltalk isn't even mentioned in that little section about the enlightenment. And in fact, Smalltalk's form of message-passing is pretty limited, it only just extends beyond method invocation and it certainly can be (and is) frequently used just like method invocation. If you want to do more sophisticated things, you mostly have to go via the DNU handler, which is a bit hacky. And in fact, Alan Kay's famous OOPSLA '97 quip "I made up the term object oriented. And I can tell you I did not have C++ in mind." was followed immediately with the slightly less famous "So, the important thing here is: I have many of the same feelings about Smalltalk". https://www.youtube.com/watch?t=634&v=oKg1hTOQXoY&feature=yout… And even message-passing is much broader: for example,…

Your comment seems to be a slightly clumsy way of saying "I don't know much about this area of software design, please tell me more", so I'll respond to it in that spirit. Firstly my terse hand wavey statement was at least clear on the fact that it was talking about communication between services. It said nothing about web browsers or any other human interface. But since this relatively narrow context is clearly the one which you can most relate to, I'll stay within it. If you think about it, the HTTP request/response cycle is actually an illusion of synchronous communication on top of a transport mechanism which is fundamentally asynchronous. It's an abstraction layer which shields you from the complexities of asynchronous communication. Broadly speaking this is known as the [request/response](https://en.wikipedia.org/wiki/Request%E2%80%93response) pattern. HTTP by no means the only implementation of the request/response pattern. Even in asychronous messaging systems it's not unusual to see it implemented - https://www.enterpriseintegrationpatterns.com/patterns/messaging/Reque… But request/reply is just one of the patterns which exists in the realm of asynchronous communication. There are other, much more common patterns such as send/receive and publish/subscribe. Exploring these patterns opens up whole new approaches to software design. I wholeheartedly recommend the book [Enterprise Integration Patterns](https://www.enterpriseintegrationpatterns.com/) by…

Yeah, for a lot of systems it really is more trouble than it's worth - I remember enjoying this article when it came out: https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.h…

This is indeed really a nice book. The accompanying website https://www.enterpriseintegrationpatterns.com is very useful as well.

This reminded me of an old classic, “Starbucks does not use two-phase commit”: https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.h… Previous discussion of it on HN (2010): https://news.ycombinator.com/item?id=1554126

A classic: Starbucks does not use two-phase commit https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.h…

Camel was libraries that codified the the Enterprise Integration Patterns (https://www.enterpriseintegrationpatterns.com/ - channels, messages, service activators, etc...) that allowed you to code against the abstractions rather than the underlying implementations (JMS w/ topics & queues, AMQP w/ exchanges, etc..) Spring offered a competing set of libs with Spring Integration. Those we mostly just libraries, though, so you have something running under them for distribution (like a message broker or service bus or, these days, a cloud managed service).

I don't know about correlating output with commands, but if you're looking to correlate output with input, one option is to stick an ID on every message, and, for messages that are created in response to other messages, also list which one(s) it's responding to. I would say that loosening requirements is also a reasonable option. You can't assume that anything downstream will be up, or healthy, or whatever. On a system that's large enough to benefit from a message bus, you have to assume that failures are the exception and not the norm. And trying to get a system that acts like that is the case is likely to be more expensive than it's worth. For a decent blog post that touches on the subject, see "Starbucks Does Not Use Two-Phase Commit"[1]. [1]: https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.h…

The niche is often called ESB or Enterprise Service Bus but I think a much better description is Integration Service, i.e. something that implements http://www.enterpriseintegrationpatterns.com/ A good website for info and books about this area. See also http://stackoverflow.com/questions/8845186/what-exactly-is-apache-came… which has some useful links.

I've been saying it for a while. I talked to the developers of this product about a decade ago: https://www.kuali.org/kfs and that was the thinking behind how that system was built. Also this is the paradigm behind http://www.enterpriseintegrationpatterns.com/ and it must predate the computer.

It probably doesn't help that books on the subject have desperately unhip titles like "Enterprise Integration Patterns". The introduction to that book is actually a great essay on when and how to use asynchronous messaging, and it is online here: http://www.enterpriseintegrationpatterns.com/Introduction.html

It is a pity that so much effort has been put on writing/designing this booklet. Software Architecture is very hard to 'compress' into a booklet this size. And the concepts explained seem all mixed up. Also, I never understood the difference between Software Architect and Solution Architect, I think they are just marketing tags to specify a bit more what you can do as an architect (install/customize a turn-key solution, networking architecture, software development guidance from scratch, etc). There is so much noise in the space of Software Architecture. And I think is something natural: building software is not architecture, nor engineering, nor mathematics... still it is all that at the same time. It also has strong social, linguistic and design components. Maybe it is just too new a discipline to define it clearly. Personally I find these resources more convincing than the booklet or the references mentioned inside it: For the technical/organizational (Dev teams) part Architecture without Architects: https://www.youtube.com/watch?v=qVyt3qQ_7TA Clean Architecture: https://www.youtube.com/watch?v=Nsjsiz2A9mg https://www.amazon.com/Things-Architect-Knows-About-Transformation/dp/… https://www.enterpriseintegrationpatterns.com For the Enterprise Architect…

Perhaps due to the title (which is evocative in Seattle) I often think of "Starbucks Does Not Use Two-Phase Commit" [0] [0] http://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.ht… P.S.: I feel like I may have thrown this one out without enough context, so I'm just gonna bloviate a bit in this edit. The "object" referred to in the OP link -- or what I like to vaguely call the "unit of consistency" -- is a single Starbucks employee. They hopefully have an internally-consistent history of what they believe to be true about the universe. (And stay sane during coffee-rush times.) The phrase "eventual consistency" describes the relationship between multiple employees. Individuals can drastically disagree with one-another, but there's a framework for detecting disagreements and resolving the discrepancy in some way, even if that just means agreeing to ignore it and logging it for management to fix. A lot of eventually-consistent systems involve allowing certain kinds of discrepancies to occur, while simultaneously promoting those errors into real business concepts in the domain. Banking and accounting systems are particularly great demonstrations of this, because they started doing it centuries ago when nodes were in cities connected by ink-and-parchments packets on horse-ridden routes.

It’s an EIP, called Claim Check. https://www.enterpriseintegrationpatterns.com/patterns/messaging/Store…

Martin Fowler: https://www.enterpriseintegrationpatterns.com/patterns/messaging/Share…

Possibly related? "Starbucks does not use two phase commit": https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.h…

It is important to carefully look into the functional context where that abstraction is used. If you are looking for example into System Integration, Data Integration, ETL and so on, not using a canonical format from the beginning, will get you into the type of almost exponential grow in mappings between sources and targets. https://www.bmc.com/blogs/canonical-data-model/ https://www.enterpriseintegrationpatterns.com/CanonicalDataModel.html

I was disappointed that the article didn't mention Alistair Cockburn, who came up with this architecture style, which is also sometimes called ports and adapters. I like hexagonal architecture a lot, but I change one aspect. Instead of Interactors I use composable Functors and Pipes. A Functor is an Interactor that takes in one Entity and produces an Entity. A Pipe is software that implements one of the different Router patterns from Enterprise Integration Patterns (https://www.enterpriseintegrationpatterns.com/patterns/messaging/Messa… I guess you'd call what I use a hybrid of the EIP style and the Hexagonal style.

The real business world is not atomic. The idea is to keep track of things that you've done, and do compensating things to roll their effect back if something goes wrong. See the Saga pattern for an example. I also found Gregor Hohpe's blog post [0] titled "Starbucks Does Not Use Two-Phase Commit" very informative. [0] https://www.enterpriseintegrationpatterns.com/ramblings/18_starbucks.h…

https://www.enterpriseintegrationpatterns.com/patterns/messaging/Pipes… https://www.coursera.org/lecture/software-architecture/3-2-7-pipes-and…

What's in a name? Microservices are fine-grained and operate on a single bounded context, where applications typically have coarser-grained interfaces and involve multiple contexts. I suppose if your pet insurance system were only to provide a constrained list of values of pet insurance products, and provide an interface to associate them with a user ID from the HR monolith, then it's a microservice. (Of course, you've pushed off the integration with the _actual_ pet insurance app somewhere else, but that might be OK.) "APIs" are an enterprise integration technique [1]. There's nothing inherently creaky or frameworkey about them. [1] http://www.enterpriseintegrationpatterns.com/patterns/messaging/Compos…

My whole point is that ACID is not the only kind of consistency and it is not required to do "get accurate, up to date information at a human timescale". It is just a nice and convenient programming model, but it comes at a cost of availability, scalability, latency and throughput. You can design systems with no ACID data store and if you do it properly they can be just as accurate and up-to-date when everything works properly, but can also degrade nicely when some components fail. http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks… http://www.enterpriseintegrationpatterns.com/docs/IEEE_Software_Design…

I think this is a slightly more detailed version: http://www.enterpriseintegrationpatterns.com/docs/IEEE_Software_Design…

The workflow/orchestrator pattern is key to keeping services decoupled and simplifying your code. I'm working on a Typescript library to encapsulate all the technical detail in https://github.com/node-ts/bus/tree/master/packages/bus-workflow so that you only worry about message inputs and outputs keeping it easy to see the logical flows of your system. It's Inspired from the EIP pattern (https://www.enterpriseintegrationpatterns.com/patterns/messaging/Proce… and frameworks from other languages that have had this type of thing for years. The author also hints about an alternative, commonly known as "routing slips" (https://www.enterpriseintegrationpatterns.com/patterns/messaging/Routi… where the "next step" is loaded into the message header and gets passed down to the next handler as part of a broader process. This removes the need for a centralised orchestrator, but has a huge downside of being tough to handle failure paths, compensating logic or switching logic.

They make a couple products that work well together. Mule ESB acts as a container for flows, which are declaratively-written processes of canned components that do a specific thing (like make an SQL query, make API calls, or call your custom code), or EIP components [1] that manipulate the execution flow. They give you a visual editor to drag-and-drop build these flows, or you can write them by hand in XML. You deploy them in Mule ESB and they run indefinitely, accepting messages, say, on HTTP ports, or firing at set times like cron-jobs, or the like; the runtime detects most crashes and restarts the particular flow if something goes weird. If this kinda sounds like Erlang/OTP you're not entirely wrong, but, let's not go there... (okay, so the canned components are sometimes quite nice, whereas in vanilla Erlang you're on your own. The value something like Mule brings is more the abstractions and the ecosystem, even if the Java runtime is of no intrinsic help. Also, see RabbitMQ's slides on Erlang [2]. Mule isn't an outward-facing message middleware like RabbitMQ, but behaves a lot like one on the inside. It also gives canned components to interact with them.) A Mule flow is often basically mid-level business logic plumbing; you don't want to put these flows on the public internet without rate limiting, authentication and authorization -- you want to use some API gateway or proxy in front. Mule has one, called Mule API Gateway, and the front-facing portal API Manager. Then…

Saga Pattern. https://blog.couchbase.com/saga-pattern-implement-business-transaction… https://microservices.io/patterns/data/saga.html https://developers.redhat.com/blog/2018/10/01/patterns-for-distributed… https://www.enterpriseintegrationpatterns.com/patterns/conversation/Co… https://en.wikipedia.org/wiki/Compensating_transaction