Brave New Geek

I'm just going to leave a direct quote from Rob Pike: """ The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt. """ I read that as Rob Pike saying he wrote go for idiots google hires to write. https://bravenewgeek.com/go-is-unapologetically-flawed-heres-why-we-us…

A follow up from a reader on this article: http://bravenewgeek.com/smart-endpoints-dumb-pipes/

I'm not sure what you are suggesting. There is no getting around "at most/at least once." You can shift the posts, some, but at some point that is your choice. This is a good read on the problem: http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

> which is an abuse of DRY because it should only apply to concepts, not code As a 'copyista', I would change this to "DRY should only apply to production code, not tests." It wasn't clear from your reply if you agree or disagree. A couple of great recent links about this IMO under-discussed topic: http://www.sandimetz.com/blog/2016/1/20/the-wrong-abstraction http://programmingisterrible.com/post/139222674273/write-code-that-is-… http://bravenewgeek.com/abstraction-considered-harmful

It's worth noting that exactly-once delivery is impossible: http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

> Guaranteed delivery of a message to exactly one recipient within a messages visibility timeout. I'll just leave this here for you: http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/ Welcome to message queues. Where the network is both your best friend and enemy.

Previous discussion on bitly blog / reddit with some comparisons to zmq: http://www.reddit.com/r/programming/comments/117dsj/nsq_realtime_distr… And also a great article that compares / contrasts NSQ along with a lot of other alternatives: http://www.bravenewgeek.com/dissecting-message-queues/

Thanks for sharing, very interesting article. Your previous post is great (and relevant) too: "Dissecting Message Queues". http://www.bravenewgeek.com/dissecting-message-queues/

This is probably the most detailed analysis of retry techniques that I've seen. I really appreciated the circuit breaker and deadline propagation sections. But this is why I've pretty much abandoned all connection-oriented logic in favor of declarative programming: https://en.wikipedia.org/wiki/Declarative_programming Loosely, that means that instead of thinking of communication as client-server or peer-to-peer remote procedure calls (RPC), I think of it as state transfer. Specifically, I've moved aware from REST towards things like Firebase that encapsulate retry logic. Under this model, failure is never indicated, apps just hang until communication is reestablished. I actually think that apps can never really achieve 100% reliability, because there's no way to ever guarantee communication: https://bravenewgeek.com/you-cannot-have-exactly-once-delivery/ https://en.wikipedia.org/wiki/Byzantine_fault https://en.wikipedia.org/wiki/Two_Generals%27_Problem Although deadline propagation neatly models the human experience of feeling connected to the internet or not. Also this is why I think that microservices without declarative programming are an evolutionary dead end. So I recommend against starting any new work with them in this era of imperative programming where so many developer hours are lost to managing mutable state. A better way is to use append-only databases like CouchDB which work similarly to Redux to provide the last-known state as the reduction of all…

> Do you have a citation for that? I do indeed have a citation for that by Rob Pike - and no matter how downvoted I am, it stays a fact: > The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt. (https://bravenewgeek.com/go-is-unapologetically-flawed-heres-why-we-us… and also: > Programmers working at Google are early in their careers and are most familiar with procedural languages, particularly from the C family. The need to get programmers productive quickly in a new language means that the language cannot be too radical. (https://go.dev/talks/2012/splash.article) > As far as I remember that’s as most a second order effect -- they wanted a language that felt as clean and straightforward as C, to benefit everybody including experienced developers, not primarily inexperienced ones. No, it's not a second order effect. Or rather: both what you say (a language "as clean and straightforward as C") as well as a language for devs early in their career (i.e. inexperienced) are both second order effects from the underlying reason that the language is targeted at developers at google.

I'm totally in agreement with this. Both processes should also be idempotent (you should be able to pull multiple times without side effects, and push and pull should be able to happen at the same time without side effects). When everything is working well, the 'push' does all the work, and though the 'pull' runs every few minutes/seconds/whatever it never has anything to do. This same thing applies to time-based events: your system should not assume that the process is always running, so if something needs to happen at exactly 9:00am (and it's not okay to just skip it if missed), it should be able to run anytime later with the same outcome as if it ran at 9:00am. When I was first getting into this, it helped me to understand that events must follow one of the semantics: at-most-once, at-least-once and exactly-once -- with the trick that exactly-once is not strictly possible [1]. There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery -Mathias Verraes [2] [1] http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/ [2] https://twitter.com/mathiasverraes/status/632260618599403520

(project author) I appreciate all the work that Tyler has put into his series of blog posts on messaging systems (and NSQ bug reports!) but I think this particular article is one that misses the mark [1]. While raw performance is important, I think comparing the guarantees and related operational and development semantics are far more interesting and useful. These systems vary a great deal in the interfaces and functionality they expose to users building on top of them and operators dealing with them when they're deployed in production (and inevitably break!). This is ultimately what matters most. P.S. single node (localhost) performance is essentially useless. Most of these tools are designed to be deployed as the backbone of large distributed systems (NSQ in particular), so scalability and performance in that context is a better indication of real-world performance. [1] He's since posted this follow-up that outlines many of the same things http://www.bravenewgeek.com/benchmark-responsibly/

That reminds me of this talk[0] by Gil Tene called "How NOT to Measure Latency" at the Strangeloop conference in 2015 (or read this blog post[1] that contains the most important points). [0] https://www.youtube.com/watch?v=lJ8ydIuPFeU [1] https://bravenewgeek.com/everything-you-know-about-latency-is-wrong/

You can get exactly once processing, but not exactly once delivery. https://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

There _is_ control over the source code from both sides, yes. > Not everything can be made idempotent. Can you expand on this, please? I'd appreciate it. If you mean to say that true idempotence can never be achieved, then you're right [1]. If you mean to say that the framework doesn't allow for the same level of guarantees that other widely used distributed system tools (queues, pub-sub, idempotency key headers) allow, then I'd challenge that notion. [1] https://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

This article brought to mind another article which I personally found a bit more profound: https://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

It looks like Liftbridge occupies a nice little niche between full-blown, durable Kafka and lightweight, ephemeral NATS. Additional reading: https://bravenewgeek.com/introducing-liftbridge-lightweight-fault-tole…

Nats streaming isn't just a persistence layer to NATS. It's an entirely different system that basically acts as a client to NATS and then records messages it sees. Basically think of how you would design a persistent queue on top of the ephemeral NATS pub/sub and that's what NATS streaming is. Here's a good post (and series) about distributed logs and NATS design issues: https://bravenewgeek.com/building-a-distributed-log-from-scratch-part-…

If you’re message consumer isn’t idempotent then no MQ can help you. Exactly once delivery is impossible other than with at least once delivery and an idempotent consumer. https://bravenewgeek.com/tag/amazon-sqs/

> give me a queue with FIFO and exactly once delivery https://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

I think this article[0] is very well written and clearly explains the problems with provisioning resources in multiple clouds. Yes, there are tools that allow you to do it, however you give up most of the reasons for going into the cloud. Switching between cloud providers is much easier than going from on-prem to the cloud. So, once your systems are "cloud" ready you should be able to avoid vendor lock in naturally. [0] https://bravenewgeek.com/multi-cloud-is-a-trap/

> Persistent connections with exactly-once messaging. Disconnection is just seen as long latency. Bullshit. https://groups.csail.mit.edu/tds/papers/Lynch/jacm85.pdf Unless you and Yarvin really think there's a valid answer to the 2 Generals problem... Im sure there's a Fields medal in there if you do. "Every major message queue in existence which provides any guarantees will market itself as at-least-once delivery. If it claims exactly-once, it’s because they are lying to your face in hopes that you will buy it or they themselves do not understand distributed systems. Either way, it’s not a good indicator." ~ http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

> Whenever the plumbing does not guarantees EOIO but the business demands it, it gets pushed into the app layer where TCP (retries and acks) is reimplemented, to various success levels. Interesting! You're saying the exact opposite of what Tyler Treat is saying: > Even with smart middleware, problems still leak out and you have to handle them at the edge—you’re now being taxed twice. This is essentially the end-to-end argument. Push responsibility to the edges, smart endpoints, dumb pipes, etc. It’s the idea that if you need business-level guarantees, build them into the business layer because the infrastructure doesn’t care about them. (source: http://bravenewgeek.com/smart-endpoints-dumb-pipes/) What do you think of his argument?

A million messages per minute ? How's that a problem? Please see http://bravenewgeek.com/dissecting-message-queues/ Almost all of the systems tested hit 17k / s (aside from NSQ) -- on a laptop. NATS (gnatsd) hits 0.2M per second, that's a magnitude above what you need and that's a brokered system, if your task fits with a brokerless one then nanomsg is yet another magnitude. I do not particularly see where the problem really is?

Note that NATS is currently pub/sub, which is a "if a tree falls in the forest" situation. Messages don't go anywhere if nobody is subscribing. So it's awesome for realtime firehose-type use cases where a websocket client connects, receives messages (every client gets all the messages, although NATS also supports load-balanced fanout) for a while, then eventually disconnects. NATS is ridiculously fast [1], too. There's an add-on currently in beta, NATS Streaming [1], which [2] has durability, acking/redelivery and replay, so covers most of what you get from both RabbitMQ and Kafka. It looks very promising. [1] http://bravenewgeek.com/tag/nats/ [2] https://nats.io/documentation/streaming/nats-streaming-intro/

Good point. I often heard that it makes no sense to NOT deal with network partitions and CAP as an "illusion of choice"[1] [1] http://bravenewgeek.com/cap-and-the-illusion-of-choice/

You almost never want to use average as your metric when dealing with time. For time, the 95 or 99 percentile latency is going to be measure of how many requests out of a hundred a less than your metric. Esp when the thing you are measuring is a piece of larger system, which is the crux, because of how a resource in turn requests k-more resources. Each request has an equal chance of being over the latency percentile. Even at 10 requests, only 90% of sessions will be under the 99% latency. See also, Gil Tene https://www.youtube.com/watch?v=lJ8ydIuPFeU A nice overview, http://bravenewgeek.com/everything-you-know-about-latency-is-wrong/ See also, http://highscalability.com/blog/2015/10/5/your-load-generator-is-proba…

> High throughput and low latency stream processing with exactly-once guarantees. Precisely how is this possible in a real-world distributed system? At-least-once and at-most-once are possible with different distribution methods, but "exactly-once" is, well, I have yet to see it actually implemented (even theoretically, see the Two Generals thought experiment). Even their checkpointing algorithm has an assumption that the underlying transport is "exactly once" and "FIFO ordered". If I had to guess, I'd say that what they're really guaranteeing is "at most once" message delivery. Unfortunately, that has a very different meaning from "exactly once". http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/

If they required of you to implement once-and-only-once reliable delivery of messages over a network, then I would argue they don't really understand the problem themselves. This blog post does a pretty good job of describing the problem: http://bravenewgeek.com/you-cannot-have-exactly-once-delivery/ I would not feel bad about missing this question. Distributed systems are hard!

> In a reasonably well-behaved datacenter environment, the timing assumptions will be satisfied most of the time – this is known as a partially synchronous system [12]. But is that good enough? Please consider this answer [0] (which I personally understood as - YES, real systems are always partial sync): "Asynchronous systems have no fixed upper bounds. In practice, systems tend to exhibit partial synchrony, which is described as one of two models by Dwork and Lynch in Consensus in the Presence of Partial Synchrony. [1]" [0] http://bravenewgeek.com/category/distributed-systems-2/ [1] http://groups.csail.mit.edu/tds/papers/Lynch/jacm88.pdf