Buoyant Blog

At my day job (https://buoyant.io/), we're using it to write a reverse proxy/load balancer --- one of the use-cases where you really, absolutely do need asynchronous concurrency. :)

Not to be confused with the other Buoyant [1], the company behind the linkerd service mesh. Not that I'm really into service meshes, but I happened to be doing some reading about the Tokio async runtime for Rust yesterday, and a lot of the work on that has been funded by that Buoyant, which is why that company was on my mind. [1]: https://buoyant.io/

The "service mesh" space is starting to take off now with the rise of Kubernetes and containers. Bouyant made LinkerD and now Conduit: https://buoyant.io/ Lyft created Envoy: https://www.envoyproxy.io Istio uses Envoy for more functionality in K8S: https://istio.io

there's quite a bit here - https://monzo.com/blog/2016/09/19/building-a-modern-bank-backend/ and https://blog.buoyant.io/2016/10/14/a-service-mesh-for-kubernetes-part-… But you should ping linkerd on twitter. They are really responsive.

See also: Twitter's Finagle [1] for the JVM, and Bouyant [2] providing Finagle-as-a-microservice on localhost for language independence. 1: https://twitter.github.io/finagle/ 2: https://buoyant.io

Totally unrelated: the effect/animation when you hover over the avatars (Safari) at https://buoyant.io/#team is really weird. Not sure if it's intentional.

This is a good starting point to your question, from William Morgan who's attributed with coining the term service mesh: https://buoyant.io/service-mesh-manifesto/. It's a fair question. Service meshes are a relatively recent development, and there aren't many papers on them, despite rapid adoption in industry (e.g. many large systems use a service mesh, and AWS App Mesh is in general release as of last year). This is a decent survey paper: https://ieeexplore.ieee.org/document/8705911. Service meshes are intended to address some of the operational complexity of running microservices. To take Morgan's definition: "A service mesh is a dedicated infrastructure layer for handling service-to-service communication. It’s responsible for the reliable delivery of requests through the complex topology of services that comprise a modern, cloud native application." To answer your question briefly, a service mesh is not a completely brand new thing, the pattern seems a natural improvement to having a set of SDKs (like Twitter's Finagle) and other things tying an SOA together. Consistency in an SOA is pretty valuable, and separating infrastructure logic (like retries, service discovery) from application logic is pretty nice too. Whether the improvement has been significant, I'd recommend searching for "service mesh" and you'll find some talks describing use cases. What I have found tricky is finding critical analysis of service meshes beyond "do we need this" (i.e. what are service…

Sounds like similar to the pitch for https://buoyant.io/2019/11/18/announcing-dive/, which is in closed beta and I haven’t tested yet. Any other competitors / SRE dashboard/metric aggregators that folks like? I think this is a level of abstraction above raw datadog/grafana dashboards. Gitlab seems to be moving into this space a little with their deploy boards and k8s monitoring tools.

Is this helpful? https://buoyant.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-o…

Totally understand why they did it but couldn't help chuckling since 6 months ago in this blogpost: https://blog.buoyant.io/2017/12/05/introducing-conduit/ they literally said: "Conduit is not Linkerd 2.0." ;-)

This explanation was helpful to me: https://buoyant.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-o…

Had to google this myself, here's a link to their description of a service mesh. https://buoyant.io/2017/04/25/whats-a-service-mesh-and-why-do-i-need-o…

linkerd contains some advanced features, that are not easily solvable with DNS. By sending specific headers (and having your application forward them) one can have per request routing[1] and/or tracing[2]. [1] https://blog.buoyant.io/2016/11/04/a-service-mesh-for-kubernetes-part-… [2] https://blog.buoyant.io/2017/03/14/a-service-mesh-for-kubernetes-part-…

I think that Linkerd is totally awesome and that it's definitely not receiving the attention that it deserves. One may think that Linkerd only fits in large projects but having that layer that manages the communication between my apps/microservice represents a huge boost forward for any project regardless its size. With Linkerd you're relying on the expertise of a team with vast knowledge on the field. For those of you worried about the memory footprint of JVM, Linkerd can perfectly run in the 32-bit JVM (Docker image buoyantio/linkerd:1.0.0-32b) and with only 256mb the results are impressive. More here: https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniques-for-mic…

Read William's post entitled "What's a service mesh?": https://blog.buoyant.io/2017/04/25/whats-a-service-mesh-and-why-do-i-n…

"The concept of the service mesh as a separate layer is tied to the rise of the cloud native application. In the cloud native model, a single application might consist of hundreds of services; each service might have thousands of instances; and each of those instances might be in a constantly-changing state as they are dynamically scheduled an orchestrator like Kubernetes. Not only is service communication in this world incredibly complex, it’s a pervasive and fundamental part of runtime behavior. Managing it is vital to ensuring end-to-end performance and reliability." https://blog.buoyant.io/2017/04/25/whats-a-service-mesh-and-why-do-i-n…

> Why not DNS round-robin? Round-robin may be sufficient for your needs. However, the load balancing techniques that linkerd employs solved real production issues for us. These techniques allowed us to be paged less frequently for minor errors and we could more reliably make changes to the system without causing user-facing issues. See https://blog.buoyant.io/2016/03/16/beyond-round-robin-load-balancing-f…

What kind of resource usage do you see with linkerd? It looks good, but I'm concerned about the resource overhead of a JVM based proxy on every node / pod. edit: Ok answered myself - found https://blog.buoyant.io/2016/06/17/small-memory-jvm-techniques-for-mic… So with some work it can be reduced from 500Mi->100Mi per instance. It'd be interesting to see the kind of CPU time it uses under load, though.

I like a good rant as well as the next person, but had some issues with this one. > What if we have a scenario where a third party authenticates the client (by verifying that they have a valid token issued by their ID provider) and then uses that to issue their own token that's much longer lived? He might as well be complaining about car keys. Yes, if I leave my car key in the door of my car, bad things will happen. If you issue a long lived bearer token, bad things will happen. Bearer tokens are credentials. Short lived and limited, but still credentials. You have to take care of credentials! Use short lived bearer tokens and refresh tokens. Threat model the ramifications of someone stealing the token. If you really higher assurances, look into client bound bearer tokens (DPoP and MTLS in the OAuth world.) For a common use case (I work for an auth provider, more details in my bio) of browsers or native applications integrating with other applications and APIs, what are other options for verifying a client is authorized to access a resource: * sessions, probably with cookies. Well known, solid technology. Requires you to either have sticky sessions (so every client request goes to the same server) or a common session store (redis, etc). * client certificates. If you control all deployment (intranet, employee laptops, etc) can be an option. https://buoyant.io/mtls-guide/ is a good resource if you are doing service to service communication. * API keys. How do you like…

That's what this does and more. What you described are techniques. Linkered is an implementation. From https://blog.buoyant.io/2017/04/25/whats-a-service-mesh-and-why-do-i-n… Linkerd applies dynamic routing rules to determine which service the requester intended. Should the request be routed to a service in production or in staging? To a service in a local datacenter or one in the cloud? To the most recent version of a service that’s being tested or to an older one that’s been vetted in production? All of these routing rules are dynamically configurable, and can be applied both globally and for arbitrary slices of traffic. Having found the correct destination, Linkerd retrieves the corresponding pool of instances from the relevant service discovery endpoint, of which there may be several. If this information diverges from what Linkerd has observed in practice, Linkerd makes a decision about which source of information to trust. Linkerd chooses the instance most likely to return a fast response based on a variety of factors, including its observed latency for recent requests. Linkerd attempts to send the request to the instance, recording the latency and response type of the result. If the instance is down, unresponsive, or fails to process the request, Linkerd retries the request on another instance (but only if it knows the request is idempotent). If an instance is consistently returning errors, Linkerd evicts it from the load balancing pool, to be periodically…