Cindy Sridharan

Observability is not just about traces. This great article[1] by @copyconstruct describes it in detail, and lists monitoring, alerting/visualization, tracing, and log aggregation/analytics as key components of an observability stack. [1] https://copyconstruct.medium.com/monitoring-and-observability-8417d195…

> Vendors can change schemas on a dime and the schema you're working with is often the schema as you know it at the time. This is partially negated through aggressive dependency management and languages with type systems and vendor provided interfaces. This is why testing in production is so important. Building monitoring and alerting tools, making code observable for all of that to be feasible, and building an incident response culture can all fall under the "testing in production" umbrella to me. Shifting testing left and right has been a big thing of mine for the last few years and I think I'm onto something. Planning your architecture and projects early and thinking about making things easy to test and observe in production, as well as monitoring and alerting after its live. Cindy Sridharan has an amazing treatise on this topic. https://copyconstruct.medium.com/testing-in-production-the-hard-parts-…

Cindy Sridharan often writes well about these topics. https://copyconstruct.medium.com/testing-in-production-the-safe-way-18… For this one I'd start about half way down, under the heading "Shadowing (also known as Dark Traffic Testing or Mirroring)" Unfortunately the terminology is a bit fragmented - shadowing, mirroring, teeing, dark traffic (ick), ad nauseam. Whatever you call it, it's often pretty high overhead to add the infrastructure, unless you're already using some sort of "service mesh" (Envoy/Istio/Caddy/whatever) that supports it. Even then, if you're dealing specifically with a storage system then there can be some thorny issues - idempotent vs. non-idempotent vs. destructive requests, requests which require other objects (files/objects or directories/buckets) to exist or be in specific states, etc. I'm not going to pretend it's easy. If you can do it, though, it can be an incredibly valuable tool. There ain't nothing like the real traffic, baby. ;) My favorite feature, which I alluded to earlier, is that you can shadow traffic from a larger production cluster onto a smaller shadow cluster and give it a serious stress test. All sorts of bugs tend to fall out that way. The one thing you can't really catch, even with a good shadow, is interactions with other services - including things like permissions or quotas. But if those are the only things you have to shake out in true production, you're doing well.

This is quaint and wholesome. I long for the simpler days when I could agree with this, blissful in my naivety of large scale organizations. Nowadays, I accept this reality is largely impossible and you must always draw some boundary. This doesn't mean all your developers should use a shared MySQL because nobody knows how to bootstrap the database- but it means you have to consciously decide where you sit on the continuum. Always expecting to run the whole system on your laptop (or even in a cloud) is also an unreasonable expectation, unless you're Netflix and your revenue per employee is into the millions. The reasons for this are many and complicated, but at a high level the work required to make it happen will cost too much and won't be a priority. I'll quote here from the excellent "Testing Microservices the Sane Way" by Cindy Sridharan: > asking to boot a cloud on a dev machine is equivalent to becoming multi-substrate, supporting more than one cloud provider, but one of them is the worst you’ve ever seen (a single laptop) https://copyconstruct.medium.com/testing-microservices-the-sane-way-9b… "Full stack in a box- a cautionary tale"

This is certainly true for blocking code reviews. I'm interested in exploring the alternative, which is review-after-commit. There's an article describing those here: https://copyconstruct.medium.com/post-commit-reviews-b4cc2163ac7a Code still gets reviewed, but you don't end up with PRs languishing for hours, days or even weeks waiting to get a review from someone.

Some links you may find useful https://copyconstruct.medium.com/small-functions-considered-harmful-91… https://github.com/EnterpriseQualityCoding/FizzBuzzEnterpriseEdition

As others have said already, that's nowhere near enough. In general, I advise my teams to stay away from any Open Source projects that don't have a robust testing approach. Toby Clemson's infodeck on Martin Fowler's site has a lot of information about testing microservices: https://martinfowler.com/articles/microservice-testing/ "Testing in production" doesn't necessarily mean "Test only in production". Cindy Sridharan has a nice article about the nuances involved: https://copyconstruct.medium.com/testing-in-production-the-safe-way-18…

Check out https://copyconstruct.medium.com/testing-in-production-the-safe-way-18… Basically, preprod and any not-prod environments cannot ultimately perfectly reproduce prod, and in a sufficiently complex system, the failure states cannot be predicted in a replicated environment. Instead, using strategies like rolling deployments, you can test on the actual environment you want to validate. If set up properly, it doesn't really risk breaking prod.