Honeycomb Blog

These are such great comments, thanks for sharing your insights. For folks looking for other options, I'd also mention https://honeycomb.io, perhaps the most promising newcomer in this space. It's essentially Facebook's Scuba for the rest of us.

Great article! I always love hearing Stripe talking about their internals. I've been using this practice and I agree that it's incredibly useful. I think because people tend to think in terms of "logs", they end up overlooking the much more useful construct of "canonical logs". Many fine-grained logs themselves are almost always less useful than the fewer fully-described canonical logs. Other observability tools often call these "events" instead of "logs" for that reason. There's a tool call Honeycomb [1] that gives you exactly what this article's talking about in a really nicely designed package out of the box. And since it handles all of the ingestion and visualization, you don't have to worry about setting up Kafka, or the performance of logplexes, or teaching everyone SQL, or how to get nice graphs. I was a little skeptical at first, but after using it for over a year now I'm completely converted. If you record fully-described "events" for each request, and you use sub-spans for the smaller segments of requests, you also get a waterfall-style trace visualization. Which eliminates the last need for fine-grained logs completely. If this article seems interesting to you, I'd highly, highly recommend Honeycomb. (Completely unaffiliated, I just think it's a great product.) [1]: https://www.honeycomb.io/

There's a rich ecosystem around distributed tracing via OpenTelemetry! Honeycomb was one of the pioneers of this, and they contributed significantly to OTel which largely standardizes this approach. OP's approach is a bit different in that errors bubble up synchronously rather than asynchronously, which could be great for certain use cases! But the OTel approach accomplishes much of the same - as long as systems log their traces with propagated parent IDs, a dedicated external telemetry sink can reconstruct the hierarchy. https://www.honeycomb.io/platform/distributed-tracing https://opentelemetry.io/docs/concepts/signals/traces/ https://opentelemetry.io/docs/concepts/context-propagation/

As proven by ongoing research in things like Linear Haskell, one doesn't need to throw GC productiviy away, to get deterministic resource management. Best part of Rust is catering to a crowd that not even at point gun will consider touching anything that might resemble having any kind of automatic resource management. Hence why I consider articles like "How We Saved 70% of CPU and 60% of Memory in Refinery’s Go Code, No Rust Required" [0] interesting to read and make awareness, even if Go isn't one of the languages I happen to be entusiastic about. [0] - https://www.honeycomb.io/blog/how-we-saved-70-cpu-60-memory-refinery

Not necessarily? It's been the basis of one of the major ways people would query their data since 2023 on a product I worked on: https://www.honeycomb.io/blog/introducing-query-assistant The difference is this feature explicitly isn't designed to do a whole lot, which is still the best way to build most LLM-based products and sandwich it between non-LLM stuff.

I haven't used anything else, but I'll gladly shill for https://honeycomb.io.

All you need are logs. Also, https://honeycomb.io looks pretty dope.

Graviton3 on AWS is better for most applications on a cost/performance basis, and in some cases also on straight latency. Honeycomb has some great blogs about it: https://www.honeycomb.io/blog/present-future-arm-aws-graviton-honeycom…

> Why? …because seniority is about responsibility, and if all you do is write good code, you’re not accepting the responsibility for anyones effort but your own. Exactly. Ownership of a certain scope. This came out not long ago, and I love the graphic. https://www.honeycomb.io/blog/engineering-levels-at-honeycomb/

> Today I would like to tell you about the AWS Nitro SSD. A bit light on technical details but very fun, very exciting. Kind of sad that such amazing work is no longer quite so public, is no longer something that say Intel is going to talk up in endless details with a product launch. A huge amount of the work & innovation here is extremely specific, extremely private- all this Elastic Fabric Adapter related stuff is advanced systems engineering, close integration of systems, that's Amazon's & Amazon's alone. Anyhow. This article pairs very well with the "Scaling Kafka at Honeycomb"[1], which I found to be a delightful read on adapting & evolving a big huge workload to ever-improving AWS hardware. [1] https://www.honeycomb.io/blog/scaling-kafka-observability-pipelines/ https://news.ycombinator.com/item?id=29396319 (38 minutes ago, 13 points)

The author is asking for, and actually built, https://honeycomb.io.

Performance per dollar is pretty great on AWS. For an example: https://www.honeycomb.io/blog/observations-on-arm64-awss-amazon-ec2-m6…

Honeycomb https://honeycomb.io/ is laser focused on this stuff. They built their own datastore (similar to Druid but schemaless) so they could create the experience they were aiming for. They talk a lot about collaborative troubleshooting, and the user interface reflects that. It's actually fun (?!) to drill down from heatmaps to individual events with Honeycomb's little comparison charts lighting the way.

You can go the https://www.honeycomb.io/ way and make structured logs your metrics. It will cost you a lot in storage, but simplifies a lot. Just throw properly structured logs into storage as long as you query them efficiently (which honeycomb provides)

Segment's blog posts on cost optimisation have plenty of detail and tips on this topic: https://segment.com/blog/the-million-dollar-eng-problem/ https://segment.com/blog/spotting-a-million-dollars-in-your-aws-accoun… https://segment.com/blog/the-10m-engineering-problem/ Similarly this Honeycomb writeup is also excellent: https://www.honeycomb.io/blog/treading-in-haunted-graveyards/ By the sounds of it, you need to take drastic action. It sounds like you will not be able to just optimise your AWS spend to get more runway, though you should definitely do some bill optimisation. You will need to optimise your product itself and maybe even getting rid of unprofitable customers. If you are not sure exactly who or what is driving the AWS cost, take a look at Honeycomb to get the ability to dive deep into what is eating up resources.

AWS has released Graviton2-based EC2 instances[0] with excellent performance[1]. It seems like a lot of workloads can be moved to ARM-based VMs with little fuss. [0]: https://aws.amazon.com/blogs/aws/new-m6g-ec2-instances-powered-by-arm-… [1]: https://www.honeycomb.io/blog/observations-on-arm64-awss-amazon-ec2-m6…

That's the "monitor from the customer's point of view" approach the OP alludes to. If you use tools like Honeycomb [1] that can easily and routinely answer questions like "show me the 95th percentile latencies for each of the 10 customers experiencing the worst latencies", then situations like you're describing are a lot easier to discover. [1] https://honeycomb.io. Disclaimer: I used to work for them.

Integrate with https://www.honeycomb.io

It should be noted that this is a very oblique ad for http://honeycomb.io. That in no way impugns the content of the post, and in fact, it's given the content of the post that I feel compelled to point out that, ultimately, this is an ad. Because what is sales and advertising, anyway? It's just a way to get you to buy a product, and you can't do that if you've never even heard about the product. I'm not currently in the market for an observability solution (something something splunk) but if I were, I've now heard of them. The question though, is why does money ruin everything? The naive questions of an open source zealot to a proprietary software salesperson are one thing, but since we all need money to live, why does money being part of the equation (eg if someone was getting paid to post here) ruin things? Would a "donate to open source" button be more successful if the donate text is "buy me a coffee", "buy me a beer", or "upgrade my beer from Coors light"? https://xkcd.com/2347/ was and is true, and if we don't figure out a way to change that, I don't really see a future for open source.

I read though half of the article thinking "This is like Honeycomb[1] way of thinking" without realizing it was written by Charity Majors[2]. So, keep in mind that the article is biased towards their business - they are in the Observability realm - and the long-term goal is making you think that you have to achieve super-deployment speed, and that involves deploying to prod often, and have a good system (Honeycomb) to know at any moment what's going on in production with real data. And if it breaks, you just have to quickly rollback. Aaaand, I buy that. I mean, I think she has totally a point on this: having a safe and quick way to production that leads you to be confident on rolling out things quickly because you have a safe and fast path back, it's a good thing for any software engineering team/organization. It's not easy to go there and you still obviously need tests while developing/integrating, you need canary tests, you need to program in a defensive manner etc etc, but not being scared of sending code to prod and "test" how it will really hold the production pressure it something I'm personally trying to achieve at $DAYJOB. [1] https://www.honeycomb.io/ [2] https://nitter.net/mipsytipsy

Hear hear. At CoreOS I set up a number of dashboards specifically to socialize the normative behavior of our systems. Think of it as the difference between the person who just drives their car versus the person who knows what feels and sounds "normal". The latter can tell when they need an oil change because of different vibrations in the engine and how the car sounds pulling up to a stop light (because of the engine sounds being reflected back into the window by parked cars). Big surprise, it had the desired effect. I'm especially with you on the notion of disparate metrics. While correlation is not causation, it's still a useful diagnostic tool. Let's say someone in marketing walks a dashboard and and sees the following: 1) a new version has been pushed 2) customer tickets are up 20% over the number they're used to seeing Does that mean that the new version caused the tickets? No. Will that allow them to ask? Absolutely. Will that urge behavior to reach out to support and release management to see if there's an interesting story to share internally (or with the world)? Hopefully. You hit the nail on the head by calling out the absolutist / "I'm the authority on this matter" / "there is a single correct perspective" tone. Your lack of shock at the author selling something can be remedied. They have a dog in this fight: https://www.honeycomb.io/teammember/charity-majors/

The real takeaway here: Honeycomb is a great tool for log and trace analysis, and by using Honeycomb's minimal data model [1] for traces you can expand tracing throughout much more of your stack. Now Honeycomb itself makes you pay by the event so Slack's internal buffer and collection system really lets them use it effectively for just the data they need. The low adoption of tracing is also due to the high cost of application level integration, which they are deferring by using the existing tracing systems service layer integration. So if you come at this as a Honeycomb user Slack has not really re-invented any wheel so much built their own backend to funnel data to Honeycomb and use it's excellent trace viewing and light weight analytics UI [0]. In fact API and model which this article goes to great lengths to explain and justify is basically the Honeycomb tracing API and data model [1]. Which itself is an evolution of the system Facebook uses [2]. It is very cool that they have plugged the gap in Honeycomb which is the lack of historical analysis capability by also funneling all data into Presto. I would love to see the system they use for keeping field/tag names under control. Honeycomb really strains under large numbers of field names. 0 - https://www.honeycomb.io/trace/ 1 - https://docs.honeycomb.io/getting-data-in/tracing/send-trace-data/#man… 2 -…

How is it different from https://www.honeycomb.io/?

> I find it a huge step backwards. As far as I know, there's no full text search and the types of reports and aggregation you can do are extremely primitive. Oof. Honeycomb is for fast, realtime analytics: starting with a high-level question in your mind ("why did our throughput drop by 50%?") and rapidly iterating on a hypothesis (examples in [0]). ELK can... be used for that, but is optimized for another (as you said, full-text search and generating static reports). Being able to flip from a funny-looking graph directly into "raw data" mode is intended to be a bonus in Honeycomb, not the primary way you interact with your data. While we believe that fulltext search has its place, beyond a certain point (most production systems, these days), sifting through log lines is a brute-force method of answering questions about your systems — especially if you're not sure what the proverbial needle you're searching for looks like. [1] (But mherdeg's answer is great, go back and read theirs while you're here :)) 0: https://www.honeycomb.io/blog/troubleshooting-in-honeycomb-choose-your… 1: https://www.honeycomb.io/blog/the-true-cost-of-search-first-problem-so…

By creating events that contain both the duration of the request and whether it succeeded, you can create graphs that show you the detail you need. Unless you include those data together at the beginning, it will be impossible to tease them apart later on. Combining them into one graph will likely conceal the difference in the two cases, as you describe, unless you feed them in to a system that an natively tease them apart as easily as show them together (such as http://honeycomb.io). So it seems like the disagreement is more about visualization than collection (the section of the blog in which that quote appears). The originally quoted advice, to show "the duration it took to serve a response to a request, also labelled by successes or errors" remains good advice, so long as the visualization of that data makes clear the separation. I absolutely agree that careful consideration is required when choosing what to put on dashboards to avoid confusion. That seems to be a separate issue. (bias alert - I work on Honeycomb, and care deeply about collecting data in a way that lets you pull away the irrelevant data to illuminate the real problems.)

Yeah, I was just commenting on a LinkedIn post[0] (don't hate the player, hate the game :) ). In it, someone talked about the difference between creating software and owning it. Creating software with AI is super easy--plan, prompt, test, go, go, go! Owning software means you're responsible for maintaining it over time, fixing edge cases, operating it well, and more. If you're building a one-off custom webapp to meet your needs, create away. If you're writing software for a business to run on, you're owning it. My fav article on this topic is this post[1] on durable vs disposable software. 0: https://www.linkedin.com/feed/update/urn:li:activity:74821232125043302… 1: https://www.honeycomb.io/blog/disposable-code-is-here-to-stay

You should almost never stop testing in prod. https://www.honeycomb.io/blog/testing-in-production

This is nice. At work, we use Datadog for logging, and I have previously used CloudWatch, Splunk, and Honeycomb. Among these, only Honeycomb makes implementing canonical log lines [1] easier. I want arbitrarily wide, structured logs [2] without paying exorbitant costs for cardinality. Our Datadog costs are outrageous, and it seems like no one cares at this point. Pydantic Logfire is also doing some good work in Python-specific environments. I use both Python and Go, but Logfire wasn’t as ergonomic in Go. [1]: https://stripe.com/blog/canonical-log-lines [2]: https://www.honeycomb.io/blog/structured-events-basis-observability

I got a lot of value from this post related to growth for ICs: https://www.honeycomb.io/blog/engineering-levels-at-honeycomb

> from the personal preferences of the engineers https://www.honeycomb.io/pricing https://www.datadoghq.com/pricing/ Am I wrong to say... having 2 is "expensive"? Maybe not if 50% of your stuff is going to Honeycomb and 50% going to DataDog. Could you save money/complexity (less places to look for things) having just DataDog or just Honeycomb?