Lightstep Blog

I define a special category of distributed system performance bugs. The fact that proper distributed tracing is not standard yet (especially outside of cloud environments) is a real pain here. The google white paper on Dapper [0] and all its implementations [1][2][3] in one form or another are appearing but usually target the Web environments. That means that for my legacy stack with multiple nodes connected via networks of different kinds I have to profile on multiple systems, aggregate different log types manually, think of time syncs (not everything logs utc..) and so on. To be honest we just reached the point where we try to get a European subsidiary project rolling where we address this problem. The time lost to trace down unexplainable time gaps between first displayed result in app and request are weeks, and in multiple occasions months. Just because we have no easy way to profile our distributed ecosystem. After identifying who is to blame: network, db nodes, processing nodes, application logic, cache (invalidation); we can talk about fixing. For fixing of course it would be great if you could extract a meta model of the application and another of the hardware. Changes to those models and some nice prediction of impact would be the ideal end goal. Or, if we are wishing anyway, some nice DSE on those models to optimize would be even better. [0] https://research.google.com/pubs/pub36356.html [1] http://opentracing.io/ [2] http://lightstep.com/ [3]…

If you're interested in distributed system tracing there's a lot going on. As a starting point, I would recommend reading Google's paper on their project "Dapper." [1] It's essentially the core of most distributed tracing systems. At least those I've encountered. There's a lot of tooling out there that take their cues from Dapper. I've recently been looking into integrating OpenZipkin[2] with our systems. I see at as a more viable alternative (no tie in!) to Yet Another Propriety Thing in AWS (YAPTA). There's other as well, like AppDash. Recently there has been a push towards an open-standard for the collection side called OpenTracing[3]. I came across it when investigating LightStep[4]. Ideally that means no vendor lock-in, which of course has lots of knock-on effects. If you know of any, I'd love to be pointed in the direction of _different_ and not just divergent techniques. [1]: http://static.googleusercontent.com/media/research.google.com/en//arch… [2]: http://zipkin.io [3]: http://opentracing.io [4]: http://lightstep.com - Impressive team behind this.

Hey! I can't speak to the exact mechanism for pricing on a large or enterprise level account, but it sounds like you're just getting started with tracing - might be worth checking out Lightstep. Pricing is pretty transparent for different tiers of non-enterprise usage and it's seats+services monitored, NOT consumption, so there shouldn't be any end-of-month surprises. (https://lightstep.com/pricing) Lightstep and several other vendors use open source standards so worst case scenario, you have great instrumentation even if you're not paying for a product right away. /shrug (full disclaimer, I work at Lightstep albeit not in sales)

LightStep | Full-Stack Product Engineer Location: San Francisco, REMOTE https://lightstep.com/careers LightStep | Software Engineer Location: San Francisco, REMOTE https://lightstep.com/careers LightStep makes complex software systems understandable, giving our customers confidence at scale. We collect, store, and query complex performance data so our customers can get insight from it. We're a 75-person company, growing quickly, and looking for hungry & humble engineers to build great software.

https://lightstep.com, but that’s the only one :)

Lightstep | San Francisco or Remote | https://lightstep.com Lightstep, recently acquired by ServiceNow, builds observability tools for proactively monitoring software in today's microservices-driven world. We write for other engineering teams, and are the only company that develops observability tools at the scale and complexity that enterprises need them. We're especially looking for people-minded engineers who are excited about shipping useful, meaningful products for the developer community. Check out our latest blog posts: https://lightstep.com/blog Our open positions: * All Roles: https://lightstep.com/about/careers * Software Engineer, Backend (remote friendly) * Software Engineer, SRE (remote friendly) *many other roles including TAMs, PM, AE, SE, etc. We're big believers in growth and that a career path can take many different shapes and forms; with that being said, for remote candidates, we do request prior professional experience in your discipline/field of interest. Tech Stack: d3.js/React/Redux single page webapp, Go backend Email swilson[at]lightstep[dot]com for inquiries.

It's Lightstep (https://lightstep.com).

I'm disappointed that all of the open-source tracing systems have adopted the Dapper [0] model. It's understandable why: it's extremely simple to implement, as it handles scaling challenges by doing client-side sampling. A bit of background about how Dapper-style distributed tracing works. Things typically start with an RPC call of some kind (typically from an external source like a public load balancer). At that point, you must decide whether to trace this request or not, which is typically done as a random sample (say, 1% of requests). At that point, the request gets assigned a _trace id_, a random identifier for that request. The trace id is stored in some request context and propagated to each subsequent service. Each service, meanwhile, divides up its request processing flow into a series of "spans" which represent some piece of computation. For example, a span cover an RPC call or a DB query. Spans are identified by a random _span id_. Once a request has been sampled, all spans for that request are sent to a central span collector where they're stored for later querying. This model is simple but very limited. It's often hard to know whether a trace is interesting at the outset, hence the reliance on random sampling. For example, you might want to understand why your 99p latency is high, but if you're just sampling 1% the 99p requests will only be 0.01% of your sample. More generally, interesting events (like errors or slow requests) tend to be rare, and sampling a…

I found this good guide on observability for devs by lightstep: https://lightstep.com/observability

Lightstep has recently changed their pricing, it's now the most cost effective option out there. You pay $100 per service with basically unlimited data ingestion, it's a fabulous deal. https://lightstep.com/pricing/ As for scrubbing PII, they are now supporting the OpenTelemetry tracing API which does this as standard. For query endpoints you will see something like "QUERY Users where name=? email=?", i.e. masking with "?" chars as you only care about the keys since those what determine your indexing or lack thereof. (This is handled in the OpenTelemetry application library/plugin level.) As an aside, PII scrubbing should be done that whether or not you own the cluster, because even if you own the cluster you generally don't want your support staff seeing PII esp. as organization grows larger.

Distributed tracing, a way of tracing requests across services, e.g. Zipkin [0] (which is what I believe Twitter uses), Jaeger [1] or Lightstep [2] [0] https://zipkin.io/ [1] https://www.jaegertracing.io/docs/1.18/architecture/ [2] https://lightstep.com/distributed-tracing/

Btw, if you are interested in distributed tracing, including the W3C and opentelemetry, we blog a weekly roundup about it here: https://lightstep.com/blog/category/distributed-tracing/

Isobel is writing weekly summaries as "OTel Me More" with the latest here: https://lightstep.com/blog/otel-me-more-opentelemetry-project-news-vol…

I found this write up with some metrics/ timeline on Twitter https://lightstep.com/blog/googles-june-2nd-outage-their-status-page-n… I still don't think it's the full picture. But better than nothing

Lightstep | SAN FRANCISCO, CA | FT, ONSITE, VISA XFER OK | 15+ Openings | www.lightstep.com Lightstep allows you to Monitor what matters most and diagnose anomalies within seconds across web, mobile, monoliths, and microservices. We are hiring for Software Engineers, Engineering Managers, Product Managers, Product Designers... Check out our team here: https://lightstep.com/about/ Open Positions: www.lightstepcareers.com

LightStep | San Francisco or Remote | https://lightstep.com LightStep builds observability tools for proactively monitoring software in today's microservices-driven world. We write for other engineering teams, and are the only company that develops observability tools at the scale and complexity that enterprises need them. We're especially looking for people-minded engineers who are excited about shipping useful, meaningful products for the developer community. To learn more about our diverse, thoughtful team; our engineering culture; and whether this is the right place for you, read our Key Values profile: https://www.keyvalues.com/lightstep You can also check out some recent blog posts, tech talks, and demo: * Play with our product for free! https://lightstep.com/play/ * "How we write code at LightStep" https://lightstep.com/blog/how-we-write-code-at-lightstep-lab-notebook… * "Trace driven development" https://www.youtube.com/watch?v=NU-fTr-udZg&t=4s * "How GCS processes cloud functions 4x faster than AWS" https://research.lightstep.com/reports/google-cloud-storage Our open positions (Hiring multiple roles but actively seeking EM): * All Roles: https://boards.greenhouse.io/lightstep?gh_src=8d0d8b781 * Engineering Manager (SF) - high priority: https://boards.greenhouse.io/lightstep/jobs/924680?t=8d0d8b781 We are not able to sponsor new H1B visas at this time. Tech Stack: d3.js/React/Redux single page webapp, Go backend Email Luis[at]lightstep[dot]com for…