Buf Blog

We keep an eye on the protobuf repo, so if a change is made we can both incorporate it into our products (https://buf.build) and into this spec. A great outcome for the ecosystem would be that Google chooses to engage with the community before making language changes. And the best possible outcome would be that the authority is eventually inverted: a specification doc becomes the definitive source of truth on the language and `protoc` is updated to conform to it, instead of vice versa.

> Open-sourcing protobuf was most likely not a big cost, and it had benefits for Google (otherwise it would not have been rational to open source it). Nah honestly it was more like I said "Hey I want to open source this!" and other engineers said "Yeah that sounds awesome!" so I did. That was Google in ~2007, you could basically do whatever you wanted and as long as it was vaguely plausibly useful management said "sure". I'm honestly not sure the company had any idea what they should do with OSS protobuf, at least for many years. Management never really gave me any credit for releasing it and gently pressured me to find a different project to work on that more directly aligned with company goals, which I eventually did. Still not clear if they even know today. They seem more serious about it today but at the same time it's still clear that they mostly don't care about external users, so much so that startups like https://buf.build have to come in and build the actual products around Protobuf that Google never delivered publicly (but has internally). Of course, companies are not hive minds -- there are clearly people at Google who want to make Protobuf a great product and are trying to do so, but I don't think management sees it as strategically interesting, so they face headwinds. > If protobuf was not open source, something like Cap'n Proto may take its place, and that's not good for Google. Honestly I don't know if Google cares. Cap'n Proto hasn't caught on much…

I'm not too surprised. Most people for whom TypeScript is an option won't have been exposed to generations of RPCs that fail to deliver simplicity. So it isn't going to be on their radar. I've been using gRPC and REST'ish+JSON APIs for years now and what I find puzzling is that REST+JSON tends to mean a lot more work, less pleasing code than gRPC, and yet people prefer it. Not because it leads to simpler, better, faster less error prone code (it doesn't. Quite the opposite), but, I think, because people feel they can understand it. The people over at https://buf.build have been doing a great job trying to tame gRPC btw. The protoc toolchain from Google has been uniquely horrible to work with, but the 'buf' tool from said company has been a real life-saver. Not to mention their Buf Schema Registry, which admittedly I have only used on a few projects so far, but should migrate more projects to. Though in general, I feel that RPC mechanisms that are too closely tied to a given language are a waste of time. But that's me. tRPC isn't something I'd get into even if I was doing server side TS. It just doesn't seem like a good long term choice.

I've only enjoyed using Protobuf + gRPC after we've started using https://buf.build. Before that it was always a pain with Makefiles building obscure commands, then developers having different versions of the Protobuf compiler installed and all kinds of paper cuts like that. Now it's just "buf generate", every developer has the exact same settings defined in the repo and on the frontend side we are just importing the generated Typescript client and have all the types instantly available there. Also nice to have a hosted documentation to link people to. My experience is mostly with Go, Python and TS.

I've been having fun with connectrpc https://connectrpc.com/ It fixes a lot of the problematic stuff with grpc and I'm excited for webtransport to finally be accepted by safari so connectrpc can develop better streaming. I initially thought https://buf.build was overkill, but the killer feature was being able to import 3rd party proto files without having to download them individually: deps: - buf.build/landeed/protopatch - buf.build/googleapis/googleapis The automatic SDK creation is also huge. I was going to grab a screenshot praising it auto-generating SDKs for ~9 languages, but it looks like they updated in the past day or two and now I count 16 languages, plus OpenAPI and some other new stuff. Edit: I too was swayed by false promises of gRPC streaming. This document exactly mirrored my experiences https://connectrpc.com/docs/go/streaming/

Buf | Software Engineer (Frontend or Infrastructure) | Remote (US or Canada) | Full time | https://buf.build/ Buf is building a Protobuf platform that provides end-to-end type safety and governance across all APIs and streaming data in Kafka. We’re deployed everywhere from small startups to the world’s largest enterprises, and we’re hiring to scale our growing business. Tech stack: Go, TypeScript, Kubernetes If you are interested to work at Buf, apply to one of our open roles here: https://buf.build/careers

Side-ish note, I really enjoyed a submission on Bufstream recently, a Kafka mq replacement. One of the things they mentioned is that they are working on building in Iceberg materialization, so Bufstream can automatically handle building a big analytics data lake out of incoming data. It feels like that could potentially tackle a bunch of the stack here. https://buf.build/blog/bufstream-kafka-lower-cost https://news.ycombinator.com/item?id=40919279 Versus what Notion is doing: > We ingest incrementally updated data from Postgres to Kafka using Debezium CDC connectors, then use Apache Hudi, an open-source data processing and storage framework, to write these updates from Kafka to S3. Feels like it would work about the same with Bufstream, replacing both Kafka & Hudi. I've heard great things about Hudi but it does seem to have significantly less adoption so far.

Our team has been using Buf (https://buf.build) recently, and they have a nice solution for schema dependency management.

It's worth checking out our take on a lot of these problems: https://buf.build/blog/connect-web-protobuf-grpc-in-the-browser

Everyone I've worked with save a few that experienced protobufs with extentions immediately finds the benefit once they use them. However I have yet to convince anyone at an org that isn't already using them that they will change things for the better. Schema once, code everywhere. It is really nice. Have you seen Buf? https://buf.build/

Buf CLI itself is licensed under a permissive Apache 2.0 License [0]. Since Buf is a compiler, its output cannot be copyrighted (similar to proprietary or GPL licensed compilers). DISCLAIMER: I am not a lawyer. Buf distinguishes a few types of plugins: the most important being local and remote. Local plugins are executables installed on your own machine, and Buf places no restrictions on use of those. Remote plugins are hosted on BSR (Buf Schema Registry) servers [1], which are rate limited. All remote plugins are also available as local plugins if you install them. It's worth to mention that the only time I've personally hit the rate limits of remote plugins is when I misconfigured makefile dependencies to run buf on every change of my code, instead of every change of proto definitions. So, for most development purposes, even remote plugins should be fine. Additionally, BSR also offers hosting of user proto schemas and plugins, and this is where pricing comes in [2]. [0] https://github.com/bufbuild/buf/blob/main/LICENSE [1] https://buf.build/blog/remote-plugin-execution [2] https://buf.build/pricing

https://buf.build/ is this, no?

buf is a parser / compiler as an alternative to protoc https://buf.build/docs/migration-guides/migrate-from-protoc/

In SpiceDB, this is known as the LookupResources [1] API, which returns all resources (of a particular type) that a particular subject (user in this case) has a particular permission on. We have a guide on doing ACL-aware filtering and listing [2] with this API and describing other approaches for larger Enterprise scales Disclaimer: I'm the co-founder and CTO of AuthZed, we develop SpiceDB, and I wrote our most recent implementation of LookupResources [1]: https://buf.build/authzed/api/docs/main:authzed.api.v1#authzed.api.v1.… [2]: https://authzed.com/docs/spicedb/modeling/protecting-a-list-endpoint

I have had a really good experience with https://connectrpc.com/ so far. Buf is doing some interesting things in this space https://buf.build/docs/ecosystem/

The official grpc JavaScript implementation is sort of bad. The one by buf.build is good from what I've seen. https://buf.build/blog/protobuf-es-the-protocol-buffers-typescript-jav…

When I was at Google around 2016, there was a significant push to convince folks that the proto3 implicit presence was superior to explicit presence. Is there a design doc with the rationale for switching back to explicit presence for Edition 2023? The closest docs I've found are https://buf.build/blog/protobuf-editions-are-here and https://github.com/protocolbuffers/protobuf/tree/main/docs/design/edit…

I’m looking at the product page [0] and wondering how those two statements are compatible: > Bufstream runs fully within your AWS or GCP VPC, giving you complete control over your data, metadata, and uptime. Unlike the alternatives, Bufstream never phones home. > Bufstream pricing is simple: just $0.002 per uncompressed GiB written (about $2 per TiB). We don't charge any per-core, per-agent, or per-call fees. Surely they wouldn’t run their entire business on the honor system? [0] https://buf.build/product/bufstream

They are the protobuf descriptors, which is basically the protobuf encoded version of the protobuf files. They can be optional and many plugins have options to exclude that part from the generated code. Generally, they're useful for the Server Reflection feature of gRPC and it's used for runtime reflection. Here's a more complete description of what they are and how they're used: https://buf.build/docs/reference/descriptors.

It'd be cool if you supported the Buf Schema Registry's Reflection API: https://buf.build/docs/bsr/reflection/overview Then, the descriptors could be delivered on demand!

I'm surprised the author doesn't mention ConnectRPC: https://connectrpc.com/ It solves ALL the problems of vanilla gRPC, and it even is compatible with the gRPC clients! It grew out of Twirp protocol, which I liked so much I made a C++ implementation: https://github.com/Cyberax/twirp-cpp But ConnectRPC guys went further, and they built a complete infrastructure for RPC. Including a package manager (buf.build), integration with observability ( https://connectrpc.com/docs/go/observability/ ). And most importantly, they also provide a library to do rich validation (mandatory fields, field limits, formats, etc): https://buf.build/bufbuild/protovalidate Oh, and for the unlimited message problem, you really need to use streaming. gRPC supports it, as does ConnectRPC.

I believe buf’s breaking change detection will pick up on this. It also allows you to specify a git ref to target. https://buf.build/docs/reference/cli/buf/breaking#against

But has a registry and has a tool to use the regiatry to call grpc https://buf.build/blog/buf-curl/ Didn't dig deeper to see if the bud tooling can help me deciding protobufs laying around when it's not about a grpc call

Better than an alternative is Connect, which is gRPC but also supports gRPC-Web and the lighter weight Connect protocol (which seems inspired by Twirp) https://buf.build/blog/connect-a-better-grpc I like it because it solves most of my complaints about using gRPC from Go without giving up on gRPC.

The DX is really cool, but I remembered protobuf exists https://buf.build/blog/connect-web-protobuf-grpc-in-the-browser The marketing rhetoric of telefunc activates my skepticism, but I hope it's just me

When shit like this happens, I just always think "I don't understand finance at all and never will." I read the company's primary blog blog post, https://buf.build/blog/api-design-is-stuck-in-the-past, about "schema driven development" and agree with a lot of it. Which is why I'm a huge fan of GraphQL and related completely free open source libraries, where I define my API endpoints with a strongly typed yet easily evolvable schema, and auto-generate my Typescript types from my GraphQL definitions. $93 million dollars is just nuts to me.

also, shout out to the Buf guys: https://buf.build/ocpx/protocol/docs

This is a pretty good analysis of the situation I think. The only thing that I would like to point out is that only a description of the API is available via the Zanzibar paper. We, and other clean-room implementations, have had to fill in the gaps around what the APIs actually look like. We've even made some of our own improvements such as resource lookup by subject[0] and a filter-based delete that mirrors the read API[1]. [0] https://authzed.com/blog/acl-filtering-in-authzed/ [1] https://buf.build/authzed/api/docs/main/authzed.api.v1#authzed.api.v1.…

This is an absolutely wonderful article. I helped build a system very similar to the one quoted below: > A deploy pipeline that queries your orchestrator for running image tags, checks your migration history against the schema registry, diffs your GraphQL schema against collected client operations, and runs Buf’s compatibility checks: this is buildable today, with off-the-shelf parts. that was largely successful at eliminating interservice compatibility errors, but it felt like we were scrambling around in a dark and dusty corner of software engineering that not many people really cared about. Good to see that there's a body of academic work that I was not entirely aware about that is looking into this. The article successfully identifies that, yes, there are ways, with sufficient effort, to statically verify that a schema change is (mostly) safe or unsafe before deployment. But even with that determination, a lot of IDLs make it still very hard to evolve types! The compatibility checker will successfully identify that strengthening a type from `optional` to `required` isn't backwards compatible: _now what_. There isn't a safe pathway for schema evolution, and you need to reach for something like Cambria (https://www.inkandswitch.com/cambria/, cited in the article) to handle the evolution. I've only seen one IDL try to do this properly by baking evolution semantics directly into its type model, typical (https://github.com/stepchowfun/typical) with it's `asymmetric` type…

ConnectRPC works over the web and I've built several web apps (SPAs) with it. It works with JSON by just setting the Content-Type header to `application/json`. You can add compression easily on the backend with a single option. Generating typed stub code is one line (`buf generate`). Connect-ES is awesome - it's a Typescript-first protobuf library for the web. Interacting with the API is very simple, for example: const resp = await gameEventClient.getRecentAnnotatedGames({ limit: annotatedPageSize, offset: recentAnnotatedGamesOffset, }); setRecentAnnotatedGames(resp.games); These function calls are typed and a lot of the code is auto-generated from the .proto spec (i.e. above the gameEventClient and getRecentAnnotatedGames are auto-generated code). On the server side the code is also obviously auto-generated. It all works seamlessly. Even the documentation for the API can be auto-generated. See for example: https://buf.build/domino14/liwords The above documentation was auto-generated from my protobuf files. This project uses ConnectRPC on the backend and on the front-end SPA. To me it seems so much simpler and better than the way I've seen people use OpenAPI - where many people seem to create the code _before_ creating the spec. I actually haven't found a good Go generator of stub code from an OpenAPI spec. With ConnectRPC it just works, it's simple, easy, fast, etc. It's easy to add interceptors to do things like parse the…