Google Engineering Tools Blog

Pointing at a big company and saying "they're doing it wrong" is easy enough to do, but you have to remember that every decision comes with tradeoffs. Take Google's codebase, since it's the one I know the best. A couple of the key decisions: * Single rooted tree. Separated repositories would make it harder to share code, leading to more dupication. * Build from head. We build everything from source, statically linked. No need to worry about multiple versions of dependencies, no lag between a bug fix and it being available to any and all binaries that need it, whenever they're next updated. I don't think that an "internal Github" is going be a magic bullet here. It's more likely it would be a matter of trading one set of hard problems for another, as we all of a sudden need to figure out how to do cross-dependencies sanely, deal with multiple versions of libraries, etc, at scale. You are correct that one monolithic Perforce repo is a bit of a pain point, but that doesn't necessarily mean that the right decision is to shatter our codebase into different pieces - we'd rather make our repo scale better. For reference, we've already got hundreds of millions of lines of code, 20+ changes/minute 6 months ago (so what, 30+ now?), and plans for scaling the next 10x are in motion. If you're interested, I recommend http://google-engtools.blogspot.com/. It details a number of the problems we've run into, and our solutions for dealing with them at scale.

I had asked a similar question on Quora [1] few years back. Here is the summary: - Facebook: https://www.facebook.com/Engineering?sk=notes - Google: http://google-engtools.blogspot.com/ - Twitter: http://engineering.twitter.com/ - Linkedin: http://engineering.linkedin.com/blog - Dropbox: https://tech.dropbox.com/ - Instagram: http://instagram-engineering.tumblr.com/ - Netflix: http://techblog.netflix.com/ - Flickr: http://code.flickr.com - Etsy: http://codeascraft.etsy.com/ - Yelp: http://engineeringblog.yelp.com/ - Heroku: https://blog.heroku.com/ - Airbnb: http://nerds.airbnb.com/ - Bitly: http://word.bitly.com/ - Evernote: https://blog.evernote.com/tech/ - Highscalability: http://highscalability.com/ [1] https://www.quora.com/What-are-the-companies-that-have-some-of-the-bes…

This is a great idea. Interestingly, Google has a tool that will analyse your git history and identify "hotspots" i.e code that is regularly associated with commit messages with words like "fix". I'm wondering if the same general idea is applicable to other types of commits given your list. For example, if you are regularly adding features and a certain part of the code base is touched, perhaps with a lower ratio of "refactor" commits, that code could be a solid candidate for refactoring. Here's the tool i mentioned anyway https://google-engtools.blogspot.co.uk/2011/12/bug-prediction-at-googl…

The BUILD file format looks a lot like Python code calling some functions. http://pantsbuild.github.io/first_tutorial.html#build-files . It'd be great to have something more readable. For what it's worth, Google's been using a similar format for several years, and folks seem to find their way OK: http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-s… The word "goal" should disappear from the command line at some point. Alas, we're not quite there yet.

Honestly, I've heard the majority of it before, and I have never been a Google employee. I have, however, listened to some of their talks on the internal workflows and processes that they've presented and posted on youtube. I think the only things I didn't hear before were some of the specific toolnames. edit: http://google-engtools.blogspot.com/ here's their engineering tools blog. http://www.youtube.com/watch?v=2qv3fcXW1mg and a talk http://www.youtube.com/watch?v=b52aXZ2yi08 and another talk Really cool stuff.

That sounds like some of the posts on this blog: http://google-engtools.blogspot.com/ Also, related to source control but not Git, a few years ago Google had a tech talk about writing a Mercurial storage system on top of BigTable: http://www.google.com/events/io/2009/sessions/MercurialBigTable.html

I believe this is Amin's paper you refer to: https://www.usenix.org/legacy/publications/library/proceedings/usenix9… And for shorter reading, there's also djb's 'redo' thing: http://cr.yp.to/redo.html - specifically, the section of "Target files depend on nonexistent files" is an interesting one if you have tools that are allowed to generate dependencies for source files that aren't part of the source files. (It turns out, if you follow that road, that your build tools and all their libraries are implicit dependencies in the graph as well. Second law of build graphs: You will at some point question if it should be legal to do unspeakable things to people who rely on pseudo-randomness in their build tools) It is a conceptually simple problem which turns out to be surprisingly hairy in practice. That's why it's fun, and so many folks noodle on it :) If you're interested in the field, I know Google has talked about theirs: https://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-… - and I'm sure many other companies as well.

What you are describing is an artifact of the old Perforce which copied everything in your client to local storage. After the conversion to srcfs and piper, which was more than a decade ago, this became unnecessary. http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-s…

And Google has Piper and Srcfs http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-s…

Here is Google’s take on bug prediction from a while back: https://google-engtools.blogspot.com/2011/12/bug-prediction-at-google.…

Not sure exactly, since I haven't used any of the dozens of similar efforts. It sounds like you're a lot more familiar with the area, so here's some resources: * Bazel's FAQ: https://bazel.build/faq.html * A description on how Blaze works: https://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-… Unfortunately the design doc for Blaze is internal and AFAIK not available publicly. I imagine they took a look at a number of the similar efforts and made decisions based on what those approaches did wrong.

Google already has a FUSE layer for source control: http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-s…

With CitC meaning all the source is instantly, "locally" available and up to date, and something like bazel.io meaning you always have a reliable and fast way to rebuild something out of cached/distributed build outputs[1], you don't need to distribute a binary in the general case. Whoever needs a tool can cd to the source of the tool, build it right then and there, and run it. [1] http://google-engtools.blogspot.com/2011/10/build-in-cloud-distributin…

So something like this: http://google-engtools.blogspot.ca/2011/12/bug-prediction-at-google.ht…

Google open-sourced their own Brazil, or at least parts of it: http://google-engtools.blogspot.ie/2011/08/build-in-cloud-how-build-sy… http://www.quora.com/Does-Google-consider-releasing-its-build-infrastr… http://bazel.io/

> You get to go on another vacation while you wait for your monorepo to update. If you care, you should read about how Google makes sure this doesn't happen. > By putting our revision history in the cloud we provide engineers with complete access to all the source, and yet almost no time is spent checking out code. http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-s… http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-s…

There's not a white paper, but there's this series of posts: http://google-engtools.blogspot.com/2011/06/build-in-cloud-accessing-s… (and an accompanying presentation)

Some publicly available information: http://google-engtools.blogspot.ie/2011/08/build-in-cloud-how-build-sy… (https://news.ycombinator.com/item?id=2899467)

Google has publically described their build system here http://google-engtools.blogspot.com/2011/08/build-in-cloud-how-build-s… (low fidelity) and here https://www.youtube.com/watch?v=2qv3fcXW1mg (high fidelity)

The title is a direct quote from the Google Engg. Tools blog post which was on HN recently - http://google-engtools.blogspot.com/2011/12/bug-prediction-at-google.h…

These are some of the top sites' engineering blogs which I collected over a period of time - I removed those which have already been posted: Google: http://googledevelopers.blogspot.com/ http://google-engtools.blogspot.com/ http://googleresearch.blogspot.com/ YouTube: http://youtube-eng.blogspot.com/ Baidu: http://usa.baidu.com/category/baidutechblog/ Yahoo: http://yahooeng.tumblr.com/ http://yahoodevelopers.tumblr.com/ Wikipedia (closest I could find): https://blog.wikimedia.org/c/technology/ Amazon: https://aws.amazon.com/blogs/aws/ Twitter: https://engineering.twitter.com/ eBay: http://www.ebaytechblog.com/ Microsoft: http://microsoftengineering.com FourSquare: http://engineering.foursquare.com/ Apple (closest): https://developer.apple.com/ Paypal: https://www.paypal-engineering.com/ https://devblog.paypal.com/ Dropbox: https://blogs.dropbox.com/tech/ Edit: More (collected from this link: http://www.quora.com/What-are-the-top-startup-engineering-blogs): Yelp: http://engineeringblog.yelp.com/ Quora: http://engineering.quora.com/ Yammer: http://eng.yammer.com/blog/ AirBnB: http://nerds.airbnb.com/ Square: https://corner.squareup.com/ Flickr: http://code.flickr.net/ Soundcloud: https://developers.soundcloud.com/blog/ Cloudera: http://blog.cloudera.com/blog/ Slideshare: http://engineering.slideshare.net/ Heroku: http://engineering.heroku.com/ Pinterest: http://engineering.pinterest.com/

I program in C++ at Google. I won't make claims about "stellar". Here are a few Google-specific reasons Go hasn't replaced C++ for me: * The long C++ build times he was complaining about are mostly gone due to distributed builds. http://google-engtools.blogspot.com/2011/09/build-in-cloud-distributin… Throwing resources at the problem isn't ideal, but it does work. * Google has a lot of C++ library code, including client libraries for things like Bigtable. Go's Google-specific libraries are a work in progress. * For serving systems, I care more about machine efficiency and latency for than about development time. I'm still skeptical of those properties in Go. Obviously the compiler (and my comfort with it) will improve over time. So will the garbage collector, to a point. It will always operate on one large heap like Java's. People say Java's GC works fine, but they spend a lot of time tuning it and then quote pause times that don't thrill me. I've seen a lot of problems caused by bad tuning parameters, like (for example) Java heap sizes that exceed the size of the container it's running in. And relatively hard-to-diagnose and slow-to-recover GC death spirals because the defaults suck: Java just bogs down in situations where a C++ binary would dump a heap sample, crash, and get restarted. So I'd be more comfortable with a system like Rust's: many smaller heaps, language enforcement of ownership rules when transferring between its equivalent of…

I'm not a grandmaster by any means, but these are things that I focus on: 1. Aggressively avoid techniques and abstractions that have produced bugs in the past. The "backwards conditional" is an example of this, in languages like C that will accept x = 0 as a conditional, it is safer to do 0 == x in an if statement in case you mistype. Mutability trips me up sometimes, so I try to never re-assign a variable. I keep links to the source code of key libraries in my bookmarks bar so I never have the guess how something actually works. Static analysis tools, and bug prediction tools [1] are two steps I haven't taken. 2. Seek out code written by experts and inspect it [2]. If I'm considering using a new language or library one of the first things I do is to search for large scale OSS projects. For a new language looking at some of the popular 3rd party libraries can be really illuminating. Especially if you're already familiar with the problem space. 3. two words: learn to fuckin' type. 4. Get familiar with what the history researchers call the Primary Sources. Not just the actual source code but the proposals, the specifications, and the research reports that are written by the people who conceived the tools that you are using. [1] http://google-engtools.blogspot.ca/2011/12/bug-prediction-at-google.ht… [2] (EDIT) According to Paul Allen, in high school Bill Gates actually went dumpster diving for source code:…