Brian Harry

MS stores windows codebase in a single repo (https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan… 300GB. I don't really see a benefit of having all code of an org in a single repo. Single product, sure. But whole company? Why. Not to mention, these are serious outliers. From security standpoint, it doesn't seem great, from practical side, it's not great (bandwidth cost ect).

Worth noting that due to the size of the Windows repo, they ended up having to extend git significantly https://devblogs.microsoft.com/bharry/scaling-git-and-some-back-story/ https://github.com/microsoft/VFSForGit

Git was once fast for my use too. Then I returned to Microsoft to work on building Windows. Even after 2.5 years of improvement,[1] "git is slow" is still one of the most common complaints about the Windows engineering system. On my Skylake+SSD desktop, it takes over ten seconds to switch branches with git checkout, several of which are used just to calculate the "your branch is ahead of origin/your-branch-name by N commits" value. We are making many great strides here at Microsoft to improve git performance, such as with VFS for Git which my Windows repo clone uses. But with hundreds of gigabytes in a unversioned download of the source tree, there is only so much we can do. [1] since https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

To follow up on this, for anyone who might have seen it, a scan of the Word app for Android indicates that it contains * Google Firebase Analytics * HockeyApp The former I've seen in about 90% of apps I've downloaded from the Google Play Store. The latter claims to be "the best way to collect live crash reports, get feedback from your users, distribute your betas, and analyze your test coverage." Oh, and Microsoft owns it. [1] Neither seems particularly unusual to me. I wonder what has caused this much outrage: * Assumptions about the nature of the "tracking" ("specializes in predictive profiling" leaves a lot to the imagination) * Assuming that other apps aren't doing the same thing * A general expression of anger at all apps that have embedded scripts like these [1] https://devblogs.microsoft.com/bharry/microsoft-acquires-hockeyapp/

There is a lot more to it than that. Check out "The largest Git repo on the planet" by Brian Harry who was in charge of the git migration and Azure DevOps (Microsoft's pendant to GitHub) https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

Microsoft had issues git’s ability to scale. According to this[1] they found and submitted solutions to allow git to scale to their needs and worked with the git team. It sounds like Facebook wasn’t partnering with them to solve the problem? The article mentions Microsoft had repos with 6m files and says things now take seconds. I’m not personally dealing with repos of this size, but it sounds like this is a problem of the past, thanks to Microsoft. I have a hard time believing Facebook would be a bigger beast than Windows. [1] https://devblogs.microsoft.com/bharry/scaling-git-and-some-back-story/

> how big is their OS engineering team? (…) maybe 100 folks or so? From https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan… : > the Windows team is about 4,000 engineers and the engineering system produces 1,760 daily “lab builds” across 440 branches in addition to thousands of pull request validation builds

That already exists. It’s called Scalar[1] and it has been built-into Git since October 2022[2], dates back to 2020[3] and is the spiritual successor or something Microsoft was using as far back as 2017[4]. 1. https://git-scm.com/docs/scalar 2. https://github.blog/2022-10-13-the-story-of-scalar/ 3. https://devblogs.microsoft.com/devops/introducing-scalar/ 4. https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

Well, interestingly Microsoft claims that Windows is the largest git repo on the planet (2017). And it looks bigger than Linux by its repo size: > the Windows code base is approximately 3.5M files and, when checked in to a Git repo, results in a repo of about 300GB. Further, the Windows team is about 4,000 engineers and the engineering system produces 1,760 daily “lab builds” across 440 branches in addition to thousands of pull request validation builds. https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

A similar effort was made by Microsoft a while back to scale up Git in order to use with the Windows monorepo: https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

Google is mimicking perforce command line. The backend is 100% proprietary. Microsoft is based on Git, but with a lot of engineering on top of it: https://devblogs.microsoft.com/bharry/scaling-git-and-some-back-story/

I belive the entire Windows codebase is in a git monorepo now, for some years: https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

This is also an interesting system in that it's an example of how you can get away with a non-distributed system if your problem is small enough but eventually that falls over. Once you get to large corporation monorepos git operations start to get real slow and use too much hard disk, so you end up with them either creating/using a new VCS or doing some complicated undertaking like https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

Clickable link to blog:- https://devblogs.microsoft.com/bharry/paid-extension-in-the-visual-stu…

https://www.google.com/search?q=microsoft+monorepo https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

Windows is in one repo at Microsoft, I believe. https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

This reminds me of VFS for Git, Microsoft’s solution for scaling Git for the Windows code base. [1] [2] [3] [1]: https://github.com/microsoft/VFSForGit [2]: https://news.ycombinator.com/item?id=14411126 [3]: https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

In this context "monorepo" means a huge repository with many many revisions. Git has several well documented deficiencies in this scenario. See [1] for microsoft's experiences with git. [1] https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

> Cases where git needs to scale tend to be isolated to companies that have the manpower to build a finely-tuned replacement (see: MS, Google). I recollect that for Windows (which also uses git), MS have actually extended git with "Git Virtual File System" rather than replace it[1]. But I do agree that broadly, not everyone needs the scale. [1] https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan…

I have not analyzed the full potentials and benefits of Diversion but I would not agree with the statements you made about the Git. I think you should not focus on Git in your pitch. >>it was built for a very different world in 2005 (slow networks, much smaller projects, no cloud) Slow network: why is this a negative thing? If something is designed for a slow network then it should perform well in a fast network. Mush small project: I do not agree. I can say that it was not designed for very very large projects initially. But many improvements were made later. When Micorosoft adopted Git for Windows, they faced this problem and solved it. Please look at this https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan… No cloud: Again I would not agree. Git is distributed so should work perfectly for the cloud. I am not able to understand what is the issue of Git in the cloud environment. >>In our previous startup, a data scientist accidentally destroyed a month’s work of his team by using the wrong Git command This is mostly a configuration issue. I guess this was done by a force push command. IFAIK, you can disable force push by configuration.

> the complete editing history of the Linux kernel (probably the largest Git repo ever created) Linux is definitely not the largest git repo ever created [0]. The big corporate monorepos are definitely larger; I know MS has moved Windows to git, and itself claims it to be the largest ever created (~300GB as of 2017, per [1]). Google and Facebook both eschew git, though. Finding data on the largest open repos is more difficult. The largest classes of projects are those that develop in monorepos that implement critical operating system [2] functionality, browser engines, and compiler implementations. The shortlist I'd make comes out to these projects (in no particular order): * gcc * LLVM * Mozilla * Chromium * Linux * OpenJDK I haven't finished downloading all of these repos (my disk is begging me to stop right now), but it looks Linux is larger than gecko-dev by a very thin margin (so a putative gecko-dev that included comm-central with its CVS history as well would easily outstrip Linux), and Chromium seems to be an order of magnitude over both. [0] To be clear here, I'm mostly thinking in terms of primarily textual repositories. Repositories with large binary assets are clearly not relevant for your means. [1] https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan… although https://news.ycombinator.com/item?id=14411724 claims that the 300 GB measures the size of the checked-out directory on disk, not the putative size of a full .git…

microsoft: in 2017, the Windows git repo contained ~3.5 million files & used 300GB when checked out as a git repo [1], In 2020, the Office git repo contained ~3 million files [2]. facebook: In 2011, the main site had around 9.2 million lines of code "excluding numerous backend services" [3]. In 2014 the android codebase was 4 million lines of code and the main site without backend code was 62 million lines of code [3]. In 2014, when facebook were talking about scaling mercurial, the main source repo was described as "many times larger than even the Linux kernel, which checked in at 17 million lines of code and 44,000 files in 2013". Around 2018 (?) facebook started using "eden", a fork of mercurial [5]. If we make the working assumption that expected lines-of-code / file of 2013 linux kernel is representative of all repos, then that gives around 386 lines-of-code / file. loc(microsoft) >= loc(windows in 2017) + loc(office in 2020) ≈ 386 loc / file * (3.5m + 3m) ≈ 2.5 billion lines of code The linked blog posts from microsoft & facebook talking about scaling version control systems are fairly interesting! [1] https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-plan… [2] https://devblogs.microsoft.com/devops/introducing-scalar/ [3] https://www.quora.com/How-many-lines-of-code-is-Facebook [4] https://engineering.fb.com/core-data/scaling-mercurial-at-facebook/ [5] https://github.com/facebookexperimental/eden