GitHub Engineering

https://shopify.engineering/ https://developer.squareup.com/blog/ https://netflixtechblog.com/ https://githubengineering.com/

I'd recommend just using plain JavaScript. There's a great blog article by Githubs engineering team about how they removed Jquery from their front-end and now rely just on vanilla JavaScript: https://githubengineering.com/removing-jquery-from-github-frontend/ Also, don't be afraid of embracing multi page app versus going for a single page app (they're largely overrated). For front end CSS design framework, I'd recommend something like Bulma. Very easy, modern and CSS-only. You can learn it all in less than an hour.

I'm impressed how far GitLab has come in a short amount of time. It's a pretty well-designed platform, when it comes down to it. I think the Vue-based user interface and the data model are probably the cornerstones of it, and using Ruby on the backend just helps them iterate quickly. Ruby also seems to be getting more robust [1]. Their CI/CD platform is really a great approach, emphasizing the GitLab Worker as a portable CI runner that communicates through the API. For resource-intensive apps it helps to not have the number and size of CI workers be limited by a SaaS plan like they tend to be with other CI/CD platforms. [1] https://githubengineering.com/removing-oobgc/

I think it's relevant that Baron's post mentions a Github incident, and Github has been doing automatic failover of MySQL masters for a while now: https://githubengineering.com/orchestrator-github/ This post is also very relevant to the topic: http://code.openark.org/blog/mysql/mysql-high-availability-tools-follo… The post you link is still relevant to this discussion, but we can't ignore that 5 years have passed. It would be good to see 2017's Baron revisit the topic :)

I'm very impressed with the level GitHub database testing is at https://githubengineering.com/mysql-testing-automation-at-github/ We still got a lot of work ahead of us. But Yorick and Greg their work has been outstanding. Regarding Spokes we're not planning something like that at this time. From https://gitlab.com/gitlab-org/gitaly/issues/650 "GitHub eventually moved to Spokes https://githubengineering.com/building-resilience-in-spokes/ that had multiple fileservers for the same repository. We will probably not need that. We run networked storage in the cloud where the public cloud provider is responsible for the redundancy of the files. To do cross availability zone failover we'll use GitLab Geo."

Just for posterity, you asked for: > A feature that'd let me browse by "topic" (e.g systems programming) regardless of language would be welcome as well. Topics are exactly what you initially described. Perhaps you didn't do a very good job describing what you'd like. Also, they're called topics. Not tags. It's in the documentation. > Such feature would require Github to do the appropriate groundwork to become great at technology-discovery, namely They did a bunch of work to automatically add topics for many repositories to solve for this: https://githubengineering.com/topics/

No. https://githubengineering.com/gh-ost-github-s-online-migration-tool-fo… has an excellent section on why triggers fail (at least in the MySQL world. I imagine the locking story is similar in postgresql). I don't work for GitHub but we had the same problems using pt-online-schema-change independently of their issues (lots of locking contention affecting the app negatively). We're finally moving to gh-ost for our large/risky migrations and so far it's amazing.

GitHub: https://githubengineering.com/ Really surprised it wasn't mentioned yet. They do really in depth posts and show metrics too. Example: https://githubengineering.com/how-we-made-diff-pages-3x-faster/

I recently read a blog post from Github about them operating their own datacenter http://githubengineering.com/githubs-metal-cloud/ Im not positive, but it sounds like a fairly recent switch from a cloud provider to their own datacenter. If thats the case, Id expect a number of outages to come in the following months.

> b) increasing the probability of malicious injection Subresource Integrity (SRI) [0] was designed to avoid this potential vulnerability. It's a brand-new browser feature (jus t landed in Chrome stable, is riding the trains to Mozilla stable now) but it's worth taking a look at. Github is already using it [1]. [0]: http://www.w3.org/TR/SRI/ [1]: http://githubengineering.com/subresource-integrity/

Reading this post-mortem and their MySQL HA post, this incident deserves a talk titled: "MySQL semi-synchronous replication with automatic inter-DC failover to a DR site: how to turn a 47s outage into a 24h outage requiring manual data fixes" https://githubengineering.com/mysql-high-availability-at-github/#semi-… > In MySQL’s semi-synchronous replication a master does not acknowledge a transaction commit until the change is known to have shipped to one or more replicas. [...] > Consistency comes with a cost: a risk to availability. Should no replica acknowledge receipt of changes, the master will block and writes will stall. Fortunately, there is a timeout configuration, after which the master can revert back to asynchronous replication mode, making writes available again. > We have set our timeout at a reasonably low value: 500ms. It is more than enough to ship changes from the master to local DC replicas, and typically also to remote DCs. https://blog.github.com/2018-10-30-oct21-post-incident-analysis/#2018-… > The database servers in the US East Coast data center contained a brief period of writes that had not been replicated to the US West Coast facility. Because the database clusters in both data centers now contained writes that were not present in the other data center, we were unable to fail the primary back over to the US East Coast data center safely. > However, applications running in the East Coast that depend on…

I believe you're right. They go into their design in their MySQL HA post: https://githubengineering.com/mysql-high-availability-at-github/#semi-… > In MySQL’s semi-synchronous replication a master does not acknowledge a transaction commit until the change is known to have shipped to one or more replicas. It provides a way to achieve lossless failovers: any change applied on the master is either applied or waiting to be applied on one of the replicas. They only require one other replica (as opposed to a quorum) to be reachable from the master for the master to continue acknowledging writes. If a new master has been elected on the other side of the partition, both will continue acknowledging writes. They noted this as a limitation that they were working on (but unfortunately a bit too late in hindsight): > Notably, on a data center isolation scenario, and assuming a master is in the isolated DC, apps in that DC are still able to write to the master. This may result in state inconsistency once network is brought back up. We are working to mitigate this split-brain by implementing a reliable STONITH from within the very isolated DC. As before, some time will pass before bringing down the master, and there could be a short period of split-brain. The operational cost of avoiding split-brains altogether is very high.

It's hard. The general rule of thumb is that you make database changes independent of code changes, and you always make the changes in such a way that they are compatible with the current deployed code AND with the next version that you plan to roll out. For example: let's say you have a feature that uses a column, but you want to move it to using a separate table. Step 1: design the new table Step 2: deploy the new table - existing code continues to run against the column Step 3: run a back-fill to ensure the new table has all of the data that exists in the current column Step 4: deploy code to use the new table instead of the column The above is the best-case scenario - but it often doesn't work like that, because you need to ensure that data added to the old column between steps 3 and 4 is correctly mirrored across. One approach here is to deploy code that dual-writes - that writes to both the old column and the new table - along with extra processes to sanity-check the conversion. GitHub's Scientist library is a smart approach to these more complex kinds of migration - which can take months to fully deploy. https://github.com/github/scientist and http://githubengineering.com/scientist/

They get us mostly there. The rest of the way is how you combine getting that data into a browser. Subresource Integrity [1] includes the hash of the content/object in the link to the resource, as well as the URL/URI; this lets you do fun things like not fetch duplicate content if its already in the browser's cache from another site (think a specific version of jquery, for example). Magnet links [2] are a great example of this in practice, but that's bittorrent-specific (although I would not be surprised if you started to see ideas from Magnet URI scheme work their way into Subresource Integrity): "The Magnet URI scheme is a de facto standard defining a uniform resource identifier (URI) scheme for Magnet links, which mainly refer to resources available for download via peer-to-peer networks. Such a link typically identifies a file not by location, but by content—more precisely, by the content's cryptographic hash value. Since it specifies a file based on content or metadata, rather than by location, a Magnet link can be considered a kind of Uniform Resource Name, rather than the more common Uniform Resource Locators. Although it could be used for other applications, it is particularly useful in a peer-to-peer context, because it allows resources to be referred to without the need for a continuously available host." Github has a great explanation as well: http://githubengineering.com/subresource-integrity/ [1] http://www.w3.org/TR/SRI/ [2]…

If the last chart in that post[1] is anything to go by, then we will hopefully see some significant performance improvements from this. [1]: https://githubengineering.com/counting-objects/#the-results

This release incorporates "delta islands" which is code GitHub has been using since 2015-ish (or earlier?) to store repositories and their forks in the same object storage, with "islands" of objects as they diverge. See https://githubengineering.com/counting-objects/ for a related blogpost of theirs. This was work sponsored by GitLab, and I know they plan to use it in production sometime soon. So it's interesting for other similar hosting sites with a "fork" feature to look into it.

Github's on MySQL https://githubengineering.com/mysql-high-availability-at-github/

This may be true of Twitch but not GitHub. GitHub uses https://github.com/libgit2/rugged to bridge between Ruby and Git. As far as I'm aware, they still use Ruby to pull the diffs etc out of Git etc. https://githubengineering.com/how-we-made-diff-pages-3x-faster/ indicates they were still doing this as late as 2016.

You've got enough rebuttal on your doubt of Azure. But another issue with this comment is this: > in RackSpace and AWS As far as I can tell, they moved off RackSpace a long time ago. And they only use AWS infrastructure for peering, which makes sense with the amount of AWS machines that pull code from Github. The majority of the heavy lifting is done in Github's own data centers: https://githubengineering.com/evolution-of-our-data-centers/

For storing things other than git repos, GitHub is heavily invested in MySQL. AFAIK all of GitHub is hosted on their own hardware. https://githubengineering.com/mysql-high-availability-at-github/

The GitHub Engineering team announced the GitHub Load Balancer and intent to open source its components back in 2016 - https://githubengineering.com/introducing-glb/. And commits to https://github.com/github/glb-director began in May. I doubt Microsoft had anything to do with releasing this other than not blocking it.

The main competitor to CommonMark is GFM, which is now based on CommonMark. https://githubengineering.com/a-formal-spec-for-github-markdown/

In case you missed it, Github open sourced Licensed, a cross-language cross-(package manager) tool to find and store metadata about all licenses used in a git repo, to surface potential dependency problems. Official announcement here: https://githubengineering.com/improving-your-oss-dependency-workflow-w… Repo here: https://github.com/github/licensed

Google Analytics has a query parameter that can be used to extract arbitrary data. So make a request to that with an ID that you control I guess? GitHub mentions it in their CSP post: https://githubengineering.com/githubs-post-csp-journey/

> I feel like Jeff Atwood and his friends abandoned the Commonmark project that tried to create a universal specification of Markdown. Even if they did, I don't think that Github, Stackoverflow, Reddit and other major Markdown users would ever convert. The CommonMark project has not been abandoned; the CommonMark C parser was implemented on GitHub earlier this year and is used to render Markdown across all repositories: https://githubengineering.com/a-formal-spec-for-github-markdown/

I don't know what addresses they use but github engineering blog has a lot of examples of really useful contributions, e.g.: > Shortly after our initial deploy, we also started the process of upstreaming the changes to Git so the whole community could benefit from them. Source: https://githubengineering.com/counting-objects/ But I guess most of their git related work time goes to libgit2.

GitHub may not be the primary contributor to git, but they're certainly a significant user of it, and they _have_ contributed patches to upstream git. (example, the '768 conflict' bug here: https://githubengineering.com/move-fast/)

I wrote laboratory[1] after reading a blog post by github[2] and deciding it'd be useful in Python. Other than riding on the coattails on the popularity of the blog post and the original ruby library, I posted it to hacker news where it did fairly well. There've been a few external contributions and there are definitely some users, but I'm not too sure how many. I don't think it's the sort of project that has much ongoing development, so it's not something a community would form around. [1] https://github.com/joealcorn/laboratory [2] https://githubengineering.com/scientist/

They are referring to companies like GitHub [1] and Slack [2]. [1] https://githubengineering.com/how-four-native-developers-wrote-an-elec… [2] https://slack.engineering/growing-pains-migrating-slacks-desktop-app-t…

This is pretty generic -- isn't that how virtually any decent footprint site with physical infrastructure operates (at least from ~2000 onward)? I'd be more interested in blog posts about either what they were doing before which was different/worse and got upgraded, or how they decided on specific things. Their in-house tool gPanel is pretty cool, though (https://githubengineering.com/githubs-metal-cloud/) -- I've seen a lot of people build their own of that in ways which are far less sophisticated, or used one of the ~10 other tools for it. How much effort an org puts into making sure firmware is all the same is probably a good indicator of how competent they are, IMO.