Facebook Engineering

Well bookmark the following and be prepared. Because Facebook posts a lot about what they do. https://newsroom.fb.com https://code.facebook.com

Lots of insight—and several great projects—on their blog: https://code.facebook.com Things work a little different for Facebook than they do for sites with more reasonable traffic. I've seen a few talks on YouTube where they speak about the challenges of scalability as they gradually hot swap legacy X or Y for their own in-house component Z. See React/JSX and Flux, and the Haxl library. They go into detail about some of the problems and solutions that they've made public with respect to those projects, and they speak a lot about testing too in many of the posts. Worth checking out.

I don't really have any scathingly brilliant goals for the D version, other than inspiring others to make use of it. I still maintain it, it'll be much easier to maintain in D, and not much work to convert it to DasBetterC. Yes, it was Warp: https://github.com/facebookarchive/warp https://code.facebook.com/posts/476987592402291/under-the-hood-warp-a-… You're welcome! It's reactions like yours that make it worthwhile.

If there are engineers you highly respect on your team, their code is a great place to start. Otherwise, many top tech companies now open source software that they've written internally, oftentimes with their own websites. And there is also a growing trend for them to actually maintain the software they open source as opposed to just throwing it over the fence. Think of companies that have a strong engineering brand, and then just search for what open source software they're released. Pick whatever seems most aligned with your interests. Some examples: Google: https://opensource.google.com/projects/list/featured?language=python Facebook: https://code.facebook.com/projects/#backend Stripe: https://stripe.com/open-source Airbnb: http://airbnb.io/projects/

I love that projects like this (and this: https://code.facebook.com/posts/717010588413497/introducing-6-pack-the… exist and I love that they provide install instructions like this: http://opennetlinux.org/wedge ... but I also wish they provided a hardware parts list that I could easily order. Does anybody know of a site that provides that kind of hardware shopping list for these projects?

:( Facebook recently opened a datacenter in Fort Worth. [0] I heard something like 100k people are moving into the DFW metroplex a month? I haven't verified that but the real estate market is definitely crushing right now. It's only a matter of time. [0] https://code.facebook.com/posts/1014459531921764/facebook-in-fort-wort…

Seriously! Great news. My company is not just a software company, so when I proposed using React before (under the BSD+Patents license) it would have killed our ability to enforce our non-software related patents against fb. https://code.facebook.com/posts/300798627056246/relicensing-react-jest…

From https://code.facebook.com/posts/112130496157735/explaining-react-s-lic… > As our business has become successful, we've become a larger target for meritless patent litigation. This type of litigation can be extremely costly in terms of both resources and attention. It would have been easy for us to stop contributing to open source, or to do what some other large companies do and only release software that isn't used in our most successful products, but we decided to take a different approach. Never heard Facebook answer: why didn't they just follow Red Hat's model and make a "Patent Promise"[1]? Then accumulation of software patents would help deter meritless litigation while staying out of the way of FLOSS developers getting their work done. [1] https://www.redhat.com/en/about/patent-promise

https://code.facebook.com/posts/1601610310055392/introducing-openbmc-a… It could be an open solution too. I have only cursory followed OCP and other hardware advances, but this being used with Raptor's power servers. Opensparc before Oracle, and recently the RISC V community might have built something similar too.

Super interesting post. Would love to read more detail about their backup and restore infrastructure. If Tom and/or Shlomi are reading this: you mention taking multiple logical backups per day. What benefit does this bring versus just having one per day and doing a point-in-time restore using binlogs? Is this just a tradeoff between time taken for a restore and storage you're willing to dedicate to backups? Disclaimer: I work on Facebook's MySQL backup and restore system (https://code.facebook.com/posts/1007323976059780/continuous-mysql-back…

A good solution to this problem is to use an SSH Certificate Authority - then you need only configure the CA certificate on each box, and you can either issue semi-long-lived certificates to each user who needs access, or use something like Vault to issue short-lived certificates intended for one-time use. This model is described in an excellent post by Facebook from a while back [1]. (Disclaimer: I used to work at HashiCorp, and put this model into production there, though the Vault support for issuing short-lived certificates was added after I left) [1]: https://code.facebook.com/posts/365787980419535/scalable-and-secure-ac…

I think I can safely say it isn't used heavily in many production environments but it is used heavily in some. For example, one of Facebook's abuse systems, dealing with 1M requests per second is written in Haskell: https://code.facebook.com/posts/745068642270222/fighting-spam-with-has… As for particular places where it shines, I don't know of any in particular. I remember when taking my Declarative Programming course in university my lecturer loved to talk about an experiment the military did where they challenged a few teams to implement some kind of radar system in their chosen language and evaluated them based on time taken, correctness and code complexity (LOC) at the end. The Haskell team performed very very well. I can't find any details on this experiment though...

No, not really. Apart from the reply above about bitkeeper, I'm not aware of any new or old (d)vcs that does "sub-modules" well. There's AFAIK been some orgs working on mercurial in order to support large mono-repos - I don't know if it's better or worse than git today. I see that Illumos (opensolaris) have shifted from mercurial to git, for example: https://wiki.illumos.org/display/illumos/Mercurial+Workflow (Solaris/Sun was a somewhat early adopter of "big" mercurial repos). Working with big mono repos are not without challenges: https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-f… FWIW I tried using sub-modules for mercurial some years ago - in the end I realized it was probably not worth it. I'm also skeptical about go's use of "vendoring" dependencies for a similar reason.

I don't think it's such a stretch. From facebook's yarn announcement[1]: > or example, updating a minor version of babel generated an 800,000-line commit that was difficult to land and triggered lint rules for invalid utf8 byte sequences, windows line endings, non png-crushed images, and more. Merging changes to node_modules would often take engineers an entire day. I just did a fresh install in a new directory and ended up with 114M worth of dependencies, so I'm not entirely sure what the difference is. My point is. 50MB, 114MB, or 500MB worth of javascript dependencies is a massive footprint. It works and I'm relatively happy with what it does, but I don't see this as a stable, long term thing. [1]: https://code.facebook.com/posts/1840075619545360

My favorite thing about the post* that introduced Yarn, itself a rewrite of a basic component, is that the author(s) unironically began with "In the JavaScript community, engineers share hundreds of thousands of pieces of code so we can avoid rewriting basic components, libraries, or frameworks of our own." *https://code.facebook.com/posts/1840075619545360?_fb_noscript=1

> Flow, introduced 3 months after TypeScript Do you think Flow was built in 3 months as a reaction to Typescript? Not that it was a big enough problem that two different groups of people independently decided to try to solve it? EDIT: disregard this, Flow was introduced much longer than 3 months after TypeScript. The blog post introducing Flow mentions TypeScript, and is probably worth reading for background [1] > Nuclide instead of Atom/Sublime/VSCode Nuclide is an Atom plugin. > Jest instead of Jasmine/Mocha Jest provides a superset of Jasmine functionality, and uses Jasmine as its test runner. [1] https://code.facebook.com/posts/1505962329687926/flow-a-new-static-typ…

> Use kerberos auth with LDAP Do you know of a good guide for how to set this up on Linux? Or a good book that covers the topic? I've looked into this before and it's seemed to be an absolute jungle. I'd like to explore setting this up at my company but don't know where to start. I'd be looking for something that's comparably easy to understand and employ as Facebook's recent post about SSH with certificates: https://code.facebook.com/posts/365787980419535/scalable-and-secure-ac… What's the best straightforward way to set up LDAP, Kerberos, etc from scratch?

The best part is, facebook already has dev blogs: https://developers.facebook.com/blog/ https://code.facebook.com/posts/ No popups, sane column widths. Yet John is still cramming blog posts into Facebook user posts.

So curious what the other hardware is/does. Dan Luu wrote that Intel's cache allocation tech particularly helped Google, so they could run multiple workloads on one machine without totally trashing the larger caches at every context switch: http://danluu.com/intel-cat/ The Xeon D chips, which provide cheap low-clocked big cores and integrated NICs for small servers, were requested by Facebook: https://code.facebook.com/posts/1711485769063510/facebook-s-new-front-… Someone who follows server chips (maybe AnandTech?) called Xeon D's one of Intel's coolest product lines to come out recently. They're also arguably a smart way for Intel to seal off the lower end of the market against approaches with tons of cheap ARM or AMD cores.

React, Casandra, Flux, and tons more, Internet.org and now this? I can now feel a little less bad that every time I click like, that at least my personal information is supporting some great projects. https://code.facebook.com/projects/

I suppose it depends tremendously on the application you are developing but combining Haskell with things like Haxl[1] and opaleye-sot[2] can help simplify the database aspect of things tremendously. Once you've abstracted away the persistence and querying details, I think you get to the best part of FP which is modeling the actual behaviour/flow of data and types. In a typical app lifecycle Haskell starts shining more as time goes on and the app grows past CRUD and in size. [1] https://code.facebook.com/projects/854888367872565/haxl/ [2] http://ren.zone/articles/opaleye-sot

"Multi-billion" is irrelevant here because most people working for a large company don't have anywhere near that kind of budget. We're talking about two programmers who were allowed to work on a research project for a month. Also, Facebook releases lots of great open source software. Apparently you haven't heard of React? For more see https://code.facebook.com/projects/

One interesting aspect of developing for different platforms came from Facebook's React Team (https://code.facebook.com/posts/1014532261909640/react-native-bringing… "It's worth noting that we're not chasing “write once, run anywhere.” Different platforms have different looks, feels, and capabilities, and as such, we should still be developing discrete apps for each platform, but the same set of engineers should be able to build applications for whatever platform they choose, without needing to learn a fundamentally different set of technologies for each. We call this approach “learn once, write anywhere.”"

Self-plug. This is based off the Haystack paper: https://www.usenix.org/legacy/event/osdi10/tech/full_papers/Beaver.pdf from Facebook. Relevant blog-post is here: https://code.facebook.com/posts/685565858139515/needle-in-a-haystack-e… I gave a talk about Haystack at PWL: http://www.meetup.com/papers-we-love-too/events/220795812/ with the relevant video here: https://www.youtube.com/watch?v=EuNumdi1Do0

Like skybrian's reply mentioned, I don't think this is actually an area where Flow is much stronger than TypeScript, if at all. AFAIK, virtually all vanilla JavaScript should also readily compile as TypeScript. In my mind the difference is that TypeScript doesn't purport to analyze the correctness of regions of code that lack type annotations. Meanwhile, Facebook claims that "Flow understands the control flow through the program."[0] That's great in theory, and maybe their control flow analysis is really quite sophisticated, but ultimately isn't this isomorphic to the halting problem? [0]: https://code.facebook.com/posts/1505962329687926/flow-a-new-static-typ…

Both are great, but for a beginner, I too feel that Mercurial is much easier to get started. You don't have to spend any time 'understanding' how it works, just 3-4 commands is all that it takes to use it. > Mercurial will get a lower and lower percentage of the market place over time Lots of big projects continue to use Hg, some even switching to it from Git (e.g. FB) https://hg.python.org/ https://hg.mozilla.org/ https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-f…

Yes, Mercurial is being actively developed; Facebook has a great post about why Mercurial and not Git with data: https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-f…

Gabriel (as well as the others on the team) have definitely looked at these areas - if things were left out/not "featurized" it was likely done via an ablation test, or showed improvement over benchmarks, or maybe just to set a baseline, as he is quoted in the main article. I don't know what techniques they used here, but I am excited to find out! On the specific issue of encoding time-dependent behaviors in models, I think it is related to a broader issue that shows up in many application areas. To me the critical factor is that these models are ruthlessly good at exploiting local dependencies and totally forgetting long-term global dependencies or respecting required structure in control/generation. This basically means it is very difficult to train long-term, time dependent behavior without tricks (early/mid/late game models, extensive handcrafting of the inputs, or using high level "macro actions"). Indeed, FAIR's recent mini-RTS engine ELF directly gives macro actions, in part to look closer at how well global strategies are really handled and remove one factor of complexity [0]. Gabriel's PhD thesis was entirely on Bayesian models for RTS AI, applied to SC:BW [1], so I am sure he is well aware of the "classic/rules based" approaches for this. [0] https://code.facebook.com/posts/132985767285406/introducing-elf-an-ext… [1] http://emotion.inrialpes.fr/people/synnaeve/phdthesis/phdthesis.html

I've just finished reading https://code.facebook.com/posts/1716776591680069/react-16-a-look-insid… and it provided lots of good insights into how you managed to roll out the new release. Thanks for the interesting write-up! That blog post mentions your featureFlag approach, the initial dogfooding and then the gradual rollout. I'd be very interested to hear on top of that: 1. How did you approach the rewrite on the coding side? What I mean is: Did you just say "Okay, let's start from scratch and use the lessons learned from v1?" Or did you use a specific approach along along the lines of: "Let's structure our code based on these principles and ...?" 2. When you started the rewrite, how did you know in advance and make sure that the rewrite would be faster and have a smaller code-size? 3. Was your focus during the rewrite on a cleaner code base (and the performance improvements followed automatically) or was your focus on speed right from the start? 4. Any other lessons learned during the rewrite? Patterns and approaches that helped/didn't help during the coding?

I disagree. I think it was entirely related. Facebook could likely put up with a few dissenters here and there, but a major defector such as Wordpress is a canary: they likely feared that this would be the first domino to topple React's supremacy. FB: "...we know that many teams went through the process of selecting an alternative library to React. We're sorry for the churn. We don't expect to win these teams back by making this change, but we do want to leave the door open." This can seem like a response to the community at large, but can also be read as a direct response to Wordpress' comments: WP: Automattic will also use whatever we choose for Gutenberg to rewrite Calypso — that will take a lot longer, and Automattic still has no issue with the patents clause, but the long-term consistency with core is worth more than a short-term hit to Automattic’s business from a rewrite. Core WordPress updates go out to over a quarter of all websites, having them all inherit the patents clause isn’t something I’m comfortable with. Now if someone (i.e. an Actual, Impartial Lawyer) can explain if changing to MIT actually changes anything. EDIT: As others have pointed out, FB reaffirmed its decision to maintain the patents clause on Aug 18th (33 days ago): We recognize that we may lose some React community members because of this decision. [1] But they change their minds 8 days after the Matt published the On React and Wordpress article. This pretty much proves that the decision…