Facebook Engineering

It's a theme I like to read about (I mean "practical issues around ML in production settings"). I find company blogs and some research publications are great resources. Examples: - https://eng.uber.com/ - https://code.fb.com/ and many more. Google also publishes papers on various engineering practices obviously, some ML-related, but I can't find a blog where they focus on that specifically. Also it's not "to keep up to date", but there's a great paper (from Google) that's often cited: Machine Learning: The High Interest Credit Card of Technical Debt https://ai.google/research/pubs/pub43146 It talks about issues you face over the long run (I've experienced some of those). It also provides interesting pointers for further reading, e.g. about "pipeline jungles". If others have pointers, I'm curious to hear about them as well.

The GPU model is a proven way to get better performance out of traditional CPUs also. You will get the best performance from modern CPUs if you treat each SIMD lane as a parallel execution on homogeneous data and also execute on multiple cores. See ispc, and Burst Compiler as examples. I think there is a view that this style of programming only applies to traditionally high compute areas like games, hpc, rendering, ML, etc but we've recently seen a lot of core building blocks of "normal" web applications like hash tables (https://code.fb.com/developer-tools/f14/) and json parsers (https://github.com/lemire/simdjson) get massive perform gains from SIMD.

There is a Ruby port [1] publicly available. From the encoding routine [2], it seems to be an sRGB correction followed by a two-dimensional DCT. The first two bytes encode parameters---the size of the output matrix n x m and the quantization scale used for ACs; next 4 bytes encode the DC, then (nm-1) quantized ACs (2 bytes per AC) follow. The comparison to Facebook's similar solution [3] would be interesting (but I realize that Blurhash tries to hide explicit images, so technically they have different goals). [1] https://github.com/Gargron/blurhash/ [2] https://github.com/Gargron/blurhash/blob/master/ext/blurhash/encode.c [3] https://code.fb.com/android/the-technology-behind-preview-photos/

I'm mostly guessing based on what I've read over the years. They've published a hefty corpus of work regarding their deployment infrastructure and greater code review/quality approach. Here's a few examples: - https://code.fb.com/web/rapid-release-at-massive-scale/. - https://www.quora.com/How-does-Facebook-release-deploy-process-work-Wh…

Facebook has a few Haskell hackers. I imagine they need a few more to combat not only spam but fake accounts, fake news and fake viral posts. Huge opportunity right there. I don't work at Facebook but follow their tech blog. https://code.fb.com/security/fighting-spam-with-haskell/

Even among the top websites, there are a lot of them that don't even do dual stack. Let alone IPv6 only. (e.g. Twitter) Also, only using IPv6 is cutting out 75% of the world population, according to Google. https://www.google.com/intl/en/ipv6/statistics.html However, public-facing dual stack but internal IPv6 only is done at e.g. Facebook IIRC (see https://code.fb.com/?s=ipv6)

I think static typing is needed in a low, middle level programming language to help the optimizer, and make sure values doesn't go out of bounds. Fanny enough higher level languages like JavaScript (and probably python to, I don't know Python) are strictly typed eg. you can never go out of bound and get overflow errors, have dangling pointers, etc. When you already have strict typing, there's less need for static typing. If the statically typed language is compiled to a dynamic language, you will lose most of the performance gains too, for example Dart is almost like JavaScript-with-static-typing, but it's twice as fast because the compiler can actually make use of the static types. I also think static typing bolted on to a dynamic higher level languages do push you into an object oriented paradigm. And I think the language becomes uglier with annotations. And it makes people lazy so they name things x, y, z because the type annotation fills in the rest. But who cares about syntactic preferences, code readability and naming things :) The real reason you use static typing is to catch bugs ... But the best way to catch bugs is to understand the code, and have many people read it. When I see less good programmers the workflow goes something like this: -I want to do foo ... autocomplete ... fooBaz, fooBar, fooFoo ... hmm, what can I do with foo ... autocomplete ... foo.bar sounds like what I need, it takes a baz, new Baz() ... damnit how do I get rid of this null exception…

First of all, the Facebook app is literally an entire operating system unto itself, among other things. Like someone pointed out, there's an entire Craigslist, an entire Snapchat, an entire Yelp, an entire YouTube, an entire web browser and so much more stuffed into a single app. On top of that, there are probably thousands of A/B tests and divergent code paths that are running simultaneously. Even if you have a feature, you probably have the code to switch between 10 different variants of that feature. There's also complex real time user instrumentation, media storage and backup features, tons of custom animation frameworks, networking frameworks, caching frameworks and on and on and on. The level of complexity and the millions of man-hours worth of code backing that app is insane. At this point, it shouldn't be surprising in the slightest that the app is a cool ~300megs. Could it be better? Sure, and I'm sure they're working on it. After all, look at their developing country strategy and the stuff they do with "Facebook For Every Phone" to shave off every last byte from their apps: https://code.fb.com/android/how-we-built-facebook-lite-for-every-andro… If you look at the incentives, it's NOT in their best interests to make updates hard. That's not how they make money and in developing countries it actively costs them business. It's not that they're intentionally bloating the app just to mess with you....it's just that there's really that much going on…

Good article on how Facebook used them for their mobile app: https://code.fb.com/android/improving-facebook-s-performance-on-androi…

And there are existing userspace daemons; see: https://code.fb.com/production-engineering/oomd/

I wonder why they chose it over facebook’s 8 bit posit: https://code.fb.com/ai-research/floating-point-math/

All Facebook services are IPv6 enabled by default and their internal network is IPv6 only. https://code.fb.com/production-engineering/legacy-support-on-ipv6-only…

Here is the full post on Facebook’s Engineering blog: https://code.fb.com/android/hermes/

One annoying part about floats is the abuse of the NaN space. Some runtimes like SpiderMonkey, JSC, and LuaJIT abuse the NaN space to store pointers in doubles. This practice is often called nan-boxing, but has a few variants. A more efficient use of bits like with Posits will break this. As for the claim about a FPU taking up less space and power for posits than floats, facebook made that claim: https://code.fb.com/ai-research/floating-point-math/

You'd be surprised if the largest content delivery platform in the world cares about being accessible to the broadest range of users? https://code.fb.com/ios/under-the-hood-building-accessibility-tools-fo…

Here's the announcement, even: https://code.fb.com/ios/under-the-hood-building-accessibility-tools-fo…

As in, there are a handful of large projects the languages are used for. For Haskell, Facebook's spam detection system, Sigma[1], comes to mind, along with some use by banks (Standard Chartered); then there's a bunch of smaller places using Haskell (Wire, a secure messaging app, like Signal; Galois, a US defense contractor doing software verification and related things) plus some open-source tools like pandoc. I know less about OCaml, but at least Jane Street is using it. It's somewhat niche, but it's not like it's only used for hobby or toy applications. [1] https://code.fb.com/security/fighting-spam-with-haskell/

>Why would anybody do that ? Because they have good reasons to?[1][2][3] There really is more to version control than Git. [1]https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billion… [2]https://www.perforce.com/9-10-top-semiconductor-companies-use-perforce [3]https://code.fb.com/core-data/scaling-mercurial-at-facebook/

> Symbolic execution is also known as abstract interpretation These aren't the same thing! They do both use abstraction of concrete values, but abstract interpretation solves for fixpoints in a lattice of abstract values, which is a fairly different process from how symbolic execution works. The answers here go into more detail: https://cstheory.stackexchange.com/questions/19708/symbolic-execution-… One way of digging into abstract interpretation non-theoretically is via this recently open-sourced framework: https://code.fb.com/open-source/sparta/

For a bit more context, see this blogpost we made: https://code.fb.com/open-source/sparta/

> Lots of support for monorepos – indeed, this is the main reason that Google, Facebook, and Mozilla are all pouring work into hg and are using it. They are improving mercurial[1], but are they using it ? [1] https://code.fb.com/core-data/scaling-mercurial-at-facebook/

Is it in maintenance mode? Chris Mason (the main btrfs developer) works for Facebook now and Facebook uses btrfs on "millions of servers" (source: https://code.fb.com/open-source/linux/).

Nope. It's all 100GbE throughout as far as I know. And people do work really hard to be able to saturate that bandwidth as it is by no means a trivial task to saturate it through the usual, naive means without the use of RDMA and Verbs. Years ago when I was there it was (IIRC) 40Gbps to each node straight up. It's a necessity really. All storage at Google has been remote and distributed for at least the past decade. That puts serious demands on network throughput if you want your CPUs to actually do work and not just sit there and wait for data. Here's some detail as of 2012: https://storage.googleapis.com/pub-tools-public-publication-data/pdf/4… Note that host speed is 40Gbps. And here's FB talking about migrating from 40Gbps to 100Gbps in 2016: https://code.fb.com/data-center-engineering/introducing-backpack-our-s…

It seems facebook built tooling to address that issue. I doubt they would tolerate that lifestyle either: https://code.fb.com/web/rapid-release-at-massive-scale/

I agree with the spirit of what you're saying. Too often I've seen folks on HN dismiss languages because they didn't have features X or Y, usually with a condescending "maybe this language's creators should read some PL theory". I remember circa 2013 or 2014, it became impossible to read threads related to Go because the entire thread would always be "a Real Language(TM) needs Generics". Look at the success Go has seen in the last decade while doing the opposite of what HN thought was correct. That said, there do exist useful products written in Haskell. For example, Facebook writes a lot of spam fighting code in Haskell. https://code.fb.com/security/fighting-spam-with-haskell/

Link to blog post: https://code.fb.com/ai-research/pytorch-developer-ecosystem-expands-1-…

If you take a look at the announcement post for the framework, you can see some discussion about that: https://code.fb.com/ml-applications/horizon/ One application that they mention related (at least loosely) to social connections is deciding whether or not to deliver a notification to a user.

What I find fascinating is that NLP tools today can learn to translate between two languages even without a dictionary or a parallel corpus. That says something about the relations that exists between languages. https://code.fb.com/ai-research/unsupervised-machine-translation-a-nov…

ACLU is unaware that FB has an entire Haskell library for snooping on messages in the name of SPAM detection? https://code.fb.com/security/fighting-spam-with-haskell/

Nice to see, this has felt like a gap in cloud offerings for a while... and the open source options have difficulties. From the little that was said, going to guess this uses something like Beringei (https://code.fb.com/core-data/beringei-a-high-performance-time-series-… under the hood