Flipboard Engineering

Alias tables are pretty cool, I wrote an interactive visualization of how they get built as part of this post a while ago: http://engineering.flipboard.com/2017/02/storyclustering . We used alias tables with MCMC and the hastings metropolis test to build a super fast LDA. Also worth reading up on are sum-heaps. Alias tables are O(1) to sample from but O(n) to build/modify. Sum-heaps let you modify in O(log(n)) at the cost of sampling in O(log(n)) as well. A good writeup is here: https://timvieira.github.io/blog/post/2016/11/21/heaps-for-incremental…

Great use case, upscaling print thumbnails. Norman Tasfi made a Neural Net upscaler for Flipoard http://engineering.flipboard.com/2015/05/scaling-convnets/ I expect video upscaling next.

Having worked on what I would classify as a very large and complicated mobile/desktop web app for a few years now, I would probably choose a different direction if I had access to enough developers. I'm fairly convinced at this point that with enough developers, the best approach is adaptive if you plan on building a web app with considerable complexity. Completely isolated development, fully custom design and code for mobile. With an adaptive site you have the choice to include or exclude elements which may have a major performance impact on the app, including for instance: entire view/component trees, media queries, heavy JS like graphs or ads. You also get to make web-unfriendly decisions like using a custom renderer if you think it will produce a higher quality experience, like Flipboard's React Canvas from several years ago: https://engineering.flipboard.com/2015/02/mobile-web Overall it's still very context dependent, but I would recommend for larger teams to not completely discount adaptive mobile sites if they're looking to build the highest quality mobile experiences.

Kinda reminds me of Flipboard https://engineering.flipboard.com/2015/02/mobile-web (2015)

That demonstrates that you can use WebAssembly to make a snappy UI, it does not demonstrate that you must use WebAssembly. It's nearly ten years old now but I remember being absolutely blown away by React Canvas, a UI toolkit that leveraged React but instead of rendering DOM nodes it rendered to a <canvas/> tag. Beautiful, 60fps stuff. All written in JS. Unfortunately all the demos seem to have disappeared since but here's a blog post about it: https://engineering.flipboard.com/2015/02/mobile-web Point is, both Makepad and React Canvas have something in common: they ditched the DOM. There are both advantages and disadvantages to doing so but the relevant point is that you don't need to use WebAssembly to do it.

The mention of accessibility at the end reminds me of this very celebrated post from 2015: https://engineering.flipboard.com/2015/02/mobile-web. The article described Flipboard's migration from a browser DOM-based app to an entirely canvas-rendered one, which made the whole thing invisible to screenreaders. There was just a throwaway line at the end on accesibility: "This area needs further exploration."

An example to prove your point: Flipboard determined that our current rendering engine was too slow for their needs and implemented their own rendering engine on top of ours while retaining the rest of the browser infrastructure. [0] https://engineering.flipboard.com/2015/02/mobile-web

Because of browser support honestly. Outside of grid-layout, there is nothing salvageable from the CSS layout engine that cannot be done better inside a programmable view such a canvas. Give it 5 years or so for WebGL/WebGPU to standardize, and new UI libraries with a better layout engine and UI constructs far better than what can be built on top of HTML/CSS will show up. Nobody thought we would take HTML/CSS as far as we have, but it has already served its purpose. Your comment reminds me of this article back then: https://engineering.flipboard.com/2015/02/mobile-web

WebAssembly does nothing for for CSS or DOM. If WebAssembly would change any of this we would just start using `<canvas>` five years ago like flipboard did for a while: https://engineering.flipboard.com/2015/02/mobile-web The Web, when properly used (and perhaps also when it's not) is much, much more powerful and useful than just painting on a screen and running logic underneath.

like Flipboard created? React Canvas rendered the site directly to canvas. https://engineering.flipboard.com/2015/02/mobile-web

Flipboard built their mobile web app in canvas: http://engineering.flipboard.com/2015/02/mobile-web

If it can be interfered from words alone, try Latent Dirichlet Allocation (e.g. with http://radimrehurek.com/gensim/) to generate tags. Some sources: * http://blog.echen.me/2011/08/22/introduction-to-latent-dirichlet-alloc… * http://alexperrier.github.io/jekyll/update/2015/09/04/topic-modeling-o… * http://engineering.flipboard.com/2017/02/storyclustering Alternatively, if you know tags, just want to see which are similar to each other, methods like word2vec should help, vide: * http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html

It's really cool to see a writeup for this on real-life images. Similar work at Pinterest http://engineering.flipboard.com/2015/05/scaling-convnets/, for Anime scaling https://github.com/nagadomi/waifu2x and for sprite scaling (can't find the reference I'm thinking of). This tech has been around for several years, and some variation was presented in concert with the Boston Marathon investigation. https://arstechnica.com/information-technology/2013/05/hallucinating-a… (Not clear if this was used as part of the investigation, or if it could be used for future investigations)

This was used by Flipboard for their Duplo algorithm which creates plausible layouts for web content. They used Perlin Noise to create variations between pages to create an 'organic' sense to the algorithmically generated layouts. http://engineering.flipboard.com/2014/03/web-layouts/

I'm looking for a technique that can upscale images nicely (using NNs). I've found this: [1], but the results seem somewhat disappointing. One of the problems is that the quality measures are (in my case) subjective (the results should look convincing but need not be "perfect", whatever that may mean). [1] http://engineering.flipboard.com/2015/05/scaling-convnets/

Flipboard Engineering published a blog post on solving this problem http://engineering.flipboard.com/2014/10/summarization/

Here is one example: http://engineering.flipboard.com/2015/02/mobile-web/

People are using the canvas now to achieve fast speeds on mobile web. See Flipboard's post on this [1]. The big issue ppl brought up was that using canvas instead of DOM meant losing accessibility. Btw Flipboard released an interesting framework called React-Canvas[2] based on their efforts. 1. http://engineering.flipboard.com/2015/02/mobile-web/ 2. https://github.com/Flipboard/react-canvas

Take a look at what Flipboard did recently, with their whole mobile site built in JS and canvas http://engineering.flipboard.com/2015/02/mobile-web/

I think this is a very good question, and the short answer in my opinion is yes, canvas rendering would make it much faster (as the guys at Flipboard have shown [1]). The downside is you would have to implement your own rendering engine from scratch which is quite challenging. [1] http://engineering.flipboard.com/2015/02/mobile-web/

Some examples of algorithmic design for page layouts: http://engineering.flipboard.com/2014/03/web-layouts/ and https://thegrid.io/ (disclaimer: I work on the latter)

More info: http://www.theverge.com/2015/2/10/8008905/flipboard-comes-to-the-web http://engineering.flipboard.com/2015/02/mobile-web/

From their blogpost[1] "Head on over to flipboard.com on your phone to see what we’ve built, or if you don’t have a Flipboard account, check out a couple[2] of magazines[3] to get a taste of Flipboard on the web." I was rather impressed with flipboard.com on my mobile. Although it seemed to hit a jank on a few scrolls and I wonder if there are UX implications for not being able to highlight and copy some of the text. Regardless very cool! [1]http://engineering.flipboard.com/2015/02/mobile-web/ [2]https://flipboard.com/@flipboard/flipboard-picks-8a1uu7ngz [3]https://flipboard.com/@flipboard/ten-for-today-k6ln1khuz

React Canvas author here. I outlined some of the reasons why it is not always the right answer: http://engineering.flipboard.com/2015/02/mobile-web/#PracticalApplicat… On Flipboard we use DOM where appropriate for text input and areas of the application where performance is less critical. And to your point about accessibility: https://github.com/flipboard/react-canvas#accessibility You are correct that the DOM just works. But we need a better option on the web in order to achieve the kinds of experiences that people have come to expect from native applications. The hope is that by pushing the browser beyond its limits that we can make progress in this area.

Agreed -- squeezing these native app elements into the document-based paradigm of the web is a horrid, Frankenstein's monster of an idea. It's like doing app development in Excel. If native web apps are to be more of a universal thing, I believe there ought to be a blank-canvas "meta-browser" layer that sits above the browser and that all web apps (including today's browsers) are built on top of. Basically a lightweight, sandboxed pseudo-OS that offers a robust standard library, a URL scheme and easy networking support, some sort of bytecode, maybe a UI toolkit, etc. Web apps would still take up the same amount of space and would still be able to run without installing, but they would now be endowed with native performance, app-specific features, and a consistent, functional UI. (Quiz: how often do your back/forward buttons fail when using, for example, your bank's website? The fact that these two ubiquitous controls simply break the web more often than not should be telling.) Shoehorning all that stuff into a hyperlinked, navigable document browser is insanity, and you can always feel it unless your web-app has basically reimplemented the DOM from scratch[1]. The web is currently layered upside-down and I think web apps won't lose their reputation until this is fixed. [1]: http://engineering.flipboard.com/2015/02/mobile-web

You can! The problem in that case is it's not a convolution (blur) per se. Each pixel is an average of subpixels P(1)=S1(1)+S2(1)+...+Sn(1), P(2)=S1(2)+...+Sn(2) -- but as you can see there are no elements in common between pixels, which is the case with convolution. That is, in the deconvolution case there are as many variables as unknowns, whereas in the upscaling case you're creating unknowns out of a single variable. To estimate those subpixels then you're going to be forced to make additional assumptions; if you assume they are independent you would simply estimate Sk(j)=P(j). The traditional (easiest) assumption is that the image is somewhat "bandlimited" -- it does not have many variations (frequencies) faster than once per pixel (so no equal or faster than subpixel variation). If this were the case, you could reconstruct the subpixels perfectly [1], save for some noise. But this is not always the case (and fails spectacularly when you have edges), resulting in upscale blur. So simple linear upscaling algorithms by reaching a compromise between blur and edge enhancing. If you want to do better though, you have to use non-linear kernels and have good underlying models for your image content. A promising approach is to use machine learning/NNs: http://engineering.flipboard.com/2015/05/scaling-convnets/ [1] https://en.wikipedia.org/wiki/Nyquist%E2%80%93Shannon_sampling_theorem

The most important thing about React's update performance is that it's ~O(n) of the size of the virtual dom. If React's performance is a problem you can window the input data, do things more efficiently in your render, tweak shouldComponentUpdate. The goal isn't to be as fast as possible but rather to be fast enough and for non-mobile you can hit that target with only the occasional tweak. For mobile, you can get as exciting as you like[1] but I've never had to go beyond the above steps. [1] http://engineering.flipboard.com/2015/02/mobile-web/ You can build things that are faster than React in benchmarks. Pretty much every vdom based library/framework/language is faster than React in benchmarks and absolute performance has never been a stated goal of the project. The framework does help you out in ways that don't show up on benchmarks like this. An example would be batching DOM updates, which is much harder when you have components mucking around directly with their internal set of nodes. I also favor using an immutable data tree with a corresponding shouldComponentUpdate and only rendering on rAF but those force architectural decisions on the project.

React (outside of React Native) is being used on many high traffic websites. - Facebook comments, chat, search, ads creation, photo upload, tablet website... and basically every new product being written. - Entire Instagram website - Yahoo Mail: http://yahooeng.tumblr.com/post/101682875656/evolving-yahoo-mail - A lot of the Khan Academy front-end: http://khan.github.io/react-components/ - Netflix website and TV experiences: http://conf.reactjs.com/schedule.html#beyond-the-dom-how-netflix-plans… - Many Airbnb new products: http://www.slideshare.net/spikebrehm/the-evolution-of-airbnbs-frontend - Flipboard mobile website: http://engineering.flipboard.com/2015/02/mobile-web/ - ...

I tried submitting a reddit thread (https://www.reddit.com/r/reactjs/comments/474odi/what_about_reactcanva… but it didn't get much traction... I think because the videos got pulled. I hope you guys don't mind if I ask this question here, and I apologize if I'm being naive in some way: Aditya Punjani's later Flipkart talk (https://youtu.be/m2tvYGCdOzs) mentioned the importance of mobile web apps and 60 FPS. Ben Alpert's talk mentioned (https://youtu.be/-RJf2jYzs8A?t=1378) "React Native for the web" would be awesome. There was even a lightning talk on react-native-web which attempts to do this using the traditional DOM (https://www.youtube.com/watch?v=RBg2_uQE4KM) Whatever happened to Flipboard's React Canvas (http://engineering.flipboard.com/2015/02/mobile-web/ and https://github.com/Flipboard/react-canvas) being a solution to these problems? There's pretty good browser availability and although there's currently no perfect solutions for things like accessibility, SEO, or server-side rendering... I failed to se a real "nail in the coffin". Did it just never gather enough momentum?