Nervana Blog

Some people are noting that TensorFlow evens the playing field, allowing other companies to poach Google's employees. Whilst that is true to some light degree, there are amazingly large caveats - Google still have the major advantage when it comes to poaching. The other important note is that open sourcing TensorFlow is likely more important for attracting recent graduates in machine learning than it is for attracting employees of other companies. Most machine learning engineers and researchers get excited by two things: underlying hardware and underlying tooling. Google have enough GPUs to throw 50 K80s for a few weeks into the hands of a single researcher for a single paper[1]. That's impressive. It gets even more impressive considering Google are making custom made ASICs that are more efficient - both power and computation - than GPUs too. None of that hardware is open source and even if it was, I don't see anyone getting them into general production in the near future. Nervana[2] are the closest to custom hardware I know about yet still have many steps until production. TensorFlow is also just the beginnings of the tooling Google has internally. The original TensorFlow whitepaper talks about their internal distributed training stack. A tiny recreation of this has reached the wild but it misses many of the interesting optimizations. (1) Automated efficient allocation of tensor operations to components (does this op go on GPU? CPU? Across the network to another GPU…

Not a general purpose CPU/GPU, but related: http://www.nervanasys.com/ Founded by experts in machine learning, neuroscience, processor architecture, and chip design, Nervana Systems is bringing unprecedented scale and simplicity to the application of brain-inspired algorithms. Deep learning has emerged as the leading strategy for making sense of a wide variety of data but is very computationally intensive. We are developing a scalable hardware solution to solve these types of problems. By making unsupervised learning dramatically faster and more scalable, customers at all levels can derive meaningful insights from their data, an ability previously available only to an elite few companies.

To be fair, the goal of those new computing architectures is not to advance algorithms. That at best is a side benefit. Those algorithmic performance improvements seen in DNN have come from improved datasets and training. Numenta and IBM arent focusing on training AFAIK. Google's Quantum Annealing [0] is the only hardware I'm aware of focused on training, although there are rumors Nervana Systems may produce something [1]. I'm sure there are others; accelerating training of DNN isnt a particularly new idea. The goal of these other computing architectures is typically to provide lower power, higher frequency/lower latency, or smaller form factor execution of trained models, but there is a question of how much value they can provide over more conventional chips to be worth the chip design costs. However without these architectures becoming as mainstream as say a GPU, I think we will continue to see advances come from the typical everyday computer. The ML community seems to be much more democratic than others. [0] http://googleresearch.blogspot.com/2015/12/when-can-quantum-annealing-… [1] http://www.nervanasys.com/about/

...later found this nice explanation of RL concepts if it helps someone: https://www.nervanasys.com/demystifying-deep-reinforcement-learning/

They aren't rebranded x86 cores, This is from Nervana systems which was acquired by Intel few months back. More details about Nervana architecture. https://www.nervanasys.com/nervana-engine-delivers-deep-learning-at-lu…

Check out the Winograd optimisations used in Nervana's neon - very fast https://www.nervanasys.com/winograd-2/

> simplified memory architecture (no cache?) No automatic cache hierarchy. There is local memory, which you manage explicitly. More info here: https://www.nervanasys.com/nervana-engine-delivers-deep-learning-at-lu…

If one doesn't want to dive deep in RL, and just want to start with recent methods then https://www.nervanasys.com/demystifying-deep-reinforcement-learning/ blog posts provide a very good starting point.

Nervana Systems http://www.nervanasys.com/products/ is using Titan X's for their cloud.

Part two: http://www.nervanasys.com/deep-reinforcement-learning-with-neon/

Intel | Compiler/Software Engineers | San Diego, Santa Clara, San Francisco, Folsom OR | REMOTE/VISA for strong candidates. Full-time. Currently in deep learning, if one wants to implement a low level kernel (LSTM, batch-norm etc), you're either going to be writing CUDA or device specific low level code. Join us as we grow the Intel Nervana Graph project (https://www.nervanasys.com/intel-nervana-graph-preview-release/) the deep learning equivalent of LLVM: an open source deep learning compiler and IR. We plan on pushing the boundaries of tensor optimizations and enable the same benefits that the programming language and compiler community have reaped from LLVM. Required: C++, modern software engineering, at_least_3(humility, curiosity, drive, on_the_right_side_of_Dunning_Kruger_effect). Desired: Python, low level optimization, numerical linear algebra, deep learning/ML http://jobs.intel.com/ListJobs/All/Search/jobtitle/computational-graph… Feel free to copy me directly at [email protected] too.