Andrej Karpathy

More users -> more data -> better product -> more users s/b More users -> more labeled data -> better product -> more users ML systems need lots of labeled data, not just lots of data. This is one of the primary why game playing AI's have had such great successes, tons of labeled data are relatively cheap. Great discourse on this and other related issues here: https://medium.com/@karpathy/alphago-in-context-c47718cb95a5

I wish we could come up with a catchier name, but I LOVE the idea of calling this programming, because that is precisely what we do when we compose deep neural nets. For example, here's how you compose a neural net consisting of two "dense" layers (linear transformations), using Keras's functional API, and then apply these two layers to some tensor x to obtain a tensor y: f = Dense(n) g = Dense(n) y = f(g(x)) This looks, smells, and tastes like programming (in this case with a strong functional flavor), doesn't it? Imagine how interesting things will get once we have nice facilities for composing large, complex applications made up of lots of components and subcomponents that are differentiable, both independently and end-to-end. Andrej Karpathy has a great post about this: https://medium.com/@karpathy/software-2-0-a64152b37c35

A couple quick things I can think of: - Voice transcription - Tesla Autopilot - Facial recognition (photo sorting on iphones, better photos) - Better graphical performance on Nvidia cards (https://developer.nvidia.com/dlss), also better compression for streaming. - Much better translation - Colorizing and repairing old photos - Visual recognition allowing better search of images I’m sure there are some I left out. I think we’ll see a lot more interesting applications (particularly around tooling) in the next few years. https://medium.com/@karpathy/software-2-0-a64152b37c35 Outside of the consumer space, there are also things that hint at more generalizable intelligence. Check out GPT-3’s performance on arithmetic tasks in the original paper (https://arxiv.org/abs/2005.14165) Pages: 21-23, 63 Which shows some generality, the best way to accurately predict an arithmetic answer is to deduce how the mathematical rules work. That paper shows some evidence of that and that’s just from a relatively dumb predict what comes next model. It’s hard to predict timelines for this kind of thing, and people are notoriously bad at it. Nobody would have predicted the results we’re seeing today in 2010. What would you expect to see in the years leading up to AGI? Does what we’re seeing look like failure? https://deepmind.com/blog/article/muzero-mastering-go-chess-shogi-and-…

I'm suggesting that you take an existing system and build up a corpus of request/response pairs. Then you use the LSTM to build a prediction model so that given a request it will tell you that the current production system will produce the following sql statement and this http response. Once the LSTM's output is indistinguishable from your current production system , for all use cases, then you replace the production system with the LSTM and a thin layer that can listen on the port, encode/decode the data, and issue sql queries. Why would I want to do this? I'm not 100% sure ... I think it would be super fast once you got it working. I think it would avoid many security bugs. You wouldn't have to read that "oh drupal 3.x has 20 new security bugs" better go patch our code. I think when I had this idea I was thinking about it terms of a parallel system that could catch hacking by noting when actual http responses diverged too much from the predicted response. The main idea being that for a given input the output really is 100% predictable, assuming your app doesn't use random numbers like in a game or something. To link this idea to the article, I think things like XML parsers could be written this way .... I can't prove it but I suspect that they would be very fast and not come with all the baggage that the article complains about. I started thinking along these lines after reading stuff like this https://medium.com/@karpathy/software-2-0-a64152b37c35

There are so many frameworks, tools and layers of abstraction out there that I find it helps to dip down a layer and taste what life is like without framework or tool X. Ideally, it's quite painful and you immediately appreciate what the tool is getting for you. You also might learn that the tool would be very helpful for circumstances you do not currently face (e.g large scale software engineering when you are exploring via a side project), and hang out in the lower layer until further motivated to climb back up. Some concrete examples: - you are using THREE.js. Try out raw webgl to draw a square. Run frighted back to THREE.js' arms and have some appreciation for what it is doing for you - you are new to web development. Try working with the DOM directly. Then try something like knockout or angular that helps track dependencies between your view model and the dom. Then try out React. React has achieved wide-spread acceptance for a reason, I think it's the best pattern. But it's also the most abstract from the reality of what's happening underneath the hood, so taking steps up to the top can really help - you are playing with scikit-learn and throwing various models at a classification problem. try implementing basic logistic regression using only numpy arrays. Try to understand how you could adapt your implementation `fit`, `transform` and `predict` interface used by sklearn. A couple of recent articles that show this habit in…

"PyTorch is better for research" is a weird, unsubstantiated statement. The fact is that few serious researchers use PyTorch (and even those complain about it). It's mostly grad students in a handful of labs. The only researchers I know who use PyTorch have been from FaceBook, and that's because they were implicitly forced to use it (PyTorch is developed by FaceBook). According to https://medium.com/@karpathy/icml-accepted-papers-institution-stats-ba… , 3 of the top research labs in the world are DeepMind, Google Brain (and the rest of Google), and Microsoft Research. Let's see: * DeepMind: TensorFlow * Google Brain: TensorFlow * Microsoft Research: CNTK Ok, so what about academia? The top deep learning groups in academia are: * Montreal: Theano * Toronto: TensorFlow * IDSIA: TensorFlow So, what about the greater academic research community? Maybe we could get some data about who uses what by looking at the frameworks cited by researchers in their papers. Andrej did that: it's mainly TensorFlow and Caffe. https://medium.com/@karpathy/a-peek-at-trends-in-machine-learning-ab8a…

(The comment below incorporates an edited version of my comment from last week's thread.) With human involvement only at the meta level, deep understanding of the generated implementations becomes more challenging and, in highly complex domains, perhaps impossible. See AutoML's generated architecture and compare that with a human-designed one. [1] [2] A future advance could allow the machines to automatically pick and learn to do new tasks that are helpful to accomplish a given high-level mission. Therefore, chances of unintended consequences become much greater. The major issue is, without a moral core that closely aligns with humanity's evolved morality, there will be moves that advanced AIs come up with that we deem abhorrent, and sometimes unforeseeable, yet they perform them innocently and we only find out the consequences once it is too late. Microsoft's Tay tweet debacle was fairly inconsequential [3]. But AI is becoming more powerful and granted more and more authority over the real world we live in, can we afford the technology without humanity? Developing a moral core should be prioritized. [1] Google Blog on AutoML https://research.googleblog.com/2017/05/using-machine-learning-to-expl… [2] Many tasks in the real world are more complex than Go and even expert humans are not capable of understanding all possible complex interactions on the Go board. See: AlphaGo, in context by Andrej Karpathy…

Humanloop | Infrastructure for AI | Backed by YC and Index | London + Remote Hiring - Software Engineers: front-end specialist - Machine Learning Engineer - Interaction designer (see jobs.humanloop.com for full details) We're a team of ML researchers and Engineers who've worked at Google, Amazon and Microsoft research on some of the biggest ML projects out there. ML and deep learning are a new software paradigm that needs new tools. We're building a platform for Human-in-the-loop ML that drastically reduces data needs and accelerates time to deployment. In the future people will program by teaching and curating datasets. (https://medium.com/@karpathy/software-2-0-a64152b37c35), we're making software 2.0 possible. Team: humanloop.com/about Contact the founders at [email protected]

Somehow remind me of Software 2.0[1] [1] https://medium.com/@karpathy/software-2-0-a64152b37c35

If you can write code to do a job, you probably should. ML makes sense for places where you can't write the code. We know many applications where this is the case: people tried for decades to write good natural language translation and image recognition, but ML does much better. A good overview of how to build real systems with ML is at https://medium.com/@karpathy/software-2-0-a64152b37c35

I mean in th sense of the sensitivity of the backpropagation updates to the loss function value. Backpropagation is essentially a big chain rule application. If you order the layers into a list (with identity transformations to pipe the values that do not change in the particular layer) then the whole process is just multiplying the derivative matrices, starting with the loss function and then the bottom-most layer. That's why it's difficult to train long unrolled sequences of RNNs - the many matrices that you have to multiply on the way back are either contracting (in which case the signal dies out) or expanding (in which case you get NaNs). That's also why people "cheat" backpropagation by using things like gradient clipping. https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06e…

Andrej Karpathy has a nice blog post on this: https://medium.com/@karpathy/software-2-0-a64152b37c35 The idea is that more and more software development will be "deleting" code, and re-writing that with neural networks.

The idea is that in the not-too-distant-future, differentiable programming will become much more mainstream. c.f. https://medium.com/@karpathy/software-2-0-a64152b37c35

In case someone doesn't believe my observations from conferences. Just look below. Neither INRIA nor EP are in the top 30 institutions with publications at ICML and only INRIA is in the top 50 (at 14, not bad!) at NIPS. EP isn't in the top 50 institutions with papers at NIPS. https://medium.com/@karpathy/icml-accepted-papers-institution-stats-ba… https://medium.com/machine-learning-in-practice/nips-accepted-papers-s… EP isn't a player in academic CS basically. (the numbers work out to be the same if you attend CVPR or ICRA or any number of other conferences).

He introduced it himself end of last year in this article: https://medium.com/@karpathy/software-2-0-a64152b37c35

Related blog post from last November: https://medium.com/@karpathy/software-2-0-a64152b37c35

Repos, no. But, no coincidence, I have open two relevant tabs right now: "Software 2.0", Andrej Karpathy https://medium.com/@karpathy/software-2-0-a64152b37c35 > It turns out that a large portion of real-world problems have the property that it is significantly easier to collect the data (or more generally, identify a desirable behavior) than to explicitly write the program. "Deep Learning is Eating Software", Pete Warden https://petewarden.com/2017/11/13/deep-learning-is-eating-software/ > The pattern is that there’s an existing software project doing data processing using explicit programming logic, and the team charged with maintaining it find they can replace it with a deep-learning-based solution.

Sounds like training a neural network with some test data? Software 2.0 as outlined here: https://medium.com/@karpathy/software-2-0-a64152b37c35

A nice complement to this is Andrej Karpathy's Software 2.0 essay. Namely that it is getting much easier to collect the data. Than it ever will be to write the complex programs which can anticipate every case. Software 2.0 https://medium.com/@karpathy/software-2-0-a64152b37c35 One Model To Learn Them All https://arxiv.org/abs/1706.05137

No, the field has moved a lot further than that. Did you read the recent Software 2.0 rant by the Tesla AI guy? https://medium.com/@karpathy/software-2-0-a64152b37c35 Other good sources are The Master Algorithm (ML) or The Brain That Changes Itself (neuroscience), the former surveying general capabilities and limitations of various approaches and the later surveying findings from medicine on just how plastic our sensory inputs and interpretive capacities are through recent research.

Sort of, it uses Monto Carlo + a bunch of other things. [0] [0] https://medium.com/@karpathy/alphago-in-context-c47718cb95a5

Below portrays a different picture to me: https://medium.com/@karpathy/icml-accepted-papers-institution-stats-ba… Granted, many of those researcher may be Chinese but that is not the point.

The answer is that many ML solutions are leaky abstractions. See: https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06e…

This trend is also evident in conferences like ICML, CVPR and EMNLP.[1] I have a semi-popular open source ML project [2] and the repo gets more Chinese visitors/users (from posts on Weibo/WeChat etc.) than all other countries including USA. I think Chinese researchers & government have a key advantage in collecting enormous datasets, e.g. I wouldn't be amazed if they end up with state of the art models for problems such as object-detection/segmentation for videos taken from Vehicles, Medicine/Radiology and face recognition [3]. Several US strategic funding agencies seem to be pursuing projects with this implicit assumption. While american government agencies do collect data at a similar scale (Body Cams, Police dashcams), inefficiency in data sharing significantly complicate their use. [1] https://medium.com/@karpathy/icml-accepted-papers-institution-stats-ba… [2] http://www.deepvideoanalytics.com/ [3] https://github.com/seetaface/SeetaFaceEngine On an unrelated note: If you have a popular Github repo, you can view the Google Analytics style stats under graphs/traffic for referring urls/unique-visitor counts etc.