Yann LeCun

Two word surnames are unusual in English, and the "Le", "La" etc. could easily be misinterpreted as a middle name, leading to wrong names like "Yann Cun". English surname prefixes like "Mac", "Mc", "Fitz", "O" [1] are generally attached to the name that follows (sometimes with the second part capitalised, sometimes not), so there is an old and widespread convention being followed here. Nowadays it's easier to search for or use as a username too. Yann LeCun spells his own name like that: http://yann.lecun.com/ [1] These are actually Celtic originally: https://en.wikipedia.org/wiki/Celtic_onomastics#Surnames

I'll start off by saying that this type of research is very interesting, and in this case well done. However, the issue to me in "all optical neural nets" has always been the non-linearity. The non-lineartiy (or activation function) is crucial to the high performance we see with deep neural networks. For example, on the task of interest here (MNIST, a classic hand-written digit classification task), a old-school linear classifier can get between 88% and 92.4% accuracy [1]. Whereas non-linear neural networks easily achieve 95%+ [1]. So until they figure out a way to represent a nice non-linearity (unless I missed something), these all optical techniques are at a permanant disadvantage. [1] http://yann.lecun.com/exdb/mnist/

The first convnet that is similar to what we use today was described in 1989 [1], and immediately became the best method to do a handwritten digit recognition. [1] http://yann.lecun.com/exdb/publis/pdf/lecun-89e.pdf

Fascinating. Accuracy on MNIST is a lousy 82.9%, 86.5%, and 87.4% for the networks with 400, 1000, and 6400 neurons, respectively -- well below the accuracy achieved by even LeNet-5, a landmark historical CNN architecture from the 1990's.[a] Nonetheless, I get excited when I come across research like this. I get excited about the possibility of building devices with internal physical/chemical network/tree structures capable of learning in response to stimuli, e.g., by settling into low-energy states that "best model" such stimuli or via some other mechanism. This is the kind of research, I think, that could eventually lead to AI systems operating with orders of magnitude more energy, space, time, and cost efficiency than current systems implemented as software running on GPU, TPU, and similar devices. [a] http://yann.lecun.com/exdb/lenet/

The University of Toronto is home to Geoffrey Hinton and his research lab. Hinton was the first to use backpropagation for machine learning and a lot of the techniques used today were pioneered by his lab. He is also manages the Google Brain team in Toronto. Yann LeCun, the director of AI at Facebook, also did a postdoc at Toronto. [0] http://learning.cs.toronto.edu:40292/people#Faculty [1] http://yann.lecun.com/

99.3% accuracy is not that great. Currently the best performer on this data set has 99.77% accuracy: http://yann.lecun.com/exdb/mnist/

Humans appear to do quite well on ImageNet (anecdotally, one person got 5.1% error: http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-aga… Of course there are recent deep models that do better than that, but the author opines (and I agree) that an ensemble of trained human annotators would do better than the best deep models. MNIST is the true toy dataset (doesn't really tell you much about your algorithm's performance) - while there aren't any reported human evaluations of MNIST, LeCun estimates the human error rate is 0.2% - better than any deep models (admittedly without justification: http://yann.lecun.com/exdb/publis/pdf/lecun-95a.pdf).

I think it will be Facebook, they have the largest collection of data on human intelligence, plus they also have Yann LeCun http://yann.lecun.com one of (if not the top) the top three AI experts (others: Yoshua Bengio, Geoffrey Hinton)

You could try the following improvements to speed up neural network training: - Resilient Propagation (RPROP), it significantly speeds up training for full batch learning: http://davinci.fmph.uniba.sk/~uhliarik4/recognition/resources/rprop/rb… - RMSProp, introduced by Geoffrey Hinton, also speeds up training but can also be used for mini-batch learning: https://class.coursera.org/neuralnets-2012-001/lecture/67 (sign up to view the video) Please consider more datasets when benchmarking methods: - MNIST ( 70k 28x28 pixel images of handwritten digits ): http://yann.lecun.com/exdb/mnist/ . There are several wrappers for Python on github. - UCI Machine Learning Repository: http://archive.ics.uci.edu/ml/datasets.html

I went to a talk by Yann LeCun [1] a few months back, "Learning Feature Hierarchies for Vision." The current state of the art in this field is mind-blowing to an outsider. The final demo was a program that he trained in a matter of seconds to recognize and distinguish faces of various random audience members, in real time, from different viewing angles. [1] http://yann.lecun.com/

> One might as well say that neural networks trained with gradient descent are a reinvention of numerical methods for function approximation. I don't know anyone who would disagree with that statement, and this is the standard framing I've encountered in nearly all neural network literature and courses. If you read any of the classic gradient based papers they fundamentally assume this position. Just take a quick read of "A Theoretical Framework for Back-Propagation (LeCun, 1988)" [0], here's a quote from the abstract: > We present a mathematical framework for studying back-propagation based on the Lagrangian formalism. In this framework, inspired by optimal control theory, back-propagation is formulated as an optimization problem with nonlinear constraints. There's no way you can read that a not recognize that you're reading a paper on numerical methods for function approximation. The issue is that Vaswani, et al never mentions this relationship. 0. http://yann.lecun.com/exdb/publis/pdf/lecun-88.pdf

I posted this on HN back in 2023, reposting now because I don't think this article goes far enough: I’ll make the bold claim that the following industries / companies would not exist without the USPS: The Airline Industry: In the early days of American aviation, air transportation was unproven and not financially viable, until the USPS built the necessary infrastructure and gave contracts to airlines to allow them financial feasibility… starting in 1918! [1] Machine Learning: In 1989 Yann LeCun wrote his seminal paper “Backpropagation Applied to Handwritten ZIP Code Recognition”, which used the USPS’s data set and has today become the hello world of machine learning tasks. More importantly this is the first commercial or industrial application of machine learning. [2] Netflix: Before Streaming became a thing, Netflix was shipping DVDs via the USPS. The Postal Service adapted its processes and equipment to make this financially feasible, supporting Netflix through its transition to streaming. [3] Amazon: Early Amazon was only a book vendor, the USPS offered special rates for books that made it possible for Bezos to be profitable from his garage … in 1994, thus birthing the behemoth it is today. [4] Chickens: okay, not really. But the USPS ships millions of pounds of live chickens and other animals each year! [5] [1] https://www.history.com/news/us-aviation-airmail-passenger-f... [2] http://yann.lecun.com/exdb/publis/pdf/lecun-89e.pdf [3]…

What is your interest in deep learning? Developing new and novel algorithms, or applying already existing algorithms to new and novel problems, domains and datasets? If it's the latter then high level math skills aren't that necessary. A solid understanding of the domain and data under investigation is much more important, as is being a decent programmer, knowing your way around data cleaning and data management and having a solid understanding the strengths and weaknesses of the different algorithms out there. If you're interested the absolute best way to get started is just to start. Download a well studied data set like this one: http://yann.lecun.com/exdb/mnist/ Grab one of the dozens of tutorials that talk about how to approach this data set and follow along in something like scikit-learn or Flux.jl. You'll soon enough have gone from zero to having developed a tool that can recognize handwritten digits. From there you can just keep going.

One of the best proposals for research/review system I have seen is from Yann LeCun: http://yann.lecun.com/ex/pamphlets/publishing-models.html. The idea is that you post your paper on some common repository like arxiv. The "reviewer entity" or RE which is self-organized group with interest in sub-field can be invited to review it. The RE accepts or rejects it. Citations graph propogates to RE as reputation score. The problem with his proposal: 1. Authors are not anonymous which can bias prestigious REs to avoid papers from unknown authors. 2. Everything still depends on citations, most of which are usually worthless as people mostly cite to fill obligatory related sections, not because they are actually using your work. 3. No obligation for REs to review anything. Doing reviews is tiresome and busy people may just avoid it unless there is a clear obligation/honor system involved. 4. Prestigious REs will be invited to review by everyone, causing highly uneven distribution for the workload. I believe openreview system was started based on LeCun's thoughts but needless to say we haven't found a good system that can resolve above issues. More importantly, any change in system needs to come from community leaders who are part of organizing commitees for conferences or website developers like arxiv and Google Scholars. Unfortunately, last two have been virtual stasis for long time. It's astonishing how little investment exists for our main engines of progress that is…

err.. I think it is an evolution. Deep architectures allow for a more efficient function approximation from fewer examples than shallow architectures do. Deep (neural) networks are really just a generalization of machine learning (on graphs). The key is that we learn the similarity function to discriminate from examples instead of specifying it a priori. You can also build classifiers other ways: linear/logistic regression based on feature vectors or by providing some similarity metric (SVM). But in these cases you are providing the discriminator function. For example, in SVMs you have to provide a similarity measure as the "kernel" maps your feature vector into a higher dimensional space where the examples can be separated. In deep neural networks we don't really know (or care to some extent) what the optimal feature vectors are or what the correct similarity metric is. We only care that at the end we've encoded it correctly (e.g. having chosen enough parameters/layers etc...) after training. Again the NN is some general function that applies a soft-max relationship from the inputs to outputs for each layer. Yann LeCun has a great paper (2007) explaining this: http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf.

As an expert in both the umbrella of methods known as compressed sensing and more recently deep learning, my opinion is that these techniques are largely complementary. Deep learning is fantastic at for instance building appearance models and exploiting hierarchical nature of natural images, and hierarchical feature building in general where appropriate, while methods from compressed sensing remain as quite useful general methods via which one can efficiently extract low dimensional structure, from noisy and at times even highly corrupted datasets. It is true that deep learning is now state of the art in certain tasks such as super-resolution for natural images, which were previously the domain of linear inverse problems, and this is due to its ability to learn useful natural image priors, something that isn't possible with simpler linear or slightly non-linear models. Meanwhile, compressed sensing style methods still excel in situations where the data does not benefit from hierarchical compression, labeled data is not available and/or the data is highly corrupted. Take for example the Netflix challenge problem, discussed in the video, for which deep learning is unlikely to offer substantial benefits, at least for the problem as stated (we just observe partial information about movie ratings). Where deep learning could potentially help in that situation is for instance grouping movies according to high level semantic information derived from text descriptions, other…

SVMs may make a lot of intuitive sense due to the intuitive nature of the max-margin method of building a classifier, but that doesn't make them "much, much better" than a neural network. Neural networks are "universal" function approximators in the sense that given sufficient number of hidden layer units and training data, they can approximate any Borel measurable function. This makes them theoretically powerful. (There are issues with this "universalness" claim though when doing cognitive modelling. See "Cognition and the computational power of connectionist networks", Hadley at http://www.informaworld.com/smpp/content~db=all?content=10.1080/095400… In practice, the best handwritten digits classifier is still neural networks. Lecun keeps scores at http://yann.lecun.com/exdb/mnist/. Are there challenges with training neural networks? Certainly. But for many applications, there are good reasons to use a neural network. For instance, inference in a neural network is fast, and the learnt model is small in size (in feedforward networks, just a few large matrices). SVMs on the other hand are "easier" to train, but care has to be taken to use the "right" kernel, and the learnt model in practice has to, in many cases, carry around with it up to half or more of the training data set (ie, the support vectors), making the model bigger in size, and inference slower (although there is work in making a sparser set of support vectors. See "An Effective Method of…

I'm not sure what you are imagining the scope of your first project to be, but I would recommend you begin by understanding and implementing some well-known algorithms. Start with a Guassian Mixture model trained with the EM algorithm. Then do linear and logistic regression, perceptron is pretty simple and a simpler version of the widely used support vector machine. The handwriting recognition database here is fantastic for testing a variety of simple ML models: http://yann.lecun.com/exdb/mnist/ In our machine learning class, we would use data from the KDD cup for our projects. Why don't you create a submission for old KDD cups and see if your model can do better than random? 1998 is good for logistic and linear regression: http://www.kdnuggets.com/meetings/kdd98/kdd-cup-98.html Using the 2007 dataset, you can try out some of the matrix factoring methods that have worked well for the Netflix prize: http://www.cs.uic.edu/~liub/Netflix-KDD-Cup-2007.html These should all be relatively small tasks. Learning how to interpret your results and iterate your models to make them better will take longer than understanding the algorithms.

> Why is most AI work done in Lisp? Yann LeCun developed Lush, which is a Lisp for neural networks, during the early days of deep architectures. See https://yann.lecun.com/ex/downloads/index.html and https://lush.sourceforge.net. Things moved to Python after a brief period when Lua was also a serious contender. LeCun is not pleased with Python. I can't find his comments now, but he thinks Python is not an ideal solution. Hard to argue with that, as its mostly a thin wrapper over C/C++/FORTRAN that poses an obvious two-language problem.

Have a read through the first paper describing a convolutional neural network, from 1998: http://yann.lecun.com/exdb/publis/pdf/lecun-01a.pdf There's absolutely no mention of biological inspiration whatsoever. At the same time, one can point to a long and rich history of convolutional filters being used in signal processing. And then there's the name, Convolutional Neural Network. The entire concept of a CNN is framed as a series of learned filters.

I didn't investigate this more deeply , but I think if you used an arena allocator on each iteration of the loop, and simply freed the whole allocator after each iteration, you might get much better performance. EDIT: the MNIST link requires authentication: http://yann.lecun.com/exdb/mnist/ How can we get access to the test run at the end?

Again, MNIST is not Drosophilia of deep learning. For deep learning, it is a trivial. And often misleading, a noticed by François Chollet: > Many good ideas will not work well on MNIST (e.g. batch norm). Inversely many bad ideas may work on MNIST and no[t] transfer to real [computer vision]. It might be considered Drosophilia of machine learning, though. Take a look at this beautiful table of the results of various classifiers http://yann.lecun.com/exdb/mnist/.

Some of the questions of depth vs breadth are explained intuitively in this Bengio Lecun paper from 2007. http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf

I think Norvig is a smart person, I enjoy his books, papers and jupyter notebooks, but I always thought he was pretty clueless regarding AI, as history indeed demonstrated. That's not an ad hominem. What is shameful was the extreme shunning of the mainstream scientific community to ANNs, in 90's-00's decades, to the point that it was considered career suicide to publish a ANN paper. I believe Yann LeCun has said similar things in the past[0], reminiscing the time when it took him several years(!) to get an ANN paper accepted for publication. [0] http://yann.lecun.com/ex/pamphlets/publishing-models.html

This is the researcher who published the original convolutional neural network papers providing a basis for modern deep neural networks. http://yann.lecun.com/exdb/publis/pdf/lecun-99.pdf

Yann Le Cun modified the spelling of his name to Yan LeCun in English by himself. [No, Your Name can't possibly be pronounced that way] http://yann.lecun.com/ex/fun/index.html#gellman

You may be looking for "Optimal Brain Damage": http://yann.lecun.com/exdb/publis/pdf/lecun-90b.pdf

Chris, glad you like it. I can relate to that feeling, which I also had as an undergrad. It was a big motivation for this work -- and I think the analogy can go even further. For example, there's a really cool lecun paper about NN training dynamics actually being a lagrangian system: http://yann.lecun.com/exdb/publis/pdf/lecun-88.pdf. Another thing I want to try is literally write down a learning problem where S (the action) is the cost function and then optimize it...

Wow, what timing. Late last night I had a conversation with someone explaining that Hacker News is not your typical message board -- it's owned and operated by YC and sits atop algorithms developed by some of the pioneers in spam and anomaly detection [1] [2], and it's is also an open dataset -- analyzed and scrutinized -- used by hackers worldwide to train and test bespoke AI. HN is a live MNIST [3] for anomaly detection. [1] http://www.paulgraham.com/spam.html [2] http://googlesystem.blogspot.com/2007/07/paul-buchheit-man-behind-gmai… [3] http://yann.lecun.com/exdb/mnist/

See Yann's tutorial on EBM: http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf We actually ran into them when doing research in our startup. It is a really powerful perspective.