Off the Convex Path

Probably the single best article I saw recently was this: http://www.offconvex.org/2017/12/08/generalization1/ I think that blog post references most of the relevant papers as well, but if not, check the sibling comments I've made.

Well, the thing that word2vec can (and should) be understood in terms of word coincidences (and pointwise mutual information) is important, but hardly new. I tried to explain it here: http://p.migdal.pl/2017/01/06/king-man-woman-queen-why.html. There is a temptation to use just the word pair counts, skipping SVD, but it won't yield in the best results. Creating vectors not only compresses data, but also finds general patterns. This compression is super important for less frequent words (otherwise we get a lot of overfitting). See "Why do low dimensional embeddings work better than high-dimensional ones?" from http://www.offconvex.org/2016/02/14/word-embeddings-2/.

Not a book or a course, but rather a blog post by Sanjeev Arora et al that made backpropagation crystal clear for me: [1]. [1] https://www.offconvex.org/2016/12/20/backprop/

Related blog post by the authors: http://www.offconvex.org/2021/04/07/ripvanwinkle/

One interesting paper on estimating the complexity of code: http://www.offconvex.org/2021/04/07/ripvanwinkle/

Yes. https://joanbruna.github.io/MathsDL-spring18/ On Tanh and ReLus, the paper on SeLUs is pretty intense. ReLUs help with vanishing gradients to some degree on a larger degree of loss and activation functions, btw. edit: On your question about a billion data points and predicting behavior, we are getting there. http://www.offconvex.org/2017/03/30/GANs2/

Not the person you replied to, but your comment was both rude and incorrect enough that I feel the need to reply. See for example http://www.offconvex.org/2016/03/22/saddlepoints/ for some discussion on this.

Thanks for your blog post! Are you familiar with http://www.offconvex.org/2016/07/10/embeddingspolysemy/? I read a blog post maybe a year ago that explained word embeddings under an 'atom interpretation', in which word embeddings are really sparse combinations of 'atoms' (I don't really remember more than that). It was very interesting, but then I forgot about it. Trying to find the post again I only came up with the above paper, which is probably the same idea, but it wasn't what I originally read. Wish I could find it.

I really like these posts by Sanjeev Arora : 1) http://www.offconvex.org/2015/12/12/word-embeddings-1/ 2) http://www.offconvex.org/2016/02/14/word-embeddings-2/ For a more theoretical explanation : https://arxiv.org/abs/1502.03520

The linked Rong Ge blog post explains the issue in a bit more detail: http://www.offconvex.org/2016/03/22/saddlepoints/

I meant in the sense of - you have benchmarkers and trainers. If you publicize your evaluation, trainers may likely have their models 'consume' it, even if only indirectly: another person creating their own benchmark from scratch may be influenced by yours, even if the new question sets are clean-room. That, and the rule of thumb that benchmark value dissipates like sqrt(age) [0] So there is a definite advantage to never publicizing your internal benchmark. But then, no one else can replicate your findings. You should assume that the space of benchmarks that are actually decent at evaluating model performance is much larger and most of the good ones, the ones that were costliest to produce, are hidden, and might not even correspond very well with the public ones. And that the public expensive benchmarks are selective and have a bias towards marketing purposes. [0] https://www.offconvex.org/2021/04/07/ripvanwinkle/

Here is a fun one: https://www.cs.cmu.edu/~tom/pubs/science2008.pdf -- "Predicting Human Brain Activity Associated with the Meanings of Nouns" "We present a computational model that predicts the functional magnetic resonance imaging (fMRI) neural activation associated with words for which fMRI data are not yet available. This model is trained with a combination of data from a trillion-word text corpus and observed fMRI data associated with viewing several dozen concrete nouns. Once trained, the model predicts fMRI activation for thousands of other concrete nouns in the text corpus,with highly significant accuracies over the 60 nouns for which we currently have fMRI data." This is a frequently cited paper - I stumbled across it recently as a drive-by mention in this blog post about word embeddings: http://www.offconvex.org/2015/12/12/word-embeddings-1/ Anyway - totally tangential to the topic of the OP. But maybe some interesting food for thought.

You're correct. Boosting was directly formulated in the PAC framework. (Source: http://l2r.cs.uiuc.edu/Teaching/CS446-17/LectureNotesNew/boosting/main… "The original boosting algorithm was proposed as an answer to a theoretical question in PAC learning [The Strength of Weak Learnability; Schapire, 89.]") It took a while, but there's been a lot of work lately explaining neural nets' performance over the last 5 years of so, from papers showing PAC learnability for specific architectures (https://arxiv.org/abs/1710.10174) to work saying that most local optima are close to global optima (http://www.offconvex.org/2016/03/22/saddlepoints/), to work saying that the optimization error incurred (as separate from approximation and estimation errors) serve as a form of regularization for deep neural networks. And understanding how these things work helps improve and speed up these methods and models: it's hybrid algorithms which are enabling performance in time-series data and more complex tasks. The future will nearly certainly use neural networks as part of many algorithms, but I doubt that the full machinery will be simple feed-forward nets of ever-increasing sizes.