Sebastian Ruder

For those who don't know, this article is by Sebastian Ruder, who has a great blog [1]. His posts are often pretty deep dives of a particular area of interest within Natural Language Processing. A favorite of mine from last year gave a summary of the state-of-the-art in "Word Embeddings" [2], which I would recommend for anyone interested in the field. [1] http://ruder.io/#open [2] http://ruder.io/word-embeddings-2017/

You might be thinking of this article by Sebastian Ruder: https://www.ruder.io/nlp-imagenet/ Note that the author has a background spanning a lot of the timespans/topics discussed - much work in multilingual NLP, translation, and more recently at DeepMind, Cohere, and Meta (in other words, someone with a great perspective on everything in the top article). Re: Machine Translation, note that Transformers were introduced for this task, and built on one of the earlier notions of attention in sequence models: https://arxiv.org/abs/1409.0473 (2014, 38k citations) That's not to say there weren't holdouts or people who really were "hurt" by a huge jump in MT capability - just that this is a logical progression in language understanding methods as seen by some folks (though who could have predicted the popularity of chat interfaces).

1. If you goal is to work in industry, don't do pure mathematics PhD. What works, and is doable in 20-30h/week is doing an applied PhD in an area related to your work. I know quite a few people who did it with great success. (Quite a few friends, and some accomplished people like https://ruder.io/.) Also, as a person with PhD - spending 4-7 years in an irrelevant field is going to seriously hurt your industry career. (I speak as someone with PhD.) Yes, PhD is worth something - but it won't come in the place of experience. 2. For the second, you need time. On their side, even if they don't need to provide funding, they invest their time. Talk to people at a local department. You may impress them enough that they are going to work with you unofficially - but the bar may be high. For later - what is the exact area you are most interested in?

Tangentially related if you're interested in keeping up to date with what's going on in the field: Sebastian Ruder's blog has many good posts on recent advancements in NLP (literature reviews, conference highlights): http://ruder.io/ The posts are concise and accessible enough that you can skim through them quickly. Then you can go check out the paper directly if something piques your interest.

Which one? Temporally Recurrent Optimal Learning [0] [1] [2] is the only one i know about, but it wasn't that famous afaik. [0] https://www.ruder.io/highlights-nips-2016/ [1] https://medium.com/the-mission/rocket-ai-2016s-most-notorious-ai-launc… [2] https://securelist.com/rocket-ai-and-the-next-generation-of-av-softwar…

A good run-down of the different algorithms: https://ruder.io/optimizing-gradient-descent/ I think people favour adaptive learning rate options like Adam in practice since they generally do seem to perform well, and are often less sensitive to initial conditions and the exact hyper-parameters used. There will always be people who like to test N optimizers with parameter sweeps to squeeze a tiny bit of extra performance out, but for the rest of us the default Adam or AdamW options and good, unobjectionable choices :)

"similar" means similar topic or similar interactivity? similar interactivity: https://epiphany.pub/@shi-yan/introduction https://www.smart-biology.com/ https://mathigon.org/ http://immersivemath.com/ila/index.html https://textbooks.math.gatech.edu/ila/ similar topic https://ruder.io/

Machine learning researchers aren't working on language modeling because they want to enable fake news. They are working on it because it improves all downstream NLP tasks. See: http://ruder.io/nlp-imagenet/. BERT, Elmo and XLNet all fall under this use case. For example if you're trying to recognize speech or translate some text, it helps a lot if you can start off producing something that is statistically grammatical even if the content is nonsense.

Nice, I was just starting to contemplate about how much work it would be to train a bert model from scratch. Context to understand the importance of this release: http://ruder.io/nlp-imagenet/ Though not named in the post, BERT is part of the same family of models as these.

Here[1] is an article describing the same, written by the author himself. [1]: http://ruder.io/optimizing-gradient-descent/index.html

2017 version: https://www.ruder.io/word-embeddings-2017/

There's definitely active research on it. But here's the basic recipe that gets state-of-the-art accuracy on most tasks. It's been around for 5 years or so now, which is why I said "just". You take an encoder transformer architecture like BERT and train it on a language modelling objective. Then you discard the part of the network that does the token prediction, and stick some randomly initialised network that does a specific task on. This is generally kept minimal. For classification tasks, often people just connect a linear layer to the vector that represents a dummy sentinel token after the sequence. The general goal is to exploit the representations learned during the language modelling task, so that the network starts out knowing general grammatical structure of the language, multi-word expressions, can distinguish word senses from each other, etc. Then it needs far fewer examples to learn a specific task. There's definitely transfer learning going on: if you initialise the network randomly instead of via the LM objective, results are very much worse. This is a good blog post from when the recipe was still fairly new: https://www.ruder.io/nlp-imagenet/