e2eml School

No, I didn't! Before I came across that image (and the paper Linearized Transformers are Secretly Fast Weight Programmers) I very much did not understand transformers. I had spent at least 20 hours with Attention Is All You Need, and probably another dozen hours with https://e2eml.school/transformers.html and wasn't getting much of anywhere. I have no ML/AI background; just a typical undergraduate-CS-level familiarity with neural networks -- the basic stuff that hasn't changed since the 1990s. I do have some experience with linear algebra, but that isn't the hard part of any of this. Frankly, most people who publish in this field go out of their way to obfuscate the key insights. Mediocre physicists (but not the truly brilliant ones) do the same thing. It's very annoying.

And https://e2eml.school/transformers.html (recommended below, also)

Ben also has posted a Part II follow up: https://benlevinstein.substack.com/p/a-conceptual-guide-to-transformer… Also, another good intro article - Transformers from Scratch: https://e2eml.school/transformers.html

The Markov chain by definition satisfies the Markov property (https://en.m.wikipedia.org/wiki/Markov_property). If you want a more in depth sense of why it's not, look at the following link that describes the transformer architecture in detail and specifically goes over moving from a Markov process to one that no longer satisfies it, specifically around the following paragraph: > The first thing that becomes apparent is that, when trying to predict the word that comes after ran, we no longer look at just one line, but rather a whole set of them. We've moved out of the Markov realm now. Each row no longer represents the state of the sequence at a particular point. Instead, each row represents one of many features that may describe the sequence at a particular point. The combination of the most recent word with each of the words that came before makes for a collection of applicable rows, maybe a large collection. Because of this change in meaning, each value in the matrix no longer represents a probability, but rather a vote. Votes will be summed and compared to determine next word predictions. - https://e2eml.school/transformers.html

https://e2eml.school/blog.html there was also a really cool portfolio site in complete black and white, link of which I couldn't find.

I think it applies because lookups can be done by multiplying one-hot vectors (or masks) with matrices; this is roughly analogous to what happens when we multiply Q with K^T in a self-attention head. Read this: https://e2eml.school/transformers.html#table_lookup And then read: https://e2eml.school/transformers.html#attention Also, multiplication is differentiable as you can calculate a partial derivative of this operation, and the ability to do that is what allows gradient descent optimization via the chain rule.

This link was posted here recently, and was the most understandable explanation I've found so far: https://e2eml.school/transformers.html

One or both of these might help Transformers, Explained: Understand the Model Behind GPT-3, BERT, and T5: https://daleonai.com/transformers-explained Transformers from Scratch: https://e2eml.school/transformers.html The first link covers Attention well.

I’m a smart(tm) guy and I couldn’t make heads or tails from that paper. That hand wavy diagram that’s supposed to explain the whole concept still makes me mad. I’m finding this page is explaining it a lot better https://e2eml.school/transformers.html

I found this article on “transformers from scratch”[0] to be a perfect (for me) middle ground between high level hand-wavy explanations and overly technical in-the-weeds academic or code treatments. [0] https://e2eml.school/transformers.html

Transformers from Scratch link: https://e2eml.school/transformers.html discussion here: https://news.ycombinator.com/item?id=29315107

I'm the author of https://jalammar.github.io/illustrated-transformer/ and have spent years since introducing people to Transformers and thinking of how best to communicate those concepts. I've found that different people need different kinds of introductions, and the thread here includes some often cited resources including: https://peterbloem.nl/blog/transformers https://e2eml.school/transformers.html I would also add Luis Serrano's article here: https://txt.cohere.com/what-are-transformer-models/ (HN discussion: https://news.ycombinator.com/item?id=35576918). Looking back at The Illustrated Transformer, when I introduce people to the topic now, I find I can hide some complexity by omitting the encoder-decoder architecture and focusing only on one. Decoders are great because now a lot of people come to Transformers having heard of GPT models (which are decoder only). So for me, my canonical intro to Transformers now only touches on a decoder model. You can see this narrative here: https://www.youtube.com/watch?v=MQnJZuBGmSQ

There are many others that are better. 1/ The Annotated Transformer Attention is All You Need http://nlp.seas.harvard.edu/annotated-transformer/ 2/ Transformers from Scratch https://e2eml.school/transformers.html 3/ Andrej Karpathy has really good series of intros: https://karpathy.ai/zero-to-hero.html Let's build GPT: from scratch, in code, spelled out. https://www.youtube.com/watch?v=kCc8FmEb1nY GPT with Andrej Karpathy: Part 1 https://medium.com/@kdwa2404/gpt-with-andrej-karpathy-part-1-865bec6fb… 4/ 3Blue1Brown: But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning https://www.youtube.com/watch?v=wjZofJX0v4M Attention in transformers, visually explained | Chapter 6, Deep Learning https://www.youtube.com/watch?v=eMlx5fFNoYc Full 3Blue1Brown Neural Networks playlist https://www.youtube.com/playlist?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3…

Transformers from scratch: 1.) https://e2eml.school/transformers.html https://news.ycombinator.com/item?id=35697627 (46 comments, 1 month ago) https://news.ycombinator.com/item?id=29315107 (17 comments, 2 years ago) 2.) http://www.peterbloem.nl/blog/transformers https://news.ycombinator.com/item?id=20773992 (28 comments, 4 years ago) https://news.ycombinator.com/item?id=29280909 (9 comments, 2 years ago) Source: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&q…