Peter Bloem

Tangential to this post but Peter Bloem’s Transformers from Scratch (2019) was one of the best early tutorials on transformers and self-attention. There have been many more tutorials since then but this was the one that solidified a mental picture for me. https://peterbloem.nl/blog/transformers

I recently stumbled across a truly excellent 5-part deep dive into PCA on Peter Bloem's blog, which I highly recommend to anyone who's interested. It assumes you have some level of linear algebra knowledge. https://peterbloem.nl/blog/pca

besides everything that was mentioned here, what made it finally click for me early in my journey was running through this excellent tutorial by Peter Bloem multiple times https://peterbloem.nl/blog/transformers highly recommend

It took me a long time to wrap my head around the whole "key/query/value" explanation (and to be honest I regularly forget which vector is which). I find the "weighted sum of vectors" explanation much simpler/more intuitive; this blog post is IMO the best on the subject: https://peterbloem.nl/blog/transformers

you can catch up very quickly. great resource on transformers, which underpin the openai breakthroughs (the "T" in GPT-3 stands for transformer): http://peterbloem.nl/blog/transformers great reddit sub for tracking ml research and trends: https://www.reddit.com/r/machinelearning great email newsletter on ml from andrew ng: https://www.deeplearning.ai/thebatch/ also, someone must maintain old code or things break. thanks for your contributions.

This is the best article I have read so far explaining the transformer architecture. The clear and intuitive explanation can’t be praised enough. Note that the teacher has a Machine Learning course with video lectures on youtube that he references throughout the article : http://www.peterbloem.nl/teaching/machine-learning

And perhaps things like: https://www.youtube.com/watch?v=VPRSBzXzavo and http://jalammar.github.io/illustrated-transformer/ and https://peterbloem.nl/blog/transformers See also: https://news.ycombinator.com/item?id=35990118

For a while now, an answer I've seen is to start with "Attention Is All You Need", the original Transformers paper. It's still pretty good, but over the past year I've led a few working sessions on grokking transformer computational fundamentals and they've turned up some helpful later additions that simplify and clarify what's going on. You can quickly get overwhelmed by the million good resources out there so I'll keep it to these three. If you have a strong CS background, they'll take you a long way: (1) Transformers from Scratch: https://peterbloem.nl/blog/transformers (2) Attention Is All You Need: https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c… (3) Formal Algorithms for Transformers: https://arxiv.org/abs/2207.09238

There are millions of "Transformers Explained" blog posts by now. The one I got the most out of is "Transformers from Scratch" by Peter Bloem: http://peterbloem.nl/blog/transformers

Yes this seems like an early work in progress, compared to Jay's previous Transformer articles. In addition to your link, I've found a really good Transformer explanation here (backed by a Github repo w/ lively Issues talk): http://www.peterbloem.nl/blog/transformers Additionally, there's a paper on visualizing self-attention: https://arxiv.org/pdf/1904.02679.pdf

I'm the author of https://jalammar.github.io/illustrated-transformer/ and have spent years since introducing people to Transformers and thinking of how best to communicate those concepts. I've found that different people need different kinds of introductions, and the thread here includes some often cited resources including: https://peterbloem.nl/blog/transformers https://e2eml.school/transformers.html I would also add Luis Serrano's article here: https://txt.cohere.com/what-are-transformer-models/ (HN discussion: https://news.ycombinator.com/item?id=35576918). Looking back at The Illustrated Transformer, when I introduce people to the topic now, I find I can hide some complexity by omitting the encoder-decoder architecture and focusing only on one. Decoders are great because now a lot of people come to Transformers having heard of GPT models (which are decoder only). So for me, my canonical intro to Transformers now only touches on a decoder model. You can see this narrative here: https://www.youtube.com/watch?v=MQnJZuBGmSQ

Here you go: http://www.peterbloem.nl/blog/transformers

Transformers from scratch: 1.) https://e2eml.school/transformers.html https://news.ycombinator.com/item?id=35697627 (46 comments, 1 month ago) https://news.ycombinator.com/item?id=29315107 (17 comments, 2 years ago) 2.) http://www.peterbloem.nl/blog/transformers https://news.ycombinator.com/item?id=20773992 (28 comments, 4 years ago) https://news.ycombinator.com/item?id=29280909 (9 comments, 2 years ago) Source: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&q…

In case you find Attention (and specially transformers) interesting I have some saved links introducing the topic: http://www.peterbloem.nl/blog/transformers https://nostalgebraist.tumblr.com/post/185326092369/the-transformer-ex… https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf http://jalammar.github.io/illustrated-transformer/ https://arxiv.org/pdf/1807.03819.pdf

not sure what 'mixing LLM models' entails but these are maybe some good starting points - karpathy - https://www.youtube.com/watch?v=kCc8FmEb1nY - https://towardsdatascience.com/beautifully-illustrated-nlp-models-from… - https://dzone.com/articles/a-deep-dive-into-the-transformer-architectu… - https://peterbloem.nl/blog/transformers - http://nlp.seas.harvard.edu/2018/04/03/attention.html - https://lilianweng.github.io/posts/2023-01-27-the-transformer-family-v… - https://blog.quickchat.ai/post/tokens-entropy-question/ - https://dugas.ch/artificial_curiosity/GPT_architecture.html - https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-doing-and-… - https://d4mucfpksywv.cloudfront.net/better-language-models/language_mo… - https://arxiv.org/pdf/2005.14165.pdf - https://arxiv.org/pdf/2303.08774.pdf - https://arxiv.org/pdf/2303.17564.pdf