Jay Alammar

Love the visual explainers for machine learning nowadays. The author has more here: https://jalammar.github.io/ Amazon has some highly interactive ones here: https://mlu-explain.github.io/ Google had: distill.pub Hope to see education in this space grow more.

Content marketing ruined blogging, just like SEO ruined search. I used to work with a lot of content marketers when it started to get hot. SEO was doing particularly poorly for a period (Google used to resist seo rather than reward it) and people realized that if you started writing "authentic" content, then people would organically link to it and it would increase your ranking. Then social media furthered this problem by people mindlessly sharing content they never read but just looked engaging. Marketing robots where just churning out content on blogs because eventually you would just hit search engine gold by nail the correct topic for some long tail search term. In my own domain, the atrocious "Towards Data Science" is a great example of this (except their "brilliant" innovation was to replace marketing robots with people eager to pad their resume). The content is largely garbage, but it looks about right and is easily shareable. Today, whether you like it or not, if you want to make people to see your blog at all you have to think at least a bit like a content marketer to get seen at all. Consider Jay Alammar's fantastic blog [0], which is very high quality content, but still requires a tremendous amount of polish and marketing to survive. All of this has made it impossible to create the network required for a "global conversation among creative weirdos". Real content does get made, but has to be produced with the effort of a real marketer. 0.…

Highly recommend instead reading the human created "The Illustrated GPT-2" by Jay Alammar - https://jalammar.github.io/illustrated-gpt2/ And his similar work. He also has a free course on "how llms work"

My problem with this is that it doesn't explain a lot. You can manually make a vector of a word and then step wise get up to word2vec approach and then document embedding. My post[1] does some of the first part and this great word2vec post[2] dives into it in more detail. [1] https://earthly.dev/blog/cosine_similarity_text_embeddings/ [2] https://jalammar.github.io/illustrated-word2vec/

Jay Alammar has my favorite sequence of tutorials from basic neural network math to GPT2. Particularly [0], [1], and [2] [0] http://jalammar.github.io/illustrated-transformer/ [1] http://jalammar.github.io/illustrated-gpt2/ [2] https://jalammar.github.io/visualizing-neural-machine-translation-mech…

Good point, thanks for the link! (one of the links there leads to this wonderful post: Highly recommended: http://jalammar.github.io/illustrated-transformer/)

More excellent resources from Jay and others: The Illustrated Transformer - http://jalammar.github.io/illustrated-transformer/ Beyond the Illustrated Transformer - https://news.ycombinator.com/item?id=35712334 LLM Visualization - https://bbycroft.net/llm

https://arxiv.org/abs/2207.09238 This is a great transparent and clear description of what is actually going on in attention layers. Alternatively, this article by Jay Alammar is also very popular http://jalammar.github.io/illustrated-transformer/

I am not sure what kind of background that you have, but with a bit of basic linear algebra you might enjoy this classic: https://jalammar.github.io/illustrated-transformer/

Not the guy you asked, but these are often recommended. https://jalammar.github.io/illustrated-transformer/ https://nlp.seas.harvard.edu/2018/04/03/attention.html

I found these resources to be helpful. https://jalammar.github.io/illustrated-transformer/ This is a good illustration of the transformer and how the math works. https://karpathy.ai/zero-to-hero.html If you want a deeper understanding of transform and how they fit in the whole picture of deep learning, this series is far and away the best resource I found. Karpathy goes into transformers by the sixth lecture, the previous lectures give a lot more context how deep learning works.

Jay alammar's Illustrated transformer, although this too is detailed. I think it's still worth taking a look, because really i don't think that people have yet "compressed" what transformers do intuitively. None of the concepts of the networks are particularly hard math - it's basic algebra. But the overall construction is complicated. https://jalammar.github.io/illustrated-transformer/

The Illustrated Transformer is pretty great. I was pretty hazy after reading the paper back in 2017 and this resource helped a lot. https://jalammar.github.io/illustrated-transformer/

This is a great comprehensive deep dive, thank you for sharing! Interestingly, there was a blog post and video in 2019 by Peter Bloem [1] with the same title which I consider to be one of the very best quick intros to transformers. Rather than diving into all of the details right away, Peter focuses on the intuition and “peels the onion” gradually. [1] http://peterbloem.nl/blog/transformers Other notable good intros to transformers are: The Illustrated Transformer by Jay Alammar https://jalammar.github.io/illustrated-transformer/ The Annotated Transformer by Harvard NLP https://nlp.seas.harvard.edu/2018/04/03/attention.html

I really liked this post. Tons of small tidbits I feel I'd only get from working in the teams they worked on. Some things I noted: * Check out `ray` as an alternative to `multiprocessing` * Check out `tqdm` * Use `pdb` more * See if fast.ai or https://jalammar.github.io/illustrated-transformer/ are worthwhile * Prioritize the papers I read better * _Leveraged_ index funds?

I also want to make an additional "Thank You" note for the author on the lovely "The Illustrated Word2Vec" [0]. I wish every concept Machine Learning or otherwise would follow such a framework. [0] https://jalammar.github.io/illustrated-word2vec/

Hi HN, This is the first video in a series where I gently explain AI/ML with an emphasis on applications rather than the math/stats/algorithms. It's meant for the general public, but I hope it helps the practitioners understand more about where the business applications are so they can optimize their careers or startups and specialize in domains that could benefit their careers. It's also the first video on my channel. I hope to later create videos touching on other topics in my blog (https://jalammar.github.io/), like the Illustrated Transformers, BERT, Word2vec, and others. I hope you enjoy it. All feedback is welcome!

I personally enjoy https://jalammar.github.io/ and https://mlexplained.com/- For machine learning & deep learning stuff

Here are two more great articles about Transformers: The Illustrated Transformer (referenced in the parent): http://jalammar.github.io/illustrated-transformer/ The Annotated Transformer: http://nlp.seas.harvard.edu/2018/04/03/attention.html

My 2 favorites are 1. BERT - easy to understand, SOTA results on most NLP tasks, code and pretrained models readily available. What more can you ask! 2. ULMFit - transfer learning for NLP, have been waiting for something like this since a long time. Both these mean getting better results/models for most NLP tasks with lesser training data, effort and time. Here is another link explaining these topics in a little more detail (a must read if you are in the NLP field) -- https://jalammar.github.io/illustrated-bert/

one of the best visual tutorials on the transformer I came across http://jalammar.github.io/illustrated-transformer/

Focusing on Deep Learning specifically: - Most LLMs currently use the transformer architecture. You can learn about this visually (https://bbycroft.net/llm), or through this blog post (https://jalammar.github.io/illustrated-transformer/), or through any number of Andrej Karpathy's blog posts and materials. - To stay on top of papers that get published every week, I read a summary every Sunday: https://github.com/dair-ai/ML-Papers-of-the-Week - To learn more about the engineering side of it, you can join Discord servers such as EleutherAI's, or follow GitHub discussions of projects like llama.cpp Personally I think the best way to develop per unit time is probably to try to re-implement some of the big papers in the field. There's a clear goal, there are clear signs of success, there are many implementations out there for you to check your work against and compare and learn from. Good luck!

I agree that Karpathy's YouTube video is an excellent resource for understanding Transformers from scratch. It provides a hands-on experience that can be particularly helpful for those who want to implement the models themselves. Here's the link to the video titled "Let's build GPT: from scratch, in code, spelled out": https://youtu.be/kCc8FmEb1nY Additionally, for more comprehensive resources on Transformers, you may find these resources useful: * The Illustrated Transformer by Jay Alammar: http://jalammar.github.io/illustrated-transformer/ * MIT 6.S191: Recurrent Neural Networks, Transformers, and Attention: https://www.youtube.com/watch?v=ySEx_Bqxvvo * Karpathy's course, Deep Learning and Generative Models (Lecture 6 covers Transformers): https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9Gv... These resources cover different aspects of Transformers and can help you grasp the underlying concepts and mechanisms better.

I have been enjoying Natural Language Processing with Transformers [1]. It's largely focused on the Huggingface library, but Chapter 3 has a very nice walkthrough that builds up the encoder portion of an encoder-decoder Transformer from "scratch" (it still uses some primitives found in PyTorch like nn.Embedding). The decoder portion is covered in less depth and they instead refer folks to Karpathy's awesome minGPT [2], which implements a decoder-only (GPT-style) Transformer in ~300 lines of nicely-commented Python+PyTorch code. For a higher-level conceptual view of how Transformers work, you can check out the now-classic "Illustrated Transformer" series [3] and this programmer-oriented explanation (with code in Rust) from someone at Anthropic [4]. [1] https://www.oreilly.com/library/view/natural-language-processing/97810… [2] https://github.com/karpathy/minGPT [3] https://jalammar.github.io/illustrated-transformer/ [4] https://blog.nelhage.com/post/transformers-for-software-engineers/

Am I alone in really disliking Towards Data Science? While their articles always look nice, their content is all written quickly by data scientists wanting to polish their resume with the ultimate aim of rapidly generating content for TDS that will match every conceivable data science related search. This post clearly exists solely so that TDS can get the top spot for "Word2vec explained" (which they have). As evidence of this tactic you can see that there already is a TDS post "Word2vec made easy" [0], offering nothing substantially different than this one. The problem is that content is almost never useful, it just looks nice at first skim through. The authors, at no real fault of their own, are just eager novices that rarely have new perspective to add to a topic. It's not uncommon to find huge conceptual errors (or at least gaps) in the content there. I personally encourage everyone at every level to write about what they can, but the issue is that TDS has manipulated this population of eager data scientists in order to dominant search results on nearly every single topic they can cover related to DS, which has made searching for anything tedious. Compare this post to the fantastic work of Jay Alammar [1]. Jay's post is truly excellent, covering a lot of interesting details about word2vec and providing excellent visuals as well. I'm assuming TDS will fold as soon as DS stops being a "hot" topic (which I think we'll be in the relatively near future), and will…

I've just recently been on a journey to understand transformers in and out, here are the resources I've found managed to drill it into my head: 1. Chapters 7, 9, 10 https://web.stanford.edu/~jurafsky/slp3/ This was really useful to really build up to the concepts of attention (although the actual attention section is still brief). 2. https://jalammar.github.io/visualizing-neural-machine-translation-mech… This was great for visualization and understanding that attention wasn't exclusively for transformers and actually was for RNNs first. 3. https://jalammar.github.io/illustrated-transformer/ Getting to understand how transformers actually work visually. 4. https://www.youtube.com/watch?v=S27pHKBEp30 This lecture by Leo Dirac was extremely helpful to finish off with, not only because it actually includes some pseudocode but it also revisits some key topics and covers why transformers are needed. One of the big confusion points for me was that the concept of ATTENTION and SELF-ATTENTION are not the same thing. Hope this helps.

That's correct - this was designed primarily for the ~12,000 people attending NeurIPS so they can physically find the poster and meet the authors. If you wish to navigate the NeurIPS papers more digitally and you aren't signed up, I'd refer you to https://jalammar.github.io/assets/neurips_2025.html

Here is another take on visualizing transformers from Georgia Tech researchers: https://poloclub.github.io/transformer-explainer/ The Illustrated Transformer: https://jalammar.github.io/illustrated-transformer/ Sebastian Raschka, PhD has a post on the architectures: https://magazine.sebastianraschka.com/p/from-gpt-2-to-gpt-oss-analyzin… This HN comment has numerous resources: https://news.ycombinator.com/item?id=35712334

I stumbled on this, I think here on hacker news, and appreciate the amount of effort the author put into providing visualizations. https://jalammar.github.io/illustrated-stable-diffusion/ Whats the goal here? Do your friends express interest in using AI within their practice? Technical or not, an artist doesn't need to understand it to achieve that... Many years ago before I better understood it, myself and other artist used generative AI to create elements of our works and no one knew or cared that we did.