Brendan Bycroft

Amazing work! Reminded me of LLM Visualization (https://bbycroft.net/llm) except this is a lot easier to wrap my head around and that I can actually run the training loops, which makes sense given the simplicity of the original microgpt. To give a sense of what the loss value means, maybe you can add a small explainer section as a question and add this explanation from Karpathy’s blog: > Over 1,000 steps the loss decreases from around 3.3 (random guessing among 27 tokens: −log(1/27)≈3.3) down to around 2.37. to reiterate that the model is being trained to predict the next token out of 27 possible tokens and is now doing better than the baseline of random guess.

For the visual learners, here's a classic intro to how LLMs work: https://bbycroft.net/llm

Innovation in data visualization? From a purely utilitarian view, the purpose of data visualization is to present data in a way people can understand it. If you’re constantly changing the method of visualizing the same thing it’s harder to do that. Sometimes a bar chart is best. As far as cool visualizations go (that are better served as nonstandard visualizations) there are two recent ones that come to mind: https://youtu.be/TkwXa7Cvfr8 (Especially around 16:56) https://bbycroft.net/llm

Illustrated Transformer is amazing as a way of understanding the original transformer architecture step-by-step, but if you want to truly visualize how information flows through a decoder-only architecture - from nanoGPT all the way up to a fully represented GPT-3 - nothing beats this: https://bbycroft.net/llm

The project links to [1], which is absolutely amazing as well. [1] https://bbycroft.net/llm

https://bbycroft.net/llm This was posted on HN a while ago and led to some great discussion. Myself and others agreed that this type of stateful visualization was _way_ more effective at conceptualizing how an LLM works than reading code or stepping through a debugger.

https://www.youtube.com/@algorithmicsimplicity - that series cleared up the fundamental question about transformers I couldn't find an answer for in many recommended materials. Here's also nice tour de building blocks, which could also double as transformers/tensorflow API reference documentation: https://www.youtube.com/watch?v=eMXuk97NeSI&t=207s The #1 visualization of architecture and size progression: https://bbycroft.net/llm

Probably the best I've found: https://bbycroft.net/llm

Focusing on Deep Learning specifically: - Most LLMs currently use the transformer architecture. You can learn about this visually (https://bbycroft.net/llm), or through this blog post (https://jalammar.github.io/illustrated-transformer/), or through any number of Andrej Karpathy's blog posts and materials. - To stay on top of papers that get published every week, I read a summary every Sunday: https://github.com/dair-ai/ML-Papers-of-the-Week - To learn more about the engineering side of it, you can join Discord servers such as EleutherAI's, or follow GitHub discussions of projects like llama.cpp Personally I think the best way to develop per unit time is probably to try to re-implement some of the big papers in the field. There's a clear goal, there are clear signs of success, there are many implementations out there for you to check your work against and compare and learn from. Good luck!

You can see how an LLM works here https://bbycroft.net/llm they are not magic.

I'd suggest the LLM visualiser (1) by Brendan Bycroft. I'm a visual learner and it helped me understand LLMs better than anything else ever could. (1) https://bbycroft.net/llm

https://bbycroft.net/llm has 3d Visualization of tiny example LLM layers that do a very good job at showing what is going on (https://news.ycombinator.com/item?id=38505211)

I'm more looking at the problem more like code https://bbycroft.net/llm My immediate thought is when the model responds "Oh I'm thinking about X"... that X isn't from the input, it's from attention, and thinking this experiment is simply injecting that token right after the input step into attn--but who knows how they select which weights

Here's a compelling visualization of the functioning of an LLM when processing a simple request: https://bbycroft.net/llm This complements the detailed description provided by 3blue1brown

Also relevant would be this interactive visualization: https://bbycroft.net/llm Prior discussion: https://news.ycombinator.com/item?id=38505211

Someone put this link up on another discussion the other day and I found it really fascinating: https://bbycroft.net/llm I believe energy minimization is literal, just look at the size of that thing and imagine the power bill.

I find this visualization is useful for showcasing the difference between the different model generations: https://bbycroft.net/llm . The difference in scale is massive. It was discussed here some time ago: https://news.ycombinator.com/item?id=38505211

So if this article is of interest to you then this was a good discussion of a nice 3D visualization of a small LLM (NanoGPT) in motion ... LLM Visualization https://news.ycombinator.com/item?id=38505211 (https://bbycroft.net/llm)

In case you missed it(it was on the frontpage here a couple of days ago), play around with this awesome 3D visualisation and animation to get a basic understanding: https://bbycroft.net/llm

I think you are confused on how transformers work. You are confusing between static embeddings and the dynamic, contextual outputs calculated by transformers. While initial embeddings (representations BEFORE processing through the transformer layers) are relatively static, the output embeddings from each layer of the transformer are highly dynamic and context-dependent. They incorporate information from the entire input sequence via the attention mechanisms. So when transformers process an input sequence, they don't just look at each token in isolation but consider the ENTIRE sequence's context through complex inter-token relationships. This makes the technique of caching outputs non-trivial and context-specific. The sequential processing does not imply independence of tokens but underscores the integral role of context and sequence in generating accurate and coherent outputs. This should help you understand better how transformers work: https://bbycroft.net/llm