Sebastian Raschka

Not Sebastian (who I assume is the OP), but his blog/substack is also a great resource https://magazine.sebastianraschka.com/

I thoroughly recommend Sebastian's newsletter "Ahead Of AI". Link on his website: https://sebastianraschka.com/

Thanks for putting this together. I recommend you check out https://sebastianraschka.com/ the author of Python Machine Learning. He's great at explaining things, wrote the bets intro to ML book IMHO and is a good Twitter follow. I also humbly submit my own guide, from the perspective of how to approach studying it amidst all the resources available: http://karlrosaen.com/ml/

http://sebastianraschka.com/ has some great articles for machine learning in Python.

Similarly, I’ve been working through Raschka’s Build a Large Language Model (From Scratch) https://sebastianraschka.com/llms-from-scratch/ and it’s been well worth it (particularly paired with his explainer videos)

I recommend his book, "Build a Reasoning Model (From Scratch)", which is also linked in the article. https://sebastianraschka.com/books/#build-a-reasoning-model-from-scrat…

This is amazing, I just spent some time scrolling through these, most of the evolution is about inference cost not capability. GQA, MoE routing, sliding window attention, all trading theoretical capacity for practical efficiency. Tbh might be the last generation of architectures designed entirely by humans. I dug into that (1) and might add another paragraph based on this if I find the time. The Big LLM Architecture Comparison (2) by Sebastian Raschka already inspired my ograph image for the blog -thanks again! (1) https://philippdubach.com/posts/the-last-architecture-designed-by-hand… (2) https://magazine.sebastianraschka.com/p/the-big-llm-architecture-compa…

I'm surprised nobody has mentioned Ahead Of AI https://magazine.sebastianraschka.com/

At the current speed of research, any book about transformers would be outdated before the ink it was printed on has dried. You got to read the papers if you want to keep up with the bleeding edge. The attention is all you need paper that tikkun linked is great, but not exactly a gentle start. This might help a little: https://sebastianraschka.com/blog/2023/llm-reading-list.html

From https://sebastianraschka.com/ "I used to hold a position as an Assistant Professor of Statistics at the University of Wisconsin-Madison (on a tenure track from 2018-2025). However, with a heavy heart, I recently resigned in 2023 to concentrate fully on my work at the Lightning AI startup, which I had joined in January 2022."

Python Machine Learning is a great book to start the author made a great curriculum if you would like to follow it. https://sebastianraschka.com/faq/docs/ml-curriculum.html This is mostly ML if you would like to dive into deep learning I think fast.ai is the best course for anyone with programming experience and you can also use the deeplearning.net Tutorial as a side reference. If you have a practical experience and would love to understand the theory behinds it then Deep Learning Book is the Bible.

I posted this article some while ago, and I thought it may not be a bad idea to archive it on arXiv. What do you think? The (Latex) PDF version would look like this: http://sebastianraschka.com/PDFs/articles/naive_bayes_1.pdf I just signed up on arXiv and see that the catch is that I'd need 2 recommendations for the categories Computer Science -> Learning or Statistics -> Machine Learning. Would anyone here who would want to give me a recommendation? I would really appreciate it!

This is a very good question and the answers are different for different models. --- Deepseek v4 Pro has three modes: no thinking, thinking ("high"), very good thinking ("max"). Max is implemented as follows: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main/encoding/… > When reasoning_effort="max" is set, a special prefix is prepended at the very beginning of the prompt (before the system message) to instruct the model to maximize its reasoning depth: > > Reasoning Effort: Absolute maximum with no shortcuts permitted. You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios. Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked. --- Inkling, apparently, trained their model to treat effort level as "how cheap is one reasoning token": https://magazine.sebastianraschka.com/p/controlling-reasoning-effort-i… > During large-scale RL, they did two things for each sample: > > 1. Specified the desired effort level in the system message. > 2. Adjusted the cost assigned to each generated token. > [...] > > Then, at inference time, Inkling receives a system message such as Thinking effort level: 0.8, and adjusts its token usage accordingly.

I'm not sure what is the state-of-the-art in sentiment analysis. However, GPT-3.5 and GPT-4 work great for it with few shot learning. You give a representative set of examples in the prompt and it works pretty well. On the topic of classification, large language models such as BERT are the state-of-the-art. But, there was a paper recently at the ACL [1] which showed that a simple approach of using gzip compression + k-nearest neighbour achieved similar level of performance. Later on, their KNN evaluation methodology came under question because instead of doing KNN with k=2 they reported accuracy based on top-2. Even then, what it shows is that we don't need LLMs for simple tasks such as classification, where traditional techniques still work well and are much cheaper to run. I'm sure some people might say that we should just discard that paper because it had that error. But Sebastian Raschka ran it on the IMDB movie reviews dataset and reported 71% accuracy [3]. I wrote an explainer article on it [2]. [1] https://aclanthology.org/2023.findings-acl.426.pdf [PDF] [2] https://codeconfessions.substack.com/p/decoding-the-acl-paper-gzip-and… [3] https://magazine.sebastianraschka.com/p/large-language-models-and-near…

Here are some high level answers: "7B" refers to the number of parameters or weights for a model. For a specific model, the versions with more parameters take more compute power to train and perform better. A foundational model is the part of a ML model that is "pretrained" on a massive data set (and usually is the bulk of the compute cost). This is usually considered the "raw" model after which it is fine-tuned for specific tasks (turned into a chatbot). "8K length" refers to the Context Window length (in tokens). This is basically an LLM's short term memory - you can think of it as its attention span and what it can generate reasonable output for. "1.5T tokens" refers to the size of the corpus of the training set. In general Wikipedia (or I suppose ChatGPT 4/Bing Chat with Web Browsing) is a decent enough place to start reading/asking basic questions. I'd recommend starting here: https://en.wikipedia.org/wiki/Large_language_model and finding the related concepts. For those going deeper, there are lot of general resources lists like https://github.com/Hannibal046/Awesome-LLM or https://github.com/Mooler0410/LLMsPracticalGuide or one I like, https://sebastianraschka.com/blog/2023/llm-reading-list.html (there are a bajillion of these and you'll find more once you get a grasp on the terms you want to surf for). Almost everything is published on arXiv, and most is fairly readable even as a layman. For non-ML programmers looking to get up to speed, I feel like Karpathy's…

> Are they literally adding a hidden system prompt that says "effort level: $level" ? Yes. https://magazine.sebastianraschka.com/p/controlling-reasoning-effort-i…

Here’s some of those sources that were apparently hard to find: https://openai.com/index/openai-o1-system-card/ https://arxiv.org/html/2410.10630v1 https://magazine.sebastianraschka.com/p/understanding-reasoning-llms

Why didn't this author compare Llama 3 with GLM 5.2 (released 1 week ago) which is a more standard attention based LLM? To compare 2 separate families of LLMs and then pointing out that they are different is not a surprising result and detracts from the point the author is trying to make. https://sebastianraschka.com/llm-architecture-gallery/?compare=llama-3… If you look at it, the diagrams are very similar, but the main differences are that the feedforward is replaced with a MoE (router to multiple feedforwards) and the model has a different attention implementation.

Qwen 3.5 and above use Gated DeltaNet which alternate attention and SSM layers: https://sebastianraschka.com/llms-from-scratch/ch04/08_deltanet/

Coincidentally, I just started on Build a Large Language Model (From Scratch), a repo/book/course by Sebastian Raschka [0][1][2]. Maybe it is a good problem to have to have to decide which learning resource to use. [0] https://github.com/rasbt/LLMs-from-scratch [1] https://www.manning.com/books/build-a-large-language-model-from-scratc… [2] https://magazine.sebastianraschka.com/p/coding-llms-from-the-ground-up

Using larger contexts often costs more in the APIs or consume more of your quota but this is becoming less of a problem with models using more clever attention mechanisms and not just full attention on all layers. You can look at: https://sebastianraschka.com/llm-architecture-gallery/ and see how much things have changed.

There is https://magazine.sebastianraschka.com/p/technical-deepseek which shows an evolution in deepseek family

I think this blog post in particular might be helpful here https://sebastianraschka.com/blog/2023/self-attention-from-scratch.htm…

Here is another take on visualizing transformers from Georgia Tech researchers: https://poloclub.github.io/transformer-explainer/ The Illustrated Transformer: https://jalammar.github.io/illustrated-transformer/ Sebastian Raschka, PhD has a post on the architectures: https://magazine.sebastianraschka.com/p/from-gpt-2-to-gpt-oss-analyzin… This HN comment has numerous resources: https://news.ycombinator.com/item?id=35712334

also https://magazine.sebastianraschka.com/p/understanding-multimodal-llms

The other recent improvement suggested for LoRA is DoRA: https://magazine.sebastianraschka.com/p/lora-and-dora-from-scratch. It really does seem to strongly outperform LoRA - see also https://www.answer.ai/posts/2024-04-26-fsdp-qdora-llama3.html

It's quadratic, because of the dot product in the attention mechanism. You can use K-V Caching to get rid of a lot of the quadratic runtime that comes from redundant matrix multiplications, but after you have cached everything, you still need to calculate the dot product k_i * q_j with i,j being index of the tokens. With n tokens, you will get O(n*n). But you have to remember that this is only n^2 multiplications. It's not exactly the end of the world at context sizes of 32k, for example. It only gets nasty in the hundred thousands to millions. Here is the source I used: https://sebastianraschka.com/blog/2023/self-attention-from-scratch.htm…

You would fine tune a pretrained LLM because those books are written in English. And natural languages are in flux, and the corpuses that describe them are not neutral, so you can impose fairness after the fact. Sebastian Raschka has written some relevant popular articles, like: https://magazine.sebastianraschka.com/p/understanding-large-language-m… https://magazine.sebastianraschka.com/p/finetuning-large-language-mode…

This note contains four papers for "historical perspective"... which would usually mean "no longer directly relevant", although I'm not sure that's really what the author means. You might be looking for the author's "Understanding Large Language Models" post [1] instead. Misspelling "Attention is All Your Need" twice in one paragraph makes for a rough start to the linked post. [1] https://magazine.sebastianraschka.com/p/understanding-large-language-m…