A
Andrej Karpathy
karpathy.github.io · Personal blog · on HN since 2014
Andrej Karpathy writes tutorials and essays on neural networks, deep learning, and training models, including a minimal GPT implementation.
What HN saysHN commenters call it the best AI blog to follow, and praise the microgpt post as a clear way to understand backprop and LLMs.
287HN people shared it
210linked it in comments
99stories submitted
7.3Kstory points
3 months agolast shared
2014people per year2026
Most-shared posts
- 1The Unreasonable Effectiveness of Recurrent Neural Networks100 people · 1.2K points · first shared 2015 · 207 comments on HN
- 2Hacker's Guide to Neural Networks30 people · 1.5K points · first shared 2014 · 61 comments on HN
- 3A Recipe for Training Neural Networks22 people · 424 points · first shared 2019 · 32 comments on HN
- 4The state of Computer Vision and AI: we are really, really far (2012)20 people · 201 points · first shared 2015 · 121 comments on HN
- 5Andrej Karpathy – Software 2.020 people · 28 points · first shared 2021 · 10 comments on HN
- 6What I learned from competing against a ConvNet on ImageNet (2014)16 people · 120 points · first shared 2014 · 5 comments on HN
- 7Deep Reinforcement Learning: Pong from Pixels14 people · 363 points · first shared 2016 · 13 comments on HN
- 8A from-scratch tour of Bitcoin in Python13 people · 1.2K points · first shared 2021 · 277 comments on HN
- 9Microgpt13 people · 6 points · first shared 2026 · 1 comments on HN
- 10Homepage8 people · first shared 2015
- 11What a Deep Neural Network thinks about selfies7 people · 264 points · first shared 2015 · 50 comments on HN
- 12Short Story on AI: Forward Pass7 people · 94 points · first shared 2021 · 17 comments on HN
- 13Biohacking Lite6 people · 764 points · first shared 2020 · 362 comments on HN
- 14Short Story on AI: A Cognitive Discontinuity6 people · 71 points · first shared 2015 · 13 comments on HN
- 15A Survival Guide to a PhD5 people · 326 points · first shared 2016 · 124 comments on HN
What HN people wrote when they linked it
striker_axel · 2020-12-26
I follow these three for AI progress and concepts
- http://karpathy.github.io/ (best one)
- http://bjlkeng.github.io/
- https://lilianweng.github.io/lil-log/
weishigoname · 2017-07-25
some machine learning blog I like to follow is http://karpathy.github.io
fchollet · 2015-06-10
Keras http://keras.io/ is a neural networks library for Python that is rather easy to get started with compared to the alternatives.
I also recommend the tutorials on http://deeplearning.net/tutorial/ (Python / Theano)
And Andrej Karpathy's blog is also a great resource for explanations of deep learning concepts in simple terms: https://karpathy.github.io/
busfahrer · 2026-07-17
For anybody else trying to grok backprop, I found it very helpful to look at karpathy's microgpt, a barebones LLM in 200 lines of python. The autograd itself is just 40 lines, and reading that code and then trying to do a clean room implementation of it is a great exercise:
https://gist.github.com/karpathy/8627fe009c40f57531cb18360106ce95
https://karpathy.github.io/2026/02/12/microgpt/
kzrdude · 2026-06-08
I think Karpathy's microgpt blogpost is the best in this genre in a long time, and it also includes a multi layer perceptron. It's a step up in the hierarchy, so reading both is helpful, of course.
https://karpathy.github.io/2026/02/12/microgpt/
anonnon · 2026-03-12
> Among many, many other things, he invented the term "vibecoding".
Yeah, that's a great reason to hate him, but the person you're responding to asked why his Twitter braindroppings belong on the front page.
It should be stated, again, that Karpathy completely missed the boat on LLMs, leaving OpenAI before they developed ChatGPT, and that he convinced Tesla to pursue a visual-only, no LIDR approach to FSD that doesn't work and probably won't ever work until after LIDR-based systems have already solved FSD.
Karpathy is the AI-equivalent of Sam Altman, who, for whatever reason, only fails upward. I think many HNers like him because he reminds them of themselves. Look at this bullshit and tell me it doesn't read like something the average HNer would write: http://karpathy.github.io/2020/06/11/biohacking-lite/
batterylake · 2026-03-09
Very insightful! I found the section on killing papers to be a helpful reminder. As a Ph.D. student, this can be particularly challenging as your environment expects somewhat steady progress (annual reviews, advisor meetings, etc.), and you're encouraged to finish papers rather than starting over.
This might also be of interest: https://karpathy.github.io/2016/09/07/phd/
gchadwick · 2025-11-02
Karpathy's contribution to teaching around deep learning is just immense. He's got a mountain of fantastic material from short articles like this, longer writing like https://karpathy.github.io/2015/05/21/rnn-effectiveness/ (on recurrent neural networks) and all of the stuff on YouTube.
Plus his GitHub. The recently released nanochat https://github.com/karpathy/nanochat is fantastic. Having minimal, understandable and complete examples like that is invaluable for anyone who really wants to understand this stuff.
jszymborski · 2025-09-21
If you are looking to learn about LSTMs and RNNs, I can't recommend this post enough:
https://colah.github.io/posts/2015-08-Understanding-LSTMs/
Another great one (just on RNNs) is:
http://karpathy.github.io/2015/05/21/rnn-effectiveness/
MasterScrat · 2025-02-06
His intro to RL (not for LLM) blog post is a great read FYI
https://karpathy.github.io/2016/05/31/rl/
killerstorm · 2024-08-19
A small language model can be quite like Markov chain, e.g. the one from Karpathy's 2015 post was pretty nonsensical: https://karpathy.github.io/2015/05/21/rnn-effectiveness/
So you can just scale down if it still makes sense.
Also you get a lot more from the base model. GPT-3 was versatile as it could continue any context. Modern LLMs are try-hards. If you want to generate humor with LLM really worth going for base model with multiple examples in the prompt.
bjourne · 2023-05-25
Almost to the date seven years ago: http://karpathy.github.io/2015/05/21/rnn-effectiveness/
"We downloaded the raw Latex source file (a 16MB file) and trained a multilayer LSTM. Amazingly, the resulting sampled Latex almost compiles. We had to step in and fix a few issues manually but then you get plausible looking math, it’s quite astonishing:"
My emphasis.
cultofmetatron · 2023-05-16
> I recently watched Andrej Karpathy's introduction to neural networks
I'm currently interested in delving into neural networks myself and find a lot of the information inaccessible atm. decided to see if I could find this resource you're taking about.
I'm guessing you're referring to this site? https://karpathy.github.io/
Alpi · 2023-05-12
I love all the series about writing your own X in 100 lines of code.
It gives you the understanding of technology and removes a lot of unnecessary details.
The great examples of this are
'A from-scratch tour of Bitcoin in Python'
https://karpathy.github.io/2021/06/21/blockchain/
and 'Let's build GPT: from scratch, in code, spelled out'
https://youtu.be/kCc8FmEb1nY
from Andrej Karpathy
I wonder if anybody tried to collect all such projects together and built his own 'Internet in just 100 lines of code'
pfd1986 · 2023-01-20
This is great. Another good source is Karpathy's recipe: https://karpathy.github.io/2019/04/25/recipe/
Does anyone know an updated version in the age of Transformers?
rsfern · 2023-01-20
This looks like a great and really comprehensive resource!
Karpathy’s recipe for training neural nets is one of the only similar documents I’ve come across http://karpathy.github.io/2019/04/25/recipe/
There’s also the (older) devil in the details papers focused on computer vision. I’d love to read something like this on modern methods like transformers. https://arxiv.org/abs/1405.3531
cdzm · 2023-01-18
Karpathy also has a great Recipe for Training Neural Networks:
http://karpathy.github.io/2019/04/25/recipe/
Invictus0 · 2021-11-07
> Diet and exercise are indeed not the path to weight loss. This is well known: most fad diets work this way in some fashion or another, and it’s well known that most fad diets fail.
I think you're misunderstanding what I'm saying. I'm not suggesting people go on keto or weight watchers. Those fad diets don't necessarily fail because they're ineffective, although they probably are--they fail because they are highly prescriptive and restrictive and it's difficult for people to actually execute the diet.
What I'm saying is that reducing total food intake for 3 days creates a lasting decrease in appetite. You can prove this to yourself by skipping breakfast for a few days: after a while, you will simply not be hungry at breakfast time.
The best post I've ever seen on weight loss comes from Andrej Karpathy, head of AI at Tesla: https://karpathy.github.io/2020/06/11/biohacking-lite/
nl · 2021-07-06
Co-pilot produces original code (as in code that has never been written before). It's not just combining snippets.
This should surprise no one who has seen the evolution of language models. Take a look at Kapathy's great write up from way back in 2015[1]. This generates Wikipedia syntax from a character-based RNN. It's operating on a per-character basis, and it doesn't have sufficient capacity to memorise large snippets. (The Paul Graham example spells this out: 1M characters in the dataset = 8M bits, and the network has 3.5M parameters).
Semantic arguments about "is this intelligence?" I'll let others fight.
[1] http://karpathy.github.io/2015/05/21/rnn-effectiveness/
king_magic · 2021-02-01
Good read. Reminds me a lot of Andrej Karpathy's "A Recipe for Training Neural Networks", which has served me extremely well in my AI career.
https://karpathy.github.io/2019/04/25/recipe/
f00zz · 2020-07-28
Right, I admit that I don't know the first thing about ML, so I tried an experiment.
Consider a language with the tokens "{[()]}" and the following grammar:
S := S S | '{' S '}' | '[' S ']' | '(' S ')' | <empty>
That is, "[()]" and "[]()" are valid sequences, but "[(])" or "))))" aren't. A child would quickly figure out the grammar if presented some valid sequences.
I generated all 73206 valid sequences with 10 tokens and used it as input to the RNN text generator code at http://karpathy.github.io/2015/05/21/rnn-effectiveness/. After 500,000 iterations I'm still getting invalid sequences.
Am I doing something stupid, or is a RNN text generator weaker than a child (or a pushdown automaton)? Is GPT fundamentally more powerful than this?
melvinroest · 2020-06-14
I'm just an enthusiast. With that said, I think the author's first image image really captures the core of forward propagation and back propagation. For people who want to get more of a feel on autodiffing, I find these 2 sources amazing [1, 2]. They don't use the word autodiffing, but looking at it, I can see it's the same thing.
I once made an attempt at animating feed forward propagation [3]. If I'd animate back propagation, I'd probably make an animation on autodiffing as well as that is the most intuitive way I know on how to explain back propagation right now. It's still on my big to do list ;-)
[1] http://karpathy.github.io/neuralnets/
[2] 3Blue1Brown does in episode 4 of his Deep Learning series: https://www.youtube.com/watch?v=tIeHLnjs5U8
[3] https://melvinroest.github.io/Interactive-Neural-Networks/
siekmanj · 2019-05-22
Andrej Karpathy has a pretty good introductory article to RNNs here: http://karpathy.github.io/2015/05/21/rnn-effectiveness/
He has some code which is pretty easy to follow to go along with the article: https://gist.github.com/karpathy/d4dee566867f8291f086
jefftk · 2019-02-27
> indistinguishable from a Markov chain generator
A Markov chain generator wouldn't:
* Capitalize the first word of each line
* Make lines of approximately the right length
* Mark text with who is to speak it
While this is just a toy example, it's powerful enough to start showing the ways RNNs can produce text that looks superficially correct.
(Generating Shakespeare is actually one of the examples given in the classic http://karpathy.github.io/2015/05/21/rnn-effectiveness/)
stared · 2018-08-01
Or: Elf or prescription drug: http://quizzes.howstuffworks.com/quiz/drug-or-tolkien-elf-quiz
As a side note - I would love to see something like that automatically generated (with LSTMs or), in the spirit of http://karpathy.github.io/2015/05/21/rnn-effectiveness/ (and example for generating names: https://pytorch.org/tutorials/intermediate/char_rnn_generation_tutoria…
alexmuro · 2017-11-13
Personally I recommend Stanford CSI 231n
http://cs231n.stanford.edu/
Its specifically geared towards visual recognition, but it starts with the basics of machine learning and moves on to feed forward nets and covnets and covers RNNs and attention towards the end.
The assignments are a great set of jupyter notebooks that really get your hands on the material and you can find a number of peoples complete assignments on github just by searching.
The lectures are available online as well https://www.youtube.com/playlist?list=PL3FW7Lu3i5JvHM8ljYj-zLfQRF3EO8s…
I've done hinton's and Ngs courses and as someone who already has a non-ai development background I found this to be the best introduction. Its really an extension of Andrej Karpathy's Neural Nets for Hackers (http://karpathy.github.io/neuralnets/)
parths · 2017-08-16
A large majority of human errors come from fine-grained categories(such as correctly identifying two similar cat species) and class unawareness. I would recommend this article by Andrej Karpathy, where he talks about his learning from competing against GoogLeNet: http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-aga…
Joris1225 · 2017-06-10
In Andrej Karpathy's excellent blogpost on RNNs[1], he links to some of the code he used for visualisation[2]. I have done something similar. In general: put your the activations of each cell memory over time through tanh and color it based on that.
[1] http://karpathy.github.io/2015/05/21/rnn-effectiveness/
[2] http://cs.stanford.edu/people/karpathy/viscode.zip
jkbschwarz · 2017-02-07
Yes happy to share once the evaluation period is over, but sadly I dont think Im allowed to share beforehand. Will have to check when but I expect it should be in approximately 2 weeks.
In the meantime, you might want to check out this excellent blog post http://r2rt.com/recurrent-neural-networks-in-tensorflow-ii.html. This will provide you with skeleton code to implement a character level generative model (similar to Practical 3, Task 2 with the exception that there you will generate words and not characters). Andrej Karpathy's blog post on LSTMs is also excellent and I believe he also provides the code in his repository as well http://karpathy.github.io/2015/05/21/rnn-effectiveness/
florianletsch · 2016-10-06
You probably know this already, but if you are just looking for an illustrated demonstration of what RNNs are capable of in the text domain, probably the best brief article is the post by Andrej Karpathy about [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/).