EleutherAI Blog
blog.eleuther.ai · Company blog · on HN since 2021
Blog of EleutherAI, an open research group that trains and releases large language models, such as GPT-NeoX-20B.
What HN saysCommenters recommend its Transformer Math post for estimating memory from parameter count, plus its first-year retrospective and posts on NeoX and long context.
22HN people shared it
7linked it in comments
17stories submitted
478story points
1 year agolast shared
2021people per year2026
Most-shared posts
- 1EleutherAI: Transformer Math 1013 people · 1 points · first shared 2023 · discussion on HN
- 2Announcing GPT-NeoX-20B2 people · 200 points · first shared 2022 · 70 comments on HN
- 3EleutherAI One Year Retrospective2 people · 142 points · first shared 2021 · 43 comments on HN
- 4Rotary Embeddings: A Relative Revolution2 people · 7 points · first shared 2021 · 1 comments on HN
- 5Why Release a Large Language Model?2 people · 3 points · first shared 2021 · discussion on HN
- 6Pile-T51 people · 59 points · first shared 2024 · 15 comments on HN
- 7Yi-34B, Llama 2, and common practices in LLM training1 people · 41 points · first shared 2024 · 3 comments on HN
- 8EleutherAI announces it has become a non-profit1 people · 12 points · first shared 2023 · 2 comments on HN
- 9The Common Pile v0.11 people · 4 points · first shared 2025 · discussion on HN
- 10Llemma: An Open Language Model for Mathematics1 people · 2 points · first shared 2023 · 1 comments on HN
- 11Exploratory Analysis of TRLX RLHF Transformers with TransformerLens1 people · 2 points · first shared 2023 · discussion on HN
- 12The Foundation Model Development Cheatsheet1 people · 2 points · first shared 2024 · discussion on HN
- 13The Practitioner's Guide to the Maximal Update Parameterization1 people · 1 points · first shared 2024 · discussion on HN
- 14Minetester: A fully open RL environment built on Minetest1 people · 1 points · first shared 2023 · discussion on HN
- 15Experiments in Weak-to-Strong Generalization1 people · 1 points · first shared 2024 · discussion on HN
What HN people wrote when they linked it
vikp · 2023-08-13
I clicked because I thought they were defining LLM developer as "someone training LLMs", but instead they define it as "someone integrating LLMs into their application".
If you also had the same initial thought as me, this is an excellent article - https://blog.eleuther.ai/transformer-math/ .
andreyk · 2022-02-06
I'd recommend reading the blog post about their first year, it's really fun and informative
https://blog.eleuther.ai/year-one/
smallerize · 2025-03-06
Well I can't be positive, but it looks like some of the factors that support a long context length might be set wrong. https://blog.eleuther.ai/yarn/
evanchisholm · 2023-05-31
It's fairly simple to estimate ram requirements based on parameter count:
https://blog.eleuther.ai/transformer-math/
v64 · 2023-03-02
not nano, Neo, and its successor NeoX, one of the largest public GPT models available https://blog.eleuther.ai/announcing-20b/
peaslock · 2022-12-18
OpenAI named their models alphabetically in order of the model size:
https://blog.eleuther.ai/gpt3-model-sizes/
Davinci performed similarly to a 175B GPT-3 model and the next smallest model is Curie which corresponds to a 6.7B GPT model.
Now, there have been two GPT models fine tuned on Code, namely davinci-codex and cushman-codex. Since cushman != curie, it is presumably based on the GPT-model in between, namely the one with 13B.
See also A.2 here: https://arxiv.org/pdf/2204.00498.pdf#page=6
stellaathena · 2021-10-18
[Disclaimer: I am an author of the above paper and played a rather minimal role. I am also a prominent member of EleutherAI.]
"Instruction-tuning" is clearly in the air. Simultaneous work at Google (released less than two weeks ago) on a model they call FLAN can be found here: https://ai.googleblog.com/2021/10/introducing-flan-more-generalizable.…
EleutherAI attempted to do something similar several months ago, but didn't succeed: https://blog.eleuther.ai/tuning-on-eval-harness/
A careful analysis of the similarities and differences between the three approaches would be likely highly beneficial to the community.