Evan Miller
evanmiller.org · Personal blog · on HN since 2009
Evan Miller is a programmer and statistician who writes about applied math, A/B testing, ranking systems and user ratings.
What HN saysCommenters call his posts on user ratings excellent, especially "How Not To Sort By Average Rating," and recommend his A/B testing tools.
301HN people shared it
155linked it in comments
218stories submitted
10Kstory points
8 months agolast shared
2009people per year2026
Most-shared posts
- 1How Not To Sort By Average Rating76 people · 2.1K points · first shared 2009 · 156 comments on HN
- 2How Not To Run An A/B Test37 people · 245 points · first shared 2010 · 23 comments on HN
- 3You can't dig upwards10 people · 165 points · first shared 2014 · 144 comments on HN
- 4Ranking items with star ratings10 people · 5 points · first shared 2014 · discussion on HN
- 5Sample Size Calculator: How many subjects are needed for an A/B test?10 people · 1 points · first shared 2013 · discussion on HN
- 6The Mathematical Hacker9 people · 656 points · first shared 2012 · 132 comments on HN
- 7The Golden Football and the Economics of Groupon9 people · 63 points · first shared 2009 · 19 comments on HN
- 8Don't Kill Math9 people · 19 points · first shared 2012 · 2 comments on HN
- 9Why I program in Erlang (2012)8 people · 549 points · first shared 2012 · 136 comments on HN
- 10Statistical shortcomings in standard math libraries (and how to fix them)8 people · 104 points · first shared 2014 · 19 comments on HN
- 11Elixir RAM and the Template of Doom7 people · 503 points · first shared 2016 · 83 comments on HN
- 12A Formula for Bayesian A/B Testing7 people · 98 points · first shared 2014 · 21 comments on HN
- 13Evan's A/B Tools7 people · 5 points · first shared 2013 · discussion on HN
- 14Homepage7 people · first shared 2014
- 15The Joy of Erlang; Or, How to Ride a Toruk (2011)6 people · 172 points · first shared 2011 · 44 comments on HN
What HN people wrote when they linked it
sideway · 2025-02-17
I've used this approach in the past with great success:
https://www.evanmiller.org/ranking-items-with-star-ratings.html
All Evan Miller's posts on user ratings are excellent - https://www.evanmiller.org/ and search for "Mathematics of user ratings".
juramento · 2021-05-27
https://www.evanmiller.org
Discusses:
- Programming
- Applied Math
And some random stuff like: Evaluating splatoon's ranking system: https://www.evanmiller.org/evaluating-splatoons-ranking-system.html
Really recommend it ;)
mcphage · 2025-03-13
The article "How Not To Sort By Average Rating" by the same author (and also linked in this article) is really good, and definitely changed my thinking about any kind of "sort by best to worst" list: https://www.evanmiller.org/how-not-to-sort-by-average-rating.html
sebg · 2024-08-08
Hi
Have you looked into these two?
- Trustworthy Online Controlled Experiments by Kohavi, Tang, and Xu
- Statistical Methods in Online A/B Testing by Georgi Georgiev
Recommended by stats stackexchange (https://stats.stackexchange.com/questions/546617/how-can-i-learn-stati…
There's a bunch of other books/courses/videos on o'reilly.
Another potential way to approach this learning goal is to look at Evan's tools (https://www.evanmiller.org/ab-testing/) and go into each one and then look at the JS code for running the tools online.
See if you can go through and comment/write out your thoughts on why it's written that way. of course, you'll have to know some JS for that, but it might be helpful to go through a file like (https://www.evanmiller.org/ab-testing/sample-size.js) and figure out what math is being done.
NightMKoder · 2024-07-29
Would be great if it included sequential sampling as well: https://www.evanmiller.org/ab-testing/sequential.html . Especially given how A/B tests usually get run in product companies, a peek proof method helps quite a bit.
arjan_sch · 2022-04-14
Related to this, this is old but still a great blogpost:
https://www.evanmiller.org/elixir-ram-and-the-template-of-doom.html
abetusk · 2021-11-12
They have an article about K-star rating systems [0] which uses Bayesian approximation [1] [2] (something I know little to nothing about, I'm just regurgitating the article).
There's a whole section on their website that has different statistics for programmers, including rating systems [3].
[0] https://www.evanmiller.org/ranking-items-with-star-ratings.html
[1] https://en.wikipedia.org/wiki/Approximate_Bayesian_computation
[2] https://www.evanmiller.org/bayesian-average-ratings.html
[3] https://www.evanmiller.org/ ("Mathematics of user ratings" section)
m12k · 2021-11-09
Great idea! But please read https://www.evanmiller.org/how-not-to-sort-by-average-rating.html
yzan · 2021-09-03
An obligatory link to an article by Evan Miller about ranking: https://www.evanmiller.org/how-not-to-sort-by-average-rating.html
However, to get the following requirement from the article:
> The algorithm should not produce false negatives, the community should find all high-quality content.
it might be better to estimate the upper confidence bound, like in Upper Confidence Bound bandit, rather than the lower confidence bound.
hansvm · 2021-06-17
> we should look at the number of votes, divided by the number of views
Closer, but still not quite what you want probably or a few stray votes can make a massive impact just from discretization effects. What you really care about is which answer is "best" by some metric, and you're trying to infer that as best as possible from the voting history. Average votes do a poor job. Check out this overview from the interblags [0].
[0] https://www.evanmiller.org/how-not-to-sort-by-average-rating.html
axlee · 2020-01-14
I remember "How Not To Sort By Average Rating" by the same author, which I am confident helped shape thousands of recommendation/quality sorting algorithms (including Reddit or Yelp), over the years that followed its writing.
https://www.evanmiller.org/how-not-to-sort-by-average-rating.html
eddyg · 2019-10-11
Plenty.
Read https://www.evanmiller.org/why-im-learning-perl-6.html or the follow-up https://www.evanmiller.org/a-review-of-perl-6.html
Or, Why Perl 6 is the "Game of Thrones" of programming languages: https://techbeacon.com/app-dev-testing/why-perl-6-game-thrones-program…
Lots more where those came from...
yaj54 · 2019-01-04
About the ranking algorithms:
"techyaks": Primarily uses signals from stories and comments on Hacker News, with some youtube signals thrown in. This is effectively a "best talks according to hacker news" ranking.
"wilsonscore": Wilson confidence intervals [0] from youtube likes and dislikes. Previous discussion about this site when it had only the Wilson score ranking: https://news.ycombinator.com/item?id=17988464
[0]: https://www.evanmiller.org/how-not-to-sort-by-average-rating.html
eximius · 2018-09-26
I should open source my cobbled together implementation for react.
I will add that a couple great things for a dashboard using MAB are
1. Pairwise variant comparisons: i.e., P(A>B) for each pair. http://www.evanmiller.org/bayesian-ab-testing.html is an amazing resource on that.
2. Time-to-Confidence estimates: how long until we will have a credible interval of 5%? 1%? 0.5%?
These can help you cull variants and experiments. Sure, it's optimized so you don't have to for the sake of the test, but code cleanliness is nice to have.
samd · 2018-07-03
So damn true, it's so difficult to get statistically significant A/B test results, and the popular tools actively lead you astray. It's rare enough for a startup to have the traffic to meaningfully get results on their homepage before the heat death of the universe, let alone random landing or in-app pages.
I'd recommend anyone reading this who does or wants to do A/B tests read: https://www.evanmiller.org/how-not-to-run-an-ab-test.html It was one of the trickiest lessons I had to learn.
Once you realize you need to set your sample size in advance, you actually have to do the math to figure out the traffic you'll need. That's when reality hits you.
untangle · 2018-04-08
If you are willing to defer dismissing this tool out of hand as a p-value generator, you can get a better feel for how the author (Evan Miller) thinks about stats from his web site [1] and a presentation he gave [2]. I think you'll find that "Wizard" is not the product of whimsy.
[1] http://www.evanmiller.org/
[2] https://www.youtube.com/watch?v=TzJMFxj7GRI
rb808 · 2017-11-28
I followed a link to another post on the blog that was one of the best honest analyses of Erlang. I've been meaning to play with it a while, maybe these upcoming holidays. http://www.evanmiller.org/why-i-program-in-erlang.html
OTOH Kotlin is on that list too, its difficult to choose!
EvanMiller · 2017-11-16
I recommend the approach described in this article:
http://www.evanmiller.org/ranking-items-with-star-ratings.html
In this formulation, s_k equals utility. Like the Wilson score formula (and unlike the linked article), the provided equation takes into account the variance of the expected utility.
zitterbewegung · 2017-05-22
Reddit did a great job when they created a new ranking algorithm. They marketed as "best"[1] and they have an informational post for nontechnical people but in the post they linked to the technical explanation[2].
[1]https://redditblog.com/2009/10/15/reddits-new-comment-sorting-system/
[2] http://www.evanmiller.org/how-not-to-sort-by-average-rating.html
pselbert · 2016-12-07
Evan Miller did some very interesting analysis of the historical Reddit hot formula. It doesn't address cheating, but it does identify the details of the algorithm and some inherent faults.
http://www.evanmiller.org/deriving-the-reddit-formula.html
There is also the followup about ranking news items with upvotes:
http://www.evanmiller.org/ranking-news-items-with-upvotes.html
sesutton · 2016-10-19
Sorting by best score is equally useless. You get random products that have one 5 star review that are often nothing like what you actually searched for. A better method would be the one described here[0], which reddit uses for its "best" comment sorting.
[0]http://www.evanmiller.org/how-not-to-sort-by-average-rating.html
didip · 2016-06-22
I respect the desire to work on low level language like C. Programmers should not be afraid of it.
But that said, building a proper HTTP stack is not trivial.
If you want to use C language, then why not create Nginx module?
Nginx already solved the hard problems:
* HTTP parser
* Distribute work via event loop on multiple workers
* Useful load balancing strategies (not as great as HAProxy, but i am satisfied with it)
* Serious effort in dealing with CVE
* Widely used and battle tested
Here's a fine guide on how to write Nginx module in C:
http://www.evanmiller.org/nginx-modules-guide.html
btilly · 2016-03-08
The procedure I suggested starts with setting the maximum number of conversions before beginning the test. It is emphatically not the procedure that Evan Miller criticizes.
In fact see http://www.evanmiller.org/sequential-ab-testing.html for Evan Miller suggesting a similar procedure to mine, based on past conversations that I have had with him. The difference between that one and mine is that I focus on "make the best decision available, even though it may be only little better than a coin flip" while he tries to only decide when you can decide with confidence. In past conversations we have agreed that each other's procedures are reasonable, but they achieve different goals.
seiji · 2016-02-26
"practical" as in: the programing model fits the problem domain model of modern systems.
But, "practical" doesn't show up on everybody's radar much these days with so much effort focused around fads-of-the-half-decade.
Erlang is made by professionals with decades of experience who grow the system towards being the most stable, scalable, safe, and as understandable as possible. People use Erlang, solve problems, stay quiet, then make money using it.
On the other hand, the thought leader these days seems to be Go (even though it's made by out of touch weirdos (review: http://www.evanmiller.org/four-days-of-go.html)) and has a creepy cult/fanatical following more suited to tribal sportsball teams than what we would expect from professional technical work needing to be stable over the long term.
brobinson · 2015-10-31
I saw this post's headline and it reminded me of a "how not to sort by average rating" post I read many years ago. I looked it up on Google [1], read it, clicked through to this HN thread... lo and behold, the top comment was written by the author of the article I just looked up. Great stuff.
[1] http://www.evanmiller.org/how-not-to-sort-by-average-rating.html
raldi · 2015-07-29
This is why the ability to downvote is so often worth the you-hurt-my-feelings tradeoff.
http://www.evanmiller.org/how-not-to-sort-by-average-rating.html
NosajHN · 2015-07-24
1) Awesome! I've signed up. Hope to hear more.
2) Absolutely, thanks again for making the site. For what it's worth, the same author recently went into much more detail on how to make an optimal 'hot' algorithm, looking at Reddit as an example. I thought it was pretty interesting, even if it's a bit more complicated. http://www.evanmiller.org/deriving-the-reddit-formula.html
yang140 · 2015-06-09
Thanks for sharing this post. Bret mentioned this in his "Media for thinking unthinkable" talk that Mathematical notation itself is a great tool for making mathematical structure visible. They are like user interface, it tell things that language can't express. The birth of mathematical notation is considered to be birth of modern mathematics. With dynamic media provided by computer, there should be a better way to dynamically represent mathematical structure. With that in mind, I would also recommend Evan Miller's comment's on Bret Victor's "Kill Math" article called "Don't Kill Math". Evan empathize that besides dynamic simulation, we should also embrace "theory and calculation" to see the underlying structure.
http://worrydream.com/MediaForThinkingTheUnthinkable/
http://www.evanmiller.org/dont-kill-math.html
616c · 2015-04-24
Also interesting is that Evan Miller is the original developer of Chicago Boss, and he shows up a lot here. You will see it on the list of his software on the home page.
http://www.evanmiller.org/
I think he has since moved on. Notice some nice CB components, like ErlyDTL, are also his work. He shows up a lot here, so I thought people would be interested in it with regards to his other pieces about statistics, modeling, and other topics.
loopbit · 2015-04-22
He has a few very, very interesting articles. I would point out the following two:
- How not to sort by average ranking (http://www.evanmiller.org/how-not-to-sort-by-average-rating.html) is a recommended read on coursera's recommender systems course.
- Rank Hotness With Newton's Law of Cooling (http://www.evanmiller.org/rank-hotness-with-newtons-law-of-cooling.htm…