Ben Tilly

Good read, but pretty dense (or perhaps I'm just dense). Personally, I get more practical value from http://elem.com/~btilly/effective-ab-testing/.

This is a good article. However if anyone is interested in following it, they should know that when there are multiple bets available, the simple Kelly criterion falls apart. See http://www.elem.com/~btilly/kelly-criterion/ for a much fuller explanation of the result, and why it is true. (I keep meaning to automatically optimize the multiple bet version of it.)

Great tools! My goto reference is still the wonderful btilly presentation about how to A/B test properly, with nice examples and code snippets: http://elem.com/~btilly/effective-ab-testing/ He provided a full on javascript tool that isn't as polished but works great: http://elem.com/~btilly/effective-ab-testing/g-test-calculator.html

Thanks for this great article. Both this article and Effective A/B Testing (http://elem.com/~btilly/effective-ab-testing/ slide 51) say changing the proportions mid-test is bad. Can anyone elaborate on why this might be? For example, if I'm randomly showing my existing landing page or a new landing page to visitors at a 2/3-1/3 split and measuring CTR to the signup page, and I change the proportion to 1/2-1/2, why would that skew results?

Awesome project, I love the use of TrueSkill Through Time here! Really interesting that it beats out Elo as well. FiveThirtyEight actually posted their WC predictions earlier today [1]. Finding where the two models diverge is a fun exercise -- they have Morocco as a 40% favorite tomorrow while your model has Iran as a 46% favorite. It would be interesting to test the model by applying the Kelly Criterion [2] when you have an odds advantage to a fake initial $1000 bankroll and seeing where you end up at the end of the tournament. If Iran does end up being a 46% favorite tomorrow then that Kelly test will start to look really good, really quickly - the market is only giving Iran a ~25% chance right now! [1] https://projects.fivethirtyeight.com/2018-world-cup-predictions/matche… [2] http://www.elem.com/~btilly/kelly-criterion/ for the background, http://www.albionresearch.com/kelly/ for a useful calculator

By the way, if you're a regular-ol'-geek and wondering "I wonder if writing about technical topics could help me career-wise", the answer is "Absolutely yes." This is so true. I think the bulk of my consulting work comes to me via my blog - I've put basically no effort into seeking it out. A conversation which happens with some regularity: Bossish person: "We are having trouble with $MATHY_STUFF. Do we know anyone who can help us with it?" Codeish person: "I've been following these tutorials on chrisstucchio.com..." I'll disagree with patrick a bit about "a visual identity distinct..." and formatting, at least for people who never touch the frontend. The people looking for you probably don't care that much. Here are some fugly blogs of people you'd almost certainly hire: http://www.johndcook.com/blog/2008/08/21/random-inequalities-v-beta-di… http://www.stephendiehl.com/posts.html http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigorous.html That said, dropping $1000 on some pretty visual design probably can't hurt.

HN user btilly has a really helpful essay on the math behind stopping tests earlier than your predetermined sample size. It calls for setting a maximum duration, and provides stopping points along the way. Works similar to the method AirBnB describes. http://elem.com/~btilly/ab-testing-multiple-looks/part2-limited-data.h…

This slide deck worked for me, with the explanations in understandable chunks: http://elem.com/~btilly/effective-ab-testing/

tl;dr: Don't bother with confidence intervals. Use a G-test instead. Calculate it here: http://elem.com/~btilly/effective-ab-testing/g-test-calculator.html Read more here: http://en.wikipedia.org/wiki/G-test And plain English here: http://en.wikipedia.org/wiki/Likelihood_ratio_test

A/B Testing Rigorously Without Losing Your Job - B. Tilly - http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigorous.html 20 Lines of Code that Will Beat A/B Testing Every Time - S. Hanov - http://stevehanov.ca/blog/index.php?id=132 These are the ones I've read recently. If I remember any others, I'll update.

First, Google Analytics (for example) runs content experiments for a minimum of two weeks regardless of results. It's hardly an unrealistic timeframe for reliable conclusions. > What practical benefit do you see to this approach? Statistically rigorous results, with minimal regret. In your example, you reach the end of the week, and your 50/50 split has one-sided p=0.10, double the usual p<0.05 criteria. What do you do? (a) Call it in favor of B, despite being uncertain about the outcome. (b) Keep running the test. This compromises the statistical rigor of your test. (c) Keep running the test, but use sequential hypothesis testing, e.g. http://elem.com/~btilly/ab-testing-multiple-looks/part1-rigorous.html This significantly increases the time to reach a conclusion, and costs you conversions in the meantime. (a) and (b) are the most popular choices, despite them being statistically unjustifiable. http://www.evanmiller.org/how-not-to-run-an-ab-test.html --- The essential difference when choosing the approach is that 50/50 split optimizes for shortest time to conclusion, and multi-bandit optimizes for fewest failures. In web A/B testing, the latter is usually the most applicable, and for that, you cannot beat Thompson sampling on the average, no matter how diabolically clever your scheme. https://www.lucidchart.com/blog/2016/10/20/the-fatal-flaw-of-ab-tests-… There are times when the former is more important, e.g. marketing wants to know how to brand a product…