Andrew Gelman

Andrew Gelman's blog provides regular insightful commentary on this issue, I highly recommend it: http://andrewgelman.com/ The post that turned me on to all of this is at: http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-…

I am a social scientist studying human behavior, and this is a huge problem in the field. Myself and my statistician friends who analyze the literature have basically concluded that most extremely "novel" and "surprising" findings in the literature aren't even worth trying to replicate (remember, replications cost money to run, so before you start you have to make some judgment about the likelihood of success.) This is especially true of the "sexiest" sub-topics in the field, like social priming and embodied cognition. If you want to learn more about this, the place to look is Andrew Gelman's blog: http://andrewgelman.com/

Andrew Gelman's blog[1] is a great resource for identifying erroneous studies in social science. [1] http://andrewgelman.com/

That happens in statistics programs. However, I have a ML-heavy minor in CS, and based on the ML course contents at our CS dept I've seen, I'm not sure if the all their CS majors go through the the full canonical statistics curriculum, nor that they were intended to. At least the ML courses had quite much introductory probability and statistics as far as ML applications were concerned, so I understood the implication was they didn't assume that the students would have already done the similar stuff in statistics (though it certainly helped), and I can't remember a single mention of p-value there. And then there's this, that even if your intro to probability course everywhere covers the classic statistics with p-values and hypothesis testing and frequentist confidence intervals and so on, you are not necessarily going to use them that much. I calculated some p-values and other tests with R for some example datasets a couple of years ago and never seen them since in coursework, everything we've done after that has been more or less fully Bayesian. The concepts are still fresh[1] in my mind mostly because I read some statistics blogs, such as Andrew Gelman's [2]. The irony is that Gelman does not exactly love frequentist framework, he just mentions its concepts often enough. [1] or not totally forgotten [2] http://andrewgelman.com/

It's an open secret in the news industry that while all of those longform articles, parallaxin' interactives and news apps do get great traffic, that traffic is nothing in comparison to that totally random and not even very good quick hit that went viral yesterday. Personally, I also agree that you shouldn't worry too much about polishing content and instead just get it out there. That's what's so lovely about the blog of statistician Andrew Gelman for example [1], he just writes what comes to mind and his readers love it because they care more about his insights than about whether or not he's presenting an eloquently argued, watertight argument. It reminds me of the old days of blogging, and I think many of us miss those days. That said, Kent Beck displays some really sloppy thinking here. "I have no clue about what the actual numbers are, and even though I could get them and run some analysis on them to determine the effects of quality vs. quantity, instead I'm just going to philosophize about how I think the world works, and justify my ramblings by saying that it's a POWER LAW, and power laws, that's like nature, dude, you can't fight it." [1] http://andrewgelman.com/

Stan is a fantastic language/library/tool. (Disclosure, I've taken 2 classes with Andrew Gelman[1]. For those who haven't used it, the typical usage is to use Stan in combination with other languages (most commonly R). In the Stan language you define a model, specifying the data you'll get, potentially transformations to that data, then a set of parameters you want to fit, then finally a model saying how those parameters interact with each other and the data. You can also define priors for parameters. Then typically you save that model, and using R pass in the variables to a Stan call. The resulting fit object is returned to your original environment automatically. It actually fits the work style reasonably well. All data munging happens as before, but instead of having some complicated model expression in, say, a glm() call, you have it in Stan language. If you're interested in this, Gelman has two great books. Gelman & Hill's Hierarchical Models[2] which is applied and geared towards social science researchers, and Bayesian Data Analysis[3]. [1] http://andrewgelman.com/ [2] http://www.stat.columbia.edu/~gelman/arm/ [3] http://www.stat.columbia.edu/~gelman/book/

Andrew Gelman put it very well here: https://andrewgelman.com/2017/02/06/not-kill-statistical-significance-… Or, if you want to some it all up in a single graph: https://andrewgelman.com/2014/11/17/power-06-looks-like-get-used/ (although your experiments hopefully have power much greater than 0.06) The gist of the point: If your sample size is small, significant results are likely to (sometimes vastly) overestimate the true effect size, and there is a non-negligible chance that the effect size is actually estimated in the wrong direction.

That seems to be part of it. However, I don't think disliking parsimony is that radical. Andrew Gelman is not a fan of parsimony. http://andrewgelman.com/2004/12/10/against_parsimo/ My current view is that parsimony is a heuristic you can use, but weak evidence at best. Plus, parsimony is not unambiguously defined, so comparing hypotheses in terms of it can still be subjective even with objective criteria as there is no agreement on which approach is best. It's most justified to say that extremely complex models are unlikely if the complexity is not necessary.

One of the best articles covering this issues is Meehl[1][2]. You can find discussion in various places like Gelman[3] and Reinhart[4]. [1] Meehl, Paul E (1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66(1), 195–244. [2] http://meehl.umn.edu/files/144whysummariespdf [3] http://andrewgelman.com/2015/03/23/paul-meehl-continues-boss/ [4] https://www.refsmmat.com/notebooks/meehl.html

I'm not sure that's true. Andrew Gelman once mentioned that Black Swan events are actually routinely _overpriced_ via the Longshot Bias. [1] He had talked about this in the context of the recent Leicester FC win and the odds on that team. [2] [1] https://en.wikipedia.org/wiki/Favourite-longshot_bias [2] http://andrewgelman.com/2016/06/02/30183/

Here is the problem: If there is an actual large effect size, then you can detect this in small samples. However, the opposite is not true. In a small sample, spurious large effects are MORE likely, not less likely (outliers have a greater impact in a small sample). See: http://andrewgelman.com/2017/08/16/also-holding-back-progress-make-mis…

FYI, Bargh (the author of this editorial) has been pushing these sorts of priming effects for a long time. Many of these (most famously the "priming people with elderly-related words makes them walk slower" study) have not held up under subsequent replications. Clearly this doesn't mean that the studies Bargh is describing here are similarly flawed – but he has shown a strong resistance to admitting his mistakes, and so I'm less inclined to give him the benefit of the doubt here, particularly when the original studies aren't linked anywhere. Further reading: http://blogs.discovermagazine.com/notrocketscience/2012/01/18/primed-b… http://andrewgelman.com/2016/02/12/priming-effects-replicate-just-fine… http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-…

As someone outside the scientific-academic community, I find his posts very accessible for learning news/issues about studies. I greatly appreciate his passion in trying to stop junk science from reaching the media. Maybe his rhetoric is too brash and coarse for the respectful standards of scientific discourse, but it's as harsh as it needs to be for scolding media outlets for running with junk science: http://andrewgelman.com/2016/05/05/npr-bites/ http://andrewgelman.com/2016/11/03/its-not-about-reality-its-all-about…

>This moment of fizziness for the discipline was already underway when Cuddy arrived at Princeton’s graduate program in 2000, transferring there to follow her adviser, Susan Fiske, with whom she first worked at the University of Massachusetts at Amherst. This is the same Susan Fiske who called people who pointed out statistical errors in psychology papers "methodological terrorists". http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-… The apple does not fall far from the tree.

No, because they want to present the conclusions of each project as a "fact" that we now know for certain, as in a constant march of progress. They feel it reflects poorly on them if they published/funded something incorrect, so they come up with any excuse to not publish any critique or do a retraction. Eg, Andrew Gelman has some pretty good stories about this: "In the review process they did not disagree with my points at all, but they refused to publish the correction on the grounds that they only publish the very best submissions to ASR." http://andrewgelman.com/2016/02/22/its-too-hard-to-publish-criticisms-…

Judea Pearl & Andrew Gelman are great people. Trevor Hastie's books are good too. https://web.stanford.edu/~hastie/pub.htm http://bayes.cs.ucla.edu/jp_home.html http://andrewgelman.com/books/ I guess it all depends on what is a mathematical background and what is advanced!

Maybe not all but here's another also with a reference to the priming kerfuffle http://andrewgelman.com/2017/02/18/pizzagate-kahneman-two-great-flavor…

It's worth noting the less-than-well-known saying that "correlation does not imply correlation" (which is a very nice turn of phrase too). [1] In the introduction they say the effect is significant at the 5% level, which for an exploratory data analysis doesn't mean much. [1] http://andrewgelman.com/2014/08/04/correlation-even-imply-correlation/

Were their experiments be recreated and properly peer-reviewed? Some reviewers argue that neuroscience experiments are mostly not worth the candle. http://andrewgelman.com/2016/09/21/what-has-happened-down-here-is-the-…

Here is a followup update: http://andrewgelman.com/2015/11/06/age-adjustment-mortality-update/

Actually, it's worse than that - theoretically, they should be doing an F-test, but those are essentially useless[0] except as exercises for unfortunate undergraduates. This gets into the flaws with hypothesis testing (and p-values) as a tool altogether, though. And while I would love to see the natural sciences eschew these rather rudimentary heuristics in favor of the more sophisticated modeling tools that we already have, I'm not holding my breath on that happening anytime soon. [0] http://andrewgelman.com/2009/05/18/noooooooooooooo/

To play devil's advocate, I'm not necessarily saying that the best education stems from these schools but it's certainly other things that you're getting value from. This other HN article seemed relevant: http://andrewgelman.com/2014/09/30/23581/

Not my area, but one starting place is http://www.andrewgelman.com

Here is Andrew Gelman's comment on this issue: http://andrewgelman.com/2013/04/16/memo-to-reinhart-and-rogoff-i-think…

The greatest factor for longevity in an individual is genetics. For a population not so much. Lots of people say if you remove the US, the greatest correlation (not causation and all that, yes yes) to longevity is health care spending. Picking the first result off google, here: http://andrewgelman.com/2009/12/healthcare_spen/ Japan is a clear outlier but so is the US.

You're not alone. I don't know if you have a ML background, but it has been somewhat widely discussed in the community that the way Netflix scored the competition -- RMSE -- isn't the best way. E.g. http://hunch.net/?p=949 and http://andrewgelman.com/2008/11/netflix_prize_s/ In the OPs article they mention they monitor if a movie is watched to completion, which gives them a much better metric to optimise. The other issue is that this is really a sequential decision making problem. Recommending a movie has an opportunity cost -- there are other movies you don't recommend -- and the recommendation is an ongoing process, so it is probably best to spend some time exploring the user's taste on the assumption this will let you make better recommendations in the future. Accounting for these issues is much harder in a competition format.

> You mean, I'm welcome to find supporting evidence for your argument? Shouldn't that be your responsibility? Fair enough. I had some trouble finding the original better example I had mentioned earlier ("I recall a study looking at such impossible aggregate statistics leading to several retractions of articles whose data had been made up outright."), but while trying to find it, I stumbled on another. Example 1: https://andrewgelman.com/2018/08/21/scandal-isnt-whats-retracted-scand… A study gets published finding a large effect, large enough to cause a replication attempt: "The effort to replicate the original study was successful in everything except the creation of the PSU-level structural stigma variable." The suspected reason for this replication failure (imputation of missing data) turns out to be wrong when the original authors have someone check their code for data analysis, which is found to contain an error. If that code had been checked during peer review (or at least afterwards, by including it with the publication), the effort would have been less than a full-blown replication attempt. Example 2: https://medium.com/@jamesheathers/the-grim-test-a-method-for-evaluatin… (Which is what I was referring to earlier.) Simply checking reported means and sample sizes for consistency revealed mathematically impossible results in 50% of tested papers. He goes to great lengths to stress that such inconsistencies…

ignoring the automation aspect, > Then you write exactly WHAT you are going to do with the data and then you send the proposed "routines" this roughly matches up with the idea of "preregistration" of research, i.e. you define and share what your experimental method is going to be before you start looking at the data and performing analysis, to help guard against some unconscious or conscious decisions to adjust the method after you have observed the experimental data. i've never heard of cos.io before but they're a top hit for me when searching for "preregistration" https://cos.io/prereg/ Andrew Gelman has written quite a lot about related topics in the past (the "replication crisis" - especially in the social sciences, "p hacking", "the garden of forking paths" http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pd… , https://andrewgelman.com/2017/03/09/preregistration-like-random-sampli… ) I quite like how Gelman theoretically frames this in his "forking paths" paper: Consider the following testing procedures: 1. Simple classical test based on a unique test statistic, T, which when applied to the observed data [ y ] yields T(y). 2. Classical test pre-chosen from a set of possible tests: thus, T(y; phi), with preregistered phi. For example, phi might correspond to choices of control variables in a regression, transformations, and data coding and excluding rules, as well as the decision of which…

- Marginal Revolution (Tyler Cowen & Alex Tabarrok): economics https://marginalrevolution.com - John Cochrane: economics https://johnhcochrane.blogspot.com - Interfluidity (Steve Randy Waldman): all over the place, but mostly economics-focused https://www.interfluidity.com - FT Alphaville: finance and economics, free signup required https://ftalphaville.ft.com - Money Stuff (Matt Levine): finance and economics, very entertaining and accessible https://www.bloomberg.com/view/contributors/ARbTQlRLRjE/matthew-s-levi… - Andrew Gelman: statistics with an emphasis on applications in the social sciences http://andrewgelman.com - Flowing Data (Nathan Yau): data visualization and statistics https://flowingdata.com - Shravan Vasishth: statistics and statistical computing, often very technical https://vasishth-statistics.blogspot.com - Rasmus Bååth: statistics and statistical computing (inactive but worth mentioning) http://www.sumsar.net/blog/archives

I remember reading http://andrewgelman.com/2016/11/13/more-on-my-paper-with-john-carlin-o… with its graph "This is what power = 0.06 looks like". So I got the point that you have to have sufficient statistical power. A useful rule of thumb is that you need a power of at least 0.8. You need to have some idea how big the effect is likely to. Perhaps from previous exploratory research, from claims of other researchers, from reasoning "well, if this is happening the way we think it is, there has to be an effect greater than x waiting to be discovered.". Then you work out how big a sample size you need to use. Then you roll up your sleeves and get down to work. But the reason for using p values rather than Bayesian inference is that it gets you out of the tricky problem of coming up with a prior. You only need to think about the null hypothesis and ask yourself whether the probability of the data, given the null hypothesis, is less than 0.05. So there is a bit of contradiction. p values don't really work unless you ensure that you have sufficient power. To do that you need a plausible effect size, to feed into your power calculation. And that is implicitly an rough approximate prior, 50:50 either null or that effect. You could just do a Bayesian update, stating how much you shifted from 50:50. Basically, if you don't already know enough to have an arguable prior to get a Bayesian approach started, you don't know enough to do a power calculation, so…