Andrew Gelman

Regarding 'cad-bloggers from Columbia: Woit does do a bit of grey area coverage. Likewise, his academic pubs are more interesting to me, personally! His reputation, however, is mixed at best. A real hero! Gelman: maybe he's trying to subdue the neuroticism*. I look forward to more compassionate, less overtly "curiosity"-driven grey-area coverage (not to mention stats papers) from him! Here is recent one, sadly also about psychologists (would be more interesting to do it to card carrying statisticians) http://stat.columbia.edu/~gelman/research/published/healing3.pdf *Not many people describe themselves as an exponent of radical self-honesty since radicality isn't a palatable, much less enjoyable, thing. It takes a real saint (or logician) not to reflexively flinch from the imagined tsunami of incoming pity. Grey areas-- you can safely explore

I would recommend BDA3 and Stan. http://www.stat.columbia.edu/~gelman/book/

Gelman (http://www.stat.columbia.edu/~gelman/research/published/philosophy_onl… "The classical or frequentist approach to statistics (in which inference is centered on significance testing), is associated with a philosophy in which science is deductive and follows Popper’s doctrine of falsification. In contrast, Bayesian inference is commonly associated with inductive reasoning and the idea that a model can be dethroned by a competing model but can never be directly falsified by a significance test. The purpose of this article is to break these associations, which I think are incorrect and have been detrimen- tal to statistical practice, in that they have steered falsificationists away from the very useful tools of Bayesian inference and have discouraged Bayesians from checking the fit of their models."

See this MLSS '20 presentation and all recommended bibliography [1], in particular top left box in slide 123. For rigorous modeling, Gelman et al. books are great [2-3]. [1] https://drive.google.com/file/d/1lPePNMGMEKoaDvxiftc8hcy-rFpobXeJ/view [2] https://avehtari.github.io/ROS-Examples [3] http://www.stat.columbia.edu/~gelman/book

ok, let me help you out: http://www.stat.columbia.edu/~gelman/ https://en.wikipedia.org/wiki/Andrew_Gelman which would give strong prior evidence that the author's writing is not absurd. Note also that he has strong connections to a leading psychologists meaning he has some background on human choices and behavior. https://en.wikipedia.org/wiki/Susan_Gelman Please be careful with attack statements such as "so absurd that it's not worth engaging with seriously" which some feel are out of character with HN.

Forking paths. [1], overfitting, fitting to the holdout set, etc etc. "P-hacking" applicable to vastly more than stock data and is not evidence of anything, evidence is what was asked for. Your response implies I'm ignorant and can't google p-hacking without being told the name of that one way of finding false correlation applicable to many, many fields. Now you're getting a bit cross about it that I've quietly pointed out that it isn't useful, and you're not actually providing expertise here which you thought you were. It's kind to provide expertise and I appreciate the kindness. You misread, fine. Best. edit: Oh wait > Or maybe that term is more general, I'm not 100% sure, but it's close. That is an edit addition to the parent after my reply, so I guess you did get the point. Best. [1] http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pd…

Did you notice if they pre-registered the separate analyses of males and females? Flexibility in the analysis of data ("researcher degrees of freedom") is likely one of the major causes of the replication crisis. Gelman's paper is excellent in showing how these issues manifest without any intended malfeasance: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pd…

Not my impression. The kind of outlier-culling technique suggested by civilized is not recommended these days because it adds unprincipled choice points to what Andrew Gelman calls the 'Garden of Forking Paths' [1, 2]. Thus they are bad for hypothesis testing, which tends to be what most statisticians care about. Additionally the technique obscures the relationship between the variance of the sample and the population variance if we do not have reliable knowledge of the population distribution; likewise for the mean if the mean is not close to the mode. These problems can be quite dramatic for long-tailed distributions. [1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pd… [2]: https://statmodeling.stat.columbia.edu/2016/09/30/why-the-garden-of-fo…

I took a course on Applied Bayesian Statistics taught by David Draper in grad school and we covered Bayesian Data Analysis (Gelman et Al.) http://www.stat.columbia.edu/~gelman/book/ and Probability Theory and tbe Logic of Science by Ed Jaynes: https://www.amazon.com/dp/0521592712/ref=cm_sw_r_em_apa_i_v35QEb3FEYSG… The former is a much recommended book since it's very comprehensive and builds everything from the ground up and was the basis for the entire course. The latter is a beast of it's own and we simply covered what was effectively the first chapter as part of the course.

Is this the book you are recommending? http://www.stat.columbia.edu/~gelman/book/

- < http://www.stat.columbia.edu/~gelman/book/ > - < http://www.stat.columbia.edu/~gelman/arm/ > If you happen to use Python, pymc3 is a great place to look for hierarchical and Bayesian timeseries models with full examples.

Cool! Thanks for sharing. I would also like to add, Stan is not that hard. Conceptually it's a bit of a jump, especially if you're coming from comp sci not statistics, but I think it's really worth it. Here are some helpful links: Stan website: http://mc-stan.org/ A primer on Stan: http://www.stat.columbia.edu/~gelman/research/published/stan_jebs_2.pd… The Python interface PyStan documentation: https://pystan.readthedocs.io/en/latest/ A case using Stan in a real startup's product: https://www.smartly.io/blog/tutorial-how-we-productized-bayesian-reven…

At least speaking as a academic (4th year PhD student), one of the challenges with researchers over reliance on NHST seems to be the apparent lack of a more compelling alternative. One candidate which is gaining traction is Bayes factors but there are challenges with this approach as well e.g. with suitable specification of priors. The best way forward will involve fundamentally restructuring how we educate incoming researchers because it will necessarily mean embracing uncertainty over dichotomization of results into yes/no. Andrew Gelman and John Carlin write elegantly about ways forward here: http://www.stat.columbia.edu/~gelman/research/published/jasa_signif_2.…

This is on everyone’s short list of problems with Bayes. In the simplest interpretation of Bayes, old evidence has zero confirming power. If evidence E was on the books long ago and it suddenly comes to light that H entails E, no change in the value of H follows. This seems odd – to most outsiders anyway. I don't understand what he's referring to here. If we now know that H entails E, then that means our model of the world changed, and thus our posterior on H changed as well. Did I miss anything? It's an interesting article, but there is a lot of debate around the interpretation of Bayesian inference, and IMO there are answers to be found. In particular, Andrew Gelman argues against the subjective interpretation: http://www.stat.columbia.edu/~gelman/research/published/philosophy_onl… It's the best article I've read on the subject.

Yep, there is really no other book like Harrell. There's a 2015 edition with some good improvements too. Definitely not an introduction, though – it assumes you already know how regression works and preferably have already used it quite a bit in practice. For an introduction, I might recommend Downey's Think Stats [1] for a gentle introduction, Kutner's Applied Linear Statistical Models for a more traditional, mathy introduction or Gelman and Hill's Data Analysis Using Regression and Multilevel/Hierarchical [2] for something in-depth but still very approachable. [1] http://greenteapress.com/thinkstats2/ [2] http://www.stat.columbia.edu/~gelman/arm/

The linked paper of Gelman is a really good read on Bayesian thinking in philosophy of science: http://www.stat.columbia.edu/~gelman/research/published/philosophy.pdf

> Would I be on firm statistical footing if I started looking for effects after the study, as long as I choose a p-value such that (1-((1-p)^n) < 0.05? No. For one thing, how do you actually know n? If in your analysis you can make c independent binary choices, n would be on the order of 2^c. For most reasonable sequence of choices, n would be impractically large. And if you think you know n, how would you convince everyone else that your value of n is reasonable/trustworthy? The xkcd case (which is solved by applying the Bonferroni correction) is special because there's only one test run in 20 experiments, so the correction is more straightforward. There is a good paper about this garden of forking paths by Gelman and Loken [1]. [1] (pdf) http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pd…

The paper is from 2005, so it might be worth saying that on the submission title. And it's also worth mentioning that because Gelman and Shalizi have recently written a really nice paper about the philosophy of Bayes: http://www.stat.columbia.edu/~gelman/research/published/philosophy.pdf (PDF) Gelman is a well-known "bayesian", and Shalizi is a well-known "not all that is interesting is bayes"-ian.

Evan, does this formula account for multiple comparisons (if you have multiple goals and multiple variations)? I guess it would suffer from the same problems that if you have 100 variations and 10 goals, some of them are bound to produce a significant result, just by random chance. Isn't it? In classical testing, you can fix it by making your calculated alpha smaller than the real alpha, so you need much more data if there are multiple comparisons. What happens in Bayesian case? Edit: I did some Googling and found this http://www.stat.columbia.edu/~gelman/research/unpublished/multiple2.pd…

That paper is a very different setup --- the hierarchical model Gelman, Hill, and Yajima use implicitly builds multiple comparisons into their process. You could do the same thing here in principle by drawing alpha and beta for different experiments from an underlying distribution, but I'd be cautious about applying that approach mindlessly. If you haven't read it, Gelman and Hill's book is excellent: http://www.stat.columbia.edu/~gelman/arm/ and so is Gelman's blog.

You are getting downvotes because you didn't follow Bayesian reasoning, but there is some justification for your instincts here http://www.stat.columbia.edu/~gelman/research/unpublished/philosophy.p…

The best first book on R I've read is Gelman and Hill's "Data Analysis Using Regression and Multilevel/Hierarchical Models" (http://www.stat.columbia.edu/~gelman/arm/). It's more focused on regression and Bayesian methods than basic probability, but it's a great book that's priced very competitively (only $39.99).

Harder the me just listing them, but you can take a look through my Google Reader shared items: http://www.google.com/reader/shared/10530692481246922106 The most frequent ones are: * Schneier on Security ( http://www.schneier.com/blog/ ) * Statistical Modeling, Causal Inference, and Social Science ( http://www.stat.columbia.edu/~gelman/blog/ ) I also follow a bunch of significant python people and the artima weblogs.

Hi, thanks for the honest and thoughtful discussion you are conducting here. Comments tend to be simplistic and it's great to see that you raise the bar by addressing criticism and questions in earnest! That said, I think the fundamental problem of such tools is unsolvable: Out of all possible analytical designs, they create boring existing results at best, and wrong results (i.e. missing confounders, misunderstanding context ...) as the worst outcome. They also pollute science with harmful findings that lack meaning in the context of a field. These issues have been well-known for about ten years and are explained excellently e.g in papers such as [1]. There is really one way to guard against bad science today, and that is true pre-registration. And that is something which LLMs fundamentally cannot do. So while tools such as data-to-paper may be helpful, they can only be so in the context of pre-registered hypotheses where they follow a path pre-defined by humans before collecting data. [1] http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pd…

Bayesian methods are not easy to use, I agree. But that's because they're trying to answer much more meaningful (but harder) questions than frequentist ones, which researchers should be trying to do. You can't ignore Bayesian epistemology just by not using Bayesian methods. The underlying considerations in the Bayesian framework will inevitably become relevant to how you interpret your data, whether or not you use a formal Bayesian method. The thing is, formally, frequentist methods don't tell you anything. If you get a significant p-value, that means the data you observed wouldn't often happen by chance (within your model of the null). This doesn't actually tell you if your particular hypothesis should be favored. That requires other considerations, including ones that Simonsohn is negative about in this article. For example, Simonsohn's conclusion says: > To use Bayes factors to test hypotheses: you need to be OK with the following two things: > 1. Accepting the null when “the alternative” you consider, and reject, does not represent the theory of interest. > 2. Rejecting a theory after observing an outcome that the theory predicts. He implies that these should be points against Bayes factors. But #2 is something you actually should do sometimes. Demonstrably. In particular, if the data suggests a wildly implausible effect size, that should be a point against your theory and in favor of some more mundane explanation, like noisy data from an underpowered study…

Your comment appears to imply Bayes theorem is all you need for a scientific framework and outside of that is just inadequate mathematical know-how to deploy it. I find it amazing really, a quintessential straight-from-the-summit [1] HN comment for a couple reasons: First, it appears to imply that such a probabilistic inductive approach to science would be free of any philosophical baggage or assumptions, when deploying Bayesianism requires an interpretation about what a probability itself is. Don't take it from me though, perhaps from Andrew Gelman, the guy who wrote the book on Bayesian data analysis [2,3]. Then, wrt the charge that those who do not use such an inductive approach (or outright rejected it, e.g. in favour of falsificationism) are bad at math. Which would include the statisticians who developed the null hypothesis significance testing framework that is still pretty dominant in science today: Jerzy Neyman, Egon Pearson, Ronald Fisher (who literally coined the term 'Bayesian') etc. There's a lot of criticism worth making about Fisher, but I'm not sure if anyone has called the guy that developed linear discriminant analysis bad at math before. [1] https://www.smbc-comics.com/comic/2011-12-28 [2] Bayesian Data Analysis http://www.stat.columbia.edu/~gelman/book/ [3] Philosophy and the practice of Bayesian statistics http://www.stat.columbia.edu/~gelman/research/published/philosophy.pdf

This. Multilevel models relax the assumption of independent observations by specifying that the measures of repeated experimental units are dependent on each other. It's a way of telling your model that it has less information than it would have if all observations came from independent units. Therefore, standard errors of effects are usually larger. Otherwise, they are biased [1]. Since most researches are not aware of multilevel models, they design their experiments and aggregate their data to fit the independence assumption, which is rarely a good idea. Many are not even aware of modeling beyond hypothesis tests, and are unable or unwilling to adjust their analysis for confounding factors or non-sampling errors that arise due to experiment design flaws. Also, p-values should be deprecated, since a) nil hypothesis are strawmen at best and false by definition at worst [2] and b) they incentivize researchers to not think hard about effect sizes and uncertainty in their problems. [1] https://academic.oup.com/biomet/article/73/1/13/246001 [2] http://www.stat.columbia.edu/~gelman/research/published/failure_of.pdf

Maybe you should read something other than Casella & Berger for Bayesian statistics. The book is great, but the coverage in this topic is not. Something like Bayesian methods for hackers [1], the puppy book [2] are great tutorial style easy to read books. Big boy Gelman [2] is far more mathematical and advanced. I highly recommend you [1] cause it's free, hands on and if you have solid stats will go fast and easy. Anyways, the thing with Bayesian is you need to look at the problems from a different optic and learn new names for the same things you know from classic (Maximum likelihood is now called mean a posteriori and whatnot). If I had to summarize the whole uncertainty thing I would say: Both in classic and Bayesian it holds that a function of random variables yields another random variable. Models are functions of random variables. As such, every point estimate you give has a full distribution behind. In classic this is accounted for with asymptotics and confidence intervals. In classics you have priors all the same, but using defaults and not talking about them. In Bayesian you talk about your priors, and if you have extra information about your inference problem you include it there. Or you abuse the priors to make the model yield results that will make your boss/client happy (I worked as an economics researcher at a big bank, the whole department was Bayesian). Or choose priors that make you custom model actually compute. Or use priors that do regularization (like…

ignoring the automation aspect, > Then you write exactly WHAT you are going to do with the data and then you send the proposed "routines" this roughly matches up with the idea of "preregistration" of research, i.e. you define and share what your experimental method is going to be before you start looking at the data and performing analysis, to help guard against some unconscious or conscious decisions to adjust the method after you have observed the experimental data. i've never heard of cos.io before but they're a top hit for me when searching for "preregistration" https://cos.io/prereg/ Andrew Gelman has written quite a lot about related topics in the past (the "replication crisis" - especially in the social sciences, "p hacking", "the garden of forking paths" http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pd… , https://andrewgelman.com/2017/03/09/preregistration-like-random-sampli… ) I quite like how Gelman theoretically frames this in his "forking paths" paper: Consider the following testing procedures: 1. Simple classical test based on a unique test statistic, T, which when applied to the observed data [ y ] yields T(y). 2. Classical test pre-chosen from a set of possible tests: thus, T(y; phi), with preregistered phi. For example, phi might correspond to choices of control variables in a regression, transformations, and data coding and excluding rules, as well as the decision of which…

> "Data is data and data is good" This is a reasonable intuition, but is hard to square with the increased risk of publication bias for small studies (i.e. we should question whether the typical small study published is really "data" in the sense we'd like). Since small studies are more sensitive to noise and "fishing" for an effect, and since non-results are rarely published, what happens with meta-analysis of many published small studies is that you end up primarily looking at outliers rather than typical results. Since most research questions have a directionality to what counts as "interesting", the outliers also tend to be clustered on one side rather than evenly split. The two papers I recommend anyone read if they care about these problems are: Why Most Published Research Findings are False http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.… The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pd…