Statistical Modeling, Causal Inference, and Social Science

> A Harvard University probe into prominent researcher Francesca Gino found that her work contained manipulated data and recommended that she be fired, according to a voluminous court filing that offers a rare behind-the-scenes look at research misconduct investigations. I thought this was going to be about Dan Ariely, who has been mixed up in a similar controversy, though idk if Ariely's saga has reached a conclusion yet. Added: aha, Ariely is at Duke rather than Harvard. Can't keep track without a scorecard. Added 2: Andrew Gelman blog post from Thursday about the Gino finding: https://statmodeling.stat.columbia.edu/

Cool list! My favorite ML blog is https://www.oneweirdkerneltrick.com/ I really appreciate Andrew Gelman’s statmodeling blog. There’s some overlap with ML, but it’s mostly from a stats perspective https://statmodeling.stat.columbia.edu/

You lost me at NPR do a pretty great job at science reporting. In fact they are a punchline to the joke about how incredibly bad science reporting can be. https://statmodeling.stat.columbia.edu/?s=Npr

For the unaware I also recommend prof Gelman blog. https://statmodeling.stat.columbia.edu/

> Seventy-three independent research teams used identical cross-country survey data to test an established social science hypothesis: that more immigration will reduce public support for government provision of social policies. Instead of convergence, teams’ numerical results varied greatly, ranging from large negative to large positive effects of immigration on public support. The choices made by the research teams in designing their statistical tests explain very little of this variation: a hidden universe of uncertainty remains. That's... surprising, but keep in mind that we're talking about "soft" social sciences here. Measuring soft quantities like "public support for government provision of social policies in response to increased immigration" is very different from measuring hard quantities like, say, the location of a rocket in space at a point in time using earth as a frame of reference. I'm looking forward to seeing what Andrew Gelman and his colleagues will say about it at https://statmodeling.stat.columbia.edu/ -- surely they will want to say something about it!

I'm a fan of the stats blog hosted by Columbia that Gelman is the main contributor to: https://statmodeling.stat.columbia.edu

CITATION NEEDED — except not really, that's utter bollocks that doesn't warrant a counter. But, you triggered me lol: 1.) The Chinese meaning of "atheist" may not mean what you think it means: a.) it means not following a major established religion, and b.) they have an Orwellian surveillance-state dictatorship government which very much opposes major established religions 2.) For many people, including in China, astrology is just something like buying a lottery ticket, or asking the Magic 8-Ball for advice — fun, but not something actually believed in[1]. 3.) It's much more likely that 3% of atheists "believe in" astrology[2] — and the idea that a majority would is insane on its face So yeah, nah. [1]: https://www.pewresearch.org/religion/2025/05/21/3-in-10-americans-cons… [2]: https://statmodeling.stat.columbia.edu/2020/04/11/42552/

There's been enough follow-up on this case to come to the conclusion that he fabricated the data. The Hartford insurance company's statement contradicts his claim. The PNAS study was retracted after the Data Colada article and there are anomalies in his other work. [0]: https://openmkt.org/blog/2023/everyone-involved-in-dan-arielys-fake-da… [1]: https://statmodeling.stat.columbia.edu/2021/08/19/a-scandal-in-tedhemi… [2]: https://www.npr.org/2023/07/27/1190568472/dan-ariely-francesca-gino-ha… > It is clear the data was manipulated inappropriately and supplemented by synthesized or fabricated data.

I am curious too. I have used Stan extensively. I found it extremely polished and pleasant to use. It generated very efficient samplers for particularly weird (and enormous!) hierarchical models I had. Documentation is also great. It is also worth reading Andrew Gelman's post about Prophet: https://statmodeling.stat.columbia.edu/2017/03/01/facebooks-prophet-us…

Bomber mafia: https://lareviewofbooks.org/article/when-pop-history-bombs-a-response-… Roundup of a few (follow links): https://statmodeling.stat.columbia.edu/2020/02/29/the-paterno-defence-… If Books Could Kill (worthwhile, as is the Freakonomics episode of the same podcast): https://podcasts.apple.com/us/podcast/malcolm-gladwells-outliers/id165…

I recently started using RSS again (self-hosted with miniflux). One difficulty I have is that these kinds of RSS feeds in OP are way too high-volume for me. But it can be hard to find good blogs or RSS feeds without relying on Twitter to find them. Back in the day, more people used to include a list of blogs they follow on their own blogs, I thought this was a great practice. A few still do, like this very interesting blog on statistics: https://statmodeling.stat.columbia.edu/blogs-i-read/

> Do you have any examples where this happens? I read a lot of blogs, but hardly see any in-depth discussions appearing under the posts. Scott Alexander's blogs, Slate Star Codex[1] and Astral Codex Ten[2]. Andrew Gelman's Blog[3]. Marginal Revolution[4], though that doesn't seem heavily moderated. > Also, when the author moderates, do you trust any critique to stay online and maybe even be countered? No, but that's why I like both. I like seeing a discussion not moderated by the author, like on HN, to see more criticism, but I also like seeing a discussion more respectful towards the author, especially on topics where on HN I'd only see contempt towards to the author. [1]: https://slatestarcodex.com/ [2]: https://astralcodexten.substack.com/ [3]: https://statmodeling.stat.columbia.edu/ [4]: https://marginalrevolution.com/

It sounds like you read Andrew Gelman's blog. You might be interested to review this post: https://statmodeling.stat.columbia.edu/2012/11/10/16808/ Frequentist statistics does not force you to accept any hypothesis test with a result p < 0.05 as definitive proof of something. It does not forbid considering prior probability of a result. It just doesn't formalize the consideration of prior probabilities because it is hard to distill this consideration into a formal recipe. Everyone agrees that Bayes' Rule is valid and important, the question is when and how best to use it. > If you use bayesian methods, you're pretty much forced to spend more time considering the effect size and credibility of your results, and you're basically required to report them. You're not forced to do those things well. Any scientific method can be cargo culted.

Andrew Gelman offered a brief, mild critique of steelmanning [1] which you may find worth reading. [1] https://statmodeling.stat.columbia.edu/2022/04/28/the-challenge-of-ben…

Hey, apparently the hot hand debate has been settled and there's actually solid evidence after all this time! [1] [2] [1] https://statmodeling.stat.columbia.edu/?s=hot+hand&submit=Search [2] https://statmodeling.stat.columbia.edu/2015/07/09/hey-guess-what-there…

No, nulls matter a great deal. If you want to test a claim in Null Hypothesis Statistical Testing, the "significance" of the claim is in direct reference to the null. Changing a null will change the significance of the alternative. My favorite statement of this is from Gelman: > the p-value is a strongly nonlinear transformation of data that is interpretable only under the null hypothesis, yet the usual purpose of the p-value in practice is to reject the null. My criticism here is not merely semantic or a clever tongue-twister or a “howler” (as Deborah Mayo would say); it’s real. In settings where the null hypothesis is not a live option, the p-value does not map to anything relevant. https://statmodeling.stat.columbia.edu/2017/01/07/we-fiddle-while-rome…

It's a great little portmanteau, I had to look up exactly what he meant though: https://statmodeling.stat.columbia.edu/2021/02/07/nudgelords/

Andrew Gelman has a great essay reviewing a paper that discusses this in the context of Covid: >Greenhalgh’s message is not that we need theory without evidence, or evidence without theory. Her message, informed by what seems to me is a very reasonable reading of the history and philosophy of science, is that theories (“mental models”) are in most cases necessary, and we should recognize them as such. We should use evidence where it is available, without acting as if our evidence, positive or negative, is stronger than it is. https://statmodeling.stat.columbia.edu/2021/10/22/how-did-the-internat…

> which is so obviously a quantum leap in terms of what it could do for human productivity that's a very strong claim. and it's very likely, unfortunately, false. we have the great classics all available for free in a multitude of formats. a lot of works slowly but surely mature into the public domain each year, yet there's no discernible productivity growth based on the availability of books. also, this sadly goes for research papers too. sure, fuck Elsevier and the whole paper mill industry, but the limiting factor is expertise, tenured spots, academic integrity. (see the replicability crisis in psychology, in particular how the many times the whole scientific enterprise is backwards: https://statmodeling.stat.columbia.edu/2011/01/11/one_more_time_o/ and https://statmodeling.stat.columbia.edu/2016/09/21/what-has-happened-do… )

Some data science blogs I enjoy: - https://erikbern.com/ - https://davefernig.com/ - https://statmodeling.stat.columbia.edu/

Statmodeling https://statmodeling.stat.columbia.edu/ Daring fireball https://daringfireball.net/ Tyler coven on marginal revolution https://marginalrevolution.com/

I can't assess the quality of PNAS, but the one thing I remember about it is that Columbia statistics prof Andrew Gelman disses them all the time for peddling pseudo-science based mostly on junk statistics[1]. He paints it as basically a science oriented clickbait tabloid. [1] https://statmodeling.stat.columbia.edu/?s=PNAS, https://statmodeling.stat.columbia.edu/2017/10/04/breaking-pnas-change…

I interpreted the "difference which makes a difference” as the sensitivity of our understanding of the status quo relative to the research instance. In that sense, the critique would be about estimating impact on some dimension of understanding conditioned on a finding of credible difference. That would say that big bets are better research. In other terms of critique, Gelman talks about two components (which may appear to be in tension), one referencing the craft of research, and the other in terms of apparent novelty of effect. My interpretation is that a critique of research should factor in both the character of the bet and an assessment of the evaluation. I don't think big is better, I think incisive is better, but that broader impact (news worthiness) is conditioned on credible surprise. [0] https://statmodeling.stat.columbia.edu/2014/08/01/scientific-surprise-…

The page linked by wpietri [0] [1] is worth reading. Guzey engages in a conversation with someone named Phil who did a "PhD in a circadian rhythms/sleep lab". No firm conclusions, but topics include how applicable animal models are to humans, and the difficulty of producing conclusive evidence about sleep in humans. Plenty of references, and little argument that aspects of Walker's book seem sensationalist. [0] wpietri's comment https://news.ycombinator.com/item?id=26684989 [1] https://statmodeling.stat.columbia.edu/2019/11/24/why-we-sleep-update-…

I second the recommendation of Gelman’s blog. He’s written a lot more about this particular case since then. Here’s a more recent article on the topic, for instance: https://statmodeling.stat.columbia.edu/2020/10/07/alexey-guzey-plays-s…

https://statmodeling.stat.columbia.edu/2016/05/05/npr-bites/ > In all seriousness . . . > I have no problem with NPR. NPR is great. That’s why I’m bummed when it falls for junk science.

The real issue is the picking of one function. I would be much more convinced of his analysis if he'd chosen the function randomly. That being said, I am very unsurprised at the results, given what I've seen of scientific code. However, it's worth noting that the estimates provided by these models in the past have been reasonable: https://statmodeling.stat.columbia.edu/2020/05/08/so-the-real-scandal-… (See the P.S. where he realises that the estimates are actually pretty reasonable).

No reason you should listen to me in particular (no doubt I have made egregious statistical errors myself in my research), but a good place to start might be https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval or, as others have posted, it's worth listening to Andrew Gelman (unlike me, a statistics expert) about this! https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaws-in-stanf… Essentially the problem with the study is that the bounds on the false positive rate of the test used was such that the data are completely consistent with all of the positives being false positives.

Worth looking through here and related links: https://statmodeling.stat.columbia.edu/2020/04/19/fatal-flaws-in-stanf…

Not my impression. The kind of outlier-culling technique suggested by civilized is not recommended these days because it adds unprincipled choice points to what Andrew Gelman calls the 'Garden of Forking Paths' [1, 2]. Thus they are bad for hypothesis testing, which tends to be what most statisticians care about. Additionally the technique obscures the relationship between the variance of the sample and the population variance if we do not have reliable knowledge of the population distribution; likewise for the mean if the mean is not close to the mode. These problems can be quite dramatic for long-tailed distributions. [1]: http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pd… [2]: https://statmodeling.stat.columbia.edu/2016/09/30/why-the-garden-of-fo…