Replicability-Index

Kahneman recently (Feb 2017) wrote an online response addressing the replication issues being experienced by some of the studies that form the basis of Thinking Fast and Slow. It's one of the best responses to criticism I've ever read: https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-…

I'd like to take this opportunity to remind everyone here both of the replication crisis: https://en.wikipedia.org/wiki/Replication_crisis and of hindsight bias: https://en.wikipedia.org/wiki/Hindsight_bias I see quite a few comments of people looking at the result and trying to come up with explanations for why it might hold. Heck, there even seems to be some confusion with the term "bullshit-receptivity" (and "bullshit-sensitivity"), which means that, even if the paper is valid, you might be drawing the opposite conclusion. I'd like you to imagine, without going back to check right now, that you are wrong about the result. That they've found the opposite correlation to the one you thought holds. Can you find a reason why it would? Does it seem justifiable, explainable? As much so as your original belief? That should be a warning sign that your explanation might be a bit pre-emptive. I am glad for the commenters saying they found the result "counter-intuitive". Noticing your confusion in this way is a good skill to learn, especially in this day and age. That nagging feeling in the back of your mind that something is off or that you forgot something. ----- As an aside: If you are interested in more on biases, may I suggest reading "Thinking, Fast and Slow" by Daniel Kahneman. One of my favourite non-fiction books. Some of the results in there (notably priming) have themselves unfortunately fallen prey to the replication crisis, but the book as a whole is still…

My point is not that this work is flawed, or that there should not be an article reporting on this research or the topic of interviewing. Rather, I think it'd be better for a third party to write about the topic in a more objective manner, rather than a professor promoting his own research (and thus with skewed incentives). In particular, I was disappointed to find a (short) paragraph in the article that I find bogus. That does not mean the article shouldn't have been posted in the first place, but just that this paragraph should have been edited or removed. I think there is something wrong when I feel like I have to look up the actual research paper and check whether the claims made in an article are supported by data and methodology. I should not be a skeptic when reading New York times articles. To be fair, it is posted in the opinion section, but should we really just take this article as an opinion? That doesn't feel right to me either. Then onto your last question and Daniel Kahneman, we can talk about that for a long time, but let me keep it short. The best place I know (though technical) is the blog by Andrew Gelman ([1][2] turned up in a 5 second Google, but there is way more on his blog), and Daniel Kahneman himself has "admitted" flaws in his studies [3][4]. [1] http://andrewgelman.com/2014/09/03/disagree-alan-turing-daniel-kahnema… [2] http://andrewgelman.com/2016/06/26/29449/ [3]…

See also Kahneman's response to this: https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-… > I still believe that actions can be primed, sometimes even by stimuli of which the person is unaware. There is adequate evidence for all the building blocks: semantic priming, significant processing of stimuli that are not consciously perceived, and ideo-motor activation. I see no reason to draw a sharp line between the priming of thoughts and the priming of actions. A case can therefore be made for priming on this indirect evidence. But I have changed my views about the size of behavioral priming effects – they cannot be as large and as robust as my chapter suggested. (Discussed on HN: https://news.ycombinator.com/item?id=15228712)

Much of behavioral economics is based on a very shaky foundation of psychology. For instance, the priming experiments cannot be reproduced. > This result confirms Kahneman’s prediction that priming research is a train wreck and readers of his book “Thinking Fast and Slow” should not consider the presented studies as scientific evidence that subtle cues in their environment can have strong effects on their behavior outside their awareness. https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-…

Ranking of universities by replicability https://replicationindex.wordpress.com/2016/03/02/replicability-rankin…

Except entire chapters of that book are completely unscientific garbage, most notably the one on priming. See this: https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-…

Another example of studies that seem to be hard to dublicate is about the concept of priming [0]. I think sometimes the non-scientific community picks up an idea which just seems so good and starts to apply it to an array of life sitations, while the idea itself might be limited to a small amount of sitations/conclusions. Especially with priming, I read some hair-raising statemens from laymen, about how priming seems to explain virtually everything. It also happens to scientists, of course, but they tend to be a bit more careful and stick a little closer to the original experiment (which itself might not be evidence enough for the conclusion). [0] https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-…

Careful with drawing too many conclusions from that book. Your take away is pretty ironic, considering that was Kahneman's problem: "I placed too much faith in underpowered studies" [1] [1] https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-…

yes I put the link first before the quote https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-…

Is the article meant to be a link to Kahneman's comment https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-… that you quote elsewhere (https://news.ycombinator.com/item?id=15228754)? It seems weird for it to be a link to this article itself.

A lot has changed in the brief time since this article was written. Much of the evidence in support of priming was revealed to be based on weak research that didn't replicate. Don't rely on that section of the article. https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-… http://nautil.us/blog/impossibly-hungry-judges http://slatestarcodex.com/2016/08/25/devoodooifying-psychology/

No there are bayesian replacements for p values as well: https://replicationindex.wordpress.com/2015/04/30/replacing-p-values-w…

> a recent-ish study that compared test performance of various minority/gender groups... You are referring to the idea of "stereotype threat" which, alas, did not survive the replication crisis. Quote "After correcting for publication bias, this literature shows very little evidence that stereotype threat has a notable and practically significant effect on women’s math performance (Flore & Wicherts, 2014)." Some analysis: https://replicationindex.wordpress.com/2017/04/07/hidden-figures-repli… Direct study link: http://www.sciencedirect.com/science/article/pii/S0022440514000831

Regarding cognitive differences between men and women having a biological component: https://stanmed.stanford.edu/2017spring/how-mens-and-womens-brains-are… Regarding the quality and replicability of stereotype threat studies, that came in vogue after the seminal Steele & Aronson 1995 paper: https://replicationindex.wordpress.com/tag/does-stereotype-threat-infl… https://replicationindex.wordpress.com/2017/04/07/hidden-figures-repli… https://heterodoxacademy.org/2015/12/30/is-stereotype-threat-overcooke…

"Stereotype threat" isn't doing well in replication. https://replicationindex.wordpress.com/2017/04/07/hidden-figures-repli… Money quote: "Research on stereotype threat and women’s performance on math tests is one example where publication bias undermines the findings in a seminal study that produced a large literature of studies on gender differences in math performance. After correcting for publication bias, this literature shows very little evidence that stereotype threat has a notable and practically significant effect on women’s math performance (Flore & Wicherts, 2014)."

"In conclusion, a replicability analysis with the R-Index shows that stereotype-threat is an elusive phenomenon. Even large replication studies with hundreds of participants were unable to provide evidence for an effect that appeared to be a robust effect in the original article. The R-Index of the meta-analysis by Flore and Wicherts corroborates concerns that the importance of stereotype-threat as an explanation for gender differences in math performance has been exaggerated. " https://replicationindex.wordpress.com/tag/does-stereotype-threat-infl…

One thing to note is that the social priming studies in “Thinking Fast and Slow” are not very replicable [1] 1. https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-…

Reply by Kahneman in the comments: https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-… Further discussion on reddit: https://www.reddit.com/r/slatestarcodex/comments/5u7fb3/a_piece_about_…

Stereotype threat is yet another victim of the replication crisis - i.e. it probably doesn't exist. It's repeatedly failed to replicate. Funnel plots show significant evidence of publication bias. http://www.sciencedirect.com.sci-hub.cc/science/article/pii/S002244051… https://dl.dropboxusercontent.com/u/85192141/2013-ganley.pdf https://en.wikipedia.org/wiki/Stereotype_threat#Failures_to_replicate_… https://replicationindex.wordpress.com/tag/stereotype-threat-and-women…

Source: https://replicationindex.wordpress.com/2016/04/18/is-replicability-rep…

Before this thread gets too negative, I would like to point out a positive example of intellectual humility: Kahneman's retraction of a chapter of Thinking Fast and Slow [1]. In it, Kahneman acknowledges the basic points of of the Retraction Watch article, explains his mistake, and says specifically what parts of his worldview have changed as a result of the new evidence that he's been faced with. He clearly says that, although he still thinks that priming is real, he has significantly more doubts now about the generality and replicability of priming research, now that study after study has failed to find the effects of priming when attempting to reproduce famous results. He even says that he had the information to come to the same conclusions as the blog, but failed to do so because of his own biases. [1]: https://replicationindex.wordpress.com/2017/02/02/reconstruction-of-a-…