Replication Index

It's such a great book. You can skip the first few chapters as the evidence for those chapters on priming hasn't been replicated: https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec… However, the fundamental ideas of System 1 and 2 have made me rethink so many things.

kahneman was such a fascinating personality. Other than "Thinking fast and slow", I highly recommend "The Undoing Project" by Michael Lewis about Kahneman and Tversky's incredible journey changing the standard economic theory. Some interesting talks with Daniel Kahneman - https://www.edge.org/adversarial-collaboration-daniel-kahneman - https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec… Kahneman himself reponds in the comment sections to a very critical piece about his work.

For what it's worth, Kahnman answered a post that scrutinized the effect of priming: https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec…

> It’s all a bit Dunning-Kruger to me Fantastic article on the reproducability of Dunning-Kruger effect: https://replicationindex.com/2020/09/13/the-dunning-kruger-effect-expl… Also see: "The Dunning-Kruger Effect is Autocorrelation" https://economicsfromthetopdown.com/2022/04/08/the-dunning-kruger-effe… Lovely quote: “These responses to our work have also furnished us moments of delicious irony, in that each critique makes the basic claim that our account of the data displays an incompetence that we somehow were ignorant of.” (Dunning, 2011, p. 247). Of course Dunning-Kruger is self-referential. Any mention of Dunning-Kruger automatically makes you a victim on wrong side of the graph.

Note: I have neither read the book, nor knew about this issue before. The following articles look relevant: https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec… https://retractionwatch.com/2017/02/20/placed-much-faith-underpowered-… Per the following, not all types of priming are under controversy. https://en.wikipedia.org/wiki/Priming_(psychology)#Replicability_contr…

And you're well served with this response, as the actual work he did is basically not reproducible. No matter how many best selling books made into movies; it's still bullshit. https://replicationindex.com/category/thinking-fast-and-slow/ https://slate.com/technology/2016/12/kahneman-and-tversky-researched-t… https://jasoncollins.blog/2016/06/29/re-reading-kahnemans-thinking-fas… I read it on advice of a good pal, and it sure seemed like bullshit to me.

Regarding Kahneman, keep in mind that in the following years there has been a lot of doubt - including by Kahneman himself - about the scientific validity of the evidence. See also: https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec… https://retractionwatch.com/2017/02/20/placed-much-faith-underpowered-…

A tiny bit of searching: https://replicationindex.com/2018/01/05/bem-retraction/ and a lovely quote: “I’m all for rigor, but I prefer other people do it. I see its importance—it’s fun for some people—but I don’t have the patience for it. If you looked at all my past experiments, they were always rhetorical devices. I gathered data to show how my point would be made. I used data as a point of persuasion, and I never really worried about, ‘Will this replicate or will this not?” (Daryl J. Bem, in Engber, 2017) also http://osc.centerforopenscience.org/2014/06/25/a-skeptics-review/ but I don't like this: "If ever there was a paper that showed the futility of meta-analysis, this one is it. Here we have one of the most ridiculous claims that intelligent people have ever dared to make (yes; the hypothesis that aliens built the pyramids is more plausible than people being able to look into the future) – and a meta-analysis supports this claim. The unavoidable conclusion is not that psi exists; rather, it is that meta-analysis is a tool that is fraught with danger". The other points are perhaps valid. You talk about science but don't have the basic critical thinking to look for opposing viewpoints. You should be seriously embarrassed.

A good time to remind everyone that for almost 2 decades now it's been determined that the majority of published scientific findings are wrong. [0][1] Including possibly even the very determination that the majority of published scientific findings are wrong.[2] ([3]) So, if that's the case, when a new result comes out, the appropriate reaction is to assume there's a better than average likelihood it will be refuted. And honestly that's what makes science great. Unlike with some other fields of human endeavor, it is possible to firmly refute bad science. And often learn something new in the process. [0] https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed… [1] https://www.annualreviews.org/doi/10.1146/annurev-statistics-060116-05… [2] https://replicationindex.com/2019/01/15/ioannidis-2005-was-wrong-most-… [3] https://en.wikipedia.org/wiki/Epimenides_paradox

> I don't think we are near general AI Nor do I. But I don't agree with your definition of general AI at all. I think by your definition we are well on the way towards it. I'd note that you didn't address the idea that compression of input concepts into a lower dimensional space is exactly the "reliably turning symbols into meaning" idea you suggest. The fact the latent representations of concepts can be manipulated in ways that make logical sense is a good indication that the symbols have meaning. The classic Word2Vec experiments showing how the relationships Paris->France ~= London->England and King - Man + Woman = Queen show this well. Modern large language models are much more complicated of course, but the principles remain. > I subscribe to the view we think of the human mind as a pattern matching computer simply because this is the current major tech, much like people in the past thought of "humors" or "steam machines". I think some of the analogies are useful, to a point, but I don't think there's hard evidence the mind is like a neural net (irony notwithstanding) or a pattern marching GPT-like algorithm. I don't think the computational approach is an interesting question. No one thinks brains operate like software neural networks (not sure what "pattern marching GPT-like algorithm" means - it is just a neural network). That doesn't matter because all computational methods are ultimately equivalent. I think outcomes on metrics like benchmarks is important, and…

> The results were staggering; the more prevalent the idea, in any academic field, the fewer women and people of color participating in that field. This outcome held across all thirty subjects in the study. This is not evidence that natural talent is not needed, it is at best some evidence that minorities and women maybe don't think they have the requisite natural talent. > Negative messages, either explicit (“I think you’d be happier if you didn’t take that hard mathematics class”) or implicit (“I’m just not a math person”), both imply that only some people can succeed. To say the evidence for stereotype threat is "weak" is charitable at best [1], and multiple replication attempts have failed entirely. Overall, the evidence that was cited is simply not sufficient to justify the principles you list. [1] https://replicationindex.com/2017/04/07/hidden-figures-replication-fai…

I read the paper, and agree with your impression of its quality. With training, it is possible to navigate a shaky field like social psychology, so I'll provide my estimates here. The paper was written in 2013, so it's not pre-registered. The sample size of 39 is suspicious given its date. People became upset about junk papers two years before, so the authors should have known better. However, I would hesitate to draw conclusions about the integrity of the paper's authors, because each participant was paid $60, which makes expanding the sample size somewhat expensive. The authors declared no conflicting interests, but I looked up the first author and he has major conflicts of interest. The last author, DeSteno, is ranked very poorly at https://replicationindex.com/2021/01/19/personalized-p-values/. Combining these factors, I'd adjust the p-value from .02 to .45, if it were a pre-registered paper which you have pre-committed to read. This paper is worthless.

You mean the same Kahneman who got: - A Nobel Prize in Economics. - Whose book has almost 20,000 reviews with an average of almost 5 stars. - Whose book was winner of the National Academy of Sciences Best Book Award in 2012 and then ... got most of the findings undermined by this blog analysis: "Reconstruction of a Train Wreck: How Priming Research Went off the Rails" https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec… and then... commented as himself in the same blog https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec… stating and I quote : "I accept the basic conclusions of this blog..." You mean that one ? In case you are thinking anybody could create an account under his name ...I provide you with an additional reference: https://retractionwatch.com/2017/02/20/placed-much-faith-underpowered-… Lets just accept most of Academy nowadays is not far from a total joke.

The 'Dunning-Kruger effect' of internet folklore is not what was in the original paper. And the original paper hasn't held up to well either: https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect#Statistica… (and Wikipedia is very mild and nice to the authors). See eg https://replicationindex.com/2020/09/13/the-dunning-kruger-effect-expl…

The content on "priming" (significant pillar of the book) has collapsed as part of the reproducibility crisis in psychology. More here: https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec…

FWIW, Ioannidis never demonstrated that a certain number of findings (or most) in a specific discipline are actually false - he calculated estimates based on assumptions. While Ioannidis work is important, and his claims may be true for many disciplines, a more nuanced view is helpful. For example, here's an article that argues (with data) that there is actually little publication bias in medical studies in the Cochrane database: https://replicationindex.com/2020/12/24/ioannidis-is-wrong/

The studies in Thinking Fast and Slow mostly failed replication: https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-…

My initial reaction is that it is Kinda sad Kahneman's debunked work was labeled as a pervasive game changer. https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-…

My understanding was that System 1/System 2 thinking is unproven conjecture[1] that can't even be replicated[2]. It would be unwise to analyse behaviour using this framework. 1: https://www.psychologytoday.com/intl/blog/a-hovercraft-full-of-eels/20… 2: https://replicationindex.com/2016/01/31/a-revised-introduction-to-the-…

Priming for starters https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec…

This is a famous experiment. To summarize the aftermath: - Strong statistical evidence of tampering with the test. Bem has been doing more test than he has recorded, and removed failed tests from the statistics. - Replication studies can't find any effect. Raising the value of research studies in psychological science by increasing the credibility of research reports: the transparent Psi project https://royalsocietypublishing.org/doi/full/10.1098/rsos.191375 "Too good to be true: Publication bias in two prominent studies from experimental psychology", Psychonomic Bulletin & Review. 19 (2): 151–156. doi:10.3758/s13423-012-0227-9 http://www2.psych.purdue.edu/~gfrancis/Publications/GFrancis-R1.pdf The Test of Insufficient Variance (TIVA): A New Tool for the Detection of Questionable Research Practices https://replicationindex.com/2014/12/30/tiva/

My comment about priming is due to articles like https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec… As https://replicationindex.com/2017/02/02/reconstruction-of-a-train-wrec… shows, Daniel Kahneman came to agree that priming research was bunk. I included that blog post as a guide to what other results may be suspect.

About his book: https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-…

Me too, I think a bunch of it was found to be reproducible though. https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-… Same thing is happening w/ a lot of the work of Dan Ariely, but I think his situation is much worse.

"Thinking, Fast and Slow" was written before the replication crisis was found. I wouldn't call it "debunked", but some of the research it used didn't replicate. For example, see https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-… : > It is likely that Kahneman’s book, or at least some of his chapters, would be very different from the actual book, if it had been written just a few years later. However, in 2011 most psychologists believed that most published results in their journals can be trusted. [...] > Kahneman also started to wonder whether some of the results that he used in his book were real. A major concern was that implicit priming results might not be replicable. [...] Anyways, just google "thinking fast and slow replication crisis" to get a bunch of information about this topic.

Very interesting examples. For Nash, Von Neumann contributions to the field were orders of magnitude more important. For Kahneman and Tversky let's not forget that the foundations are quite shaky to quote [0] [0] https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-…

[1] below goes into detail on one of the topics, [2] is a less technical / more journalistic take. Essentially Kahneman ended up being super confident ("disbelief is not an option" he said) about the findings he cited, some of which have been shown to suffer from lack of rigor. If you're wondering "well, just some of them right?" I will ask you to ponder for a minute over the fact that this is not supposed to be some impulse aisle magazine article but a book applied epistemology ("behavioural economics" is to me just what gave this and related books some sales wheels). [1] https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-… [2] https://slate.com/technology/2016/12/kahneman-and-tversky-researched-t…

Would that be in an effort to more accurately model human thinking? If so, I hate to be the one to tell you that > Readers of “Thinking: Fast and Slow” should read the book as a subjective account by an eminent psychologists, rather than an objective summary of scientific evidence https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-…

For any readers of "Thinking: Fast and Slow", this blog on replicability with a reply from the author Daniel Kahneman is fascinating: https://replicationindex.com/2020/12/30/a-meta-scientific-perspective-…