Tal Yarkoni

https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-i… Humans are not incentive-following automata, though we have incentive-following tendencies.

R isn't a joy, but it is powerful. This is the best blog post I've read on the matter: https://www.talyarkoni.org/blog/2012/06/08/r-the-master-troll-of-stati…

The best part of Sturgeon's Law for me was realizing that it's tautological. It's just defining "crud" to be the bottom 90%, which is drawing an arbitrary line between the top and the bottom. It's a fun one in the same way that D/K is in that it just feels true, like it's validating a working theory you already had about the world. But Sturgeon's law can't be false, whereas D/K really is commonly misunderstood and might not even be generally reproducible; it might not even be broadly true at all. D/K is certainly over-generalized and in popular culture applied to situations that are very unlike the experiments in the paper, which were only pretty trivial tasks. The D/K Effect "actually goes away, and even reverses, when the ability tests given to participants are very difficult." https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-eff…

I recently wrote the following in response to a different comment here on HN: > "I was just following incentives" is becoming the new Nuremberg defense. https://news.ycombinator.com/item?id=19709730 I'm clearly not the only one who has had this thought... Blog post on this sort of issue in academia: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-it…

While I like seeing the Dunning--Kruger effect pointed out here as much as the next man, I have to comment that the original study was not as simple as it is often described. There's a great description on the [citation needed] blog: http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-effe… Here's the money quote in my opinion: > So the bias is definitively not that incompetent people think they’re better than competent people. Rather, it’s that incompetent people think they’re much better than they actually are. But they typically still don’t think they’re quite as good as people who, you know, actually are good. (It’s important to note that Dunning and Kruger never claimed to show that the unskilled think they’re better than the skilled; that’s just the way the finding is often interpreted by others.)

> I can't blame researchers for tailoring their research to help their own careers but this should help prevent that. It regularly astonishes me how easily people are willing to accept scientific malpractice with such excuses. While I agree that mechanisms like registered reports are the way to fix these things at the core, I do think that these mentalities ("it's just the system, I can't do anything but cheat with my scientific publications, my career!") are a big part of the problem. Good read on the issue: http://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-it…

It's worth pointing out that the original Dunning-Kruger paper didn't find that the ignorant are more confident than the knowledgeable, only that the ignorant are overconfident and the knowledgeable are underconfident. Many people have in mind a curve of confidence vs knowledge[0] that looks nothing like the curve from the paper[1]. [0] https://imgur.com/r/psychology/jbo2gy5 [1] http://www.talyarkoni.org/blog/wp-content/uploads/2010/07/dunning_krug… with a good writeup at http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-effe…

Title repeats an oft-repeated misrepresentation of what the Dunning-Kruger study actually demonstrated. See http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-effe… Key sentence: "Unfortunately, Kruger and Dunning never actually provided any support for this type of just-world view; their studies categorically didn’t show that incompetent people are more confident or arrogant than competent people. What they did show is this: [image showing their data, followed by explanation of what regression towards mean is]"

I recently saw this (which goes into far more depth): http://www.talyarkoni.org/blog/2013/11/18/the-homogenization-of-scient… Summary: "Constantly switching languages is a chore, and while Python isn't the best at everything, it's the best at some things and good enough at almost everything else."

http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-effe… This recent post shot down a lot of things being mistakenly attributed to the Dunning-Kruger effect. Well worth the read, since it actually explains the study in a more comprehensive way, rather than the somewhat sensationalized version most articles have. Essentially, it says that the worst people judge their abilities to be on par with average people, thus grossly overestimating their abilities. The smartest people somewhat underestimate their abilities, but not nearly to the degree that the truly incompetent do.

I'm sorry, but this cop-out really pisses me off. It is far too common and frankly, unacceptable. It really is insulting that you'd expect others to accept this as a justification. It's a lazy dismissal and not even a proper excuse. You're excuse for doing something shitty is... that someone else will? What does another person even have to do with it?! Seriously, let them have the blood on their hands. You can't even assume that someone else will! If you do it, you guarantee that it happens. Even if it is likely that someone else will, there's a big difference between a certainty. This is literally what creates enshitification. Plus, the logic is pretty slippery. Certainly you're not going to commit crimes or acts of genocide! You were "just following orders"[0], right? Or parents often say to their children "if everyone jumped off a cliff, would you?" Certainly the line is drawn somewhere, but frankly, it is the same "excuse" given when that extreme shit happened, so no, I won't accept it. You have autonomy[1], that makes you accountable. You aren't just some mindless automata. You may not be the root cause, but at best you enable it. You can't ignore that you play a role. And consider the dual: if you don't make it better, who will? I believe you have the power to make change, do you? Maybe not big, but hey, every big thing is composed of many smaller things, right? So the question is which big thing you want to contribute to. [0]…

Also, it's extremely hard to account for all kinds of biases and systematic errors in a meta-analysis. I'd take most meta-analyses, even by renowned statiticians, very, very skeptically. Teasing out small effects from samples that need corrections for lots of confounding factors, and then trying do combine many such results is next to impossible. If your effect is small enough that you need a meta-analysis in the first place, that's a warning sign. Some background: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0152… (with perhaps more approachable explanation here https://www.talyarkoni.org/blog/2016/06/11/the-great-minds-journal-clu… ) Obviously it's sometimes possible to do this right, but the reproducibiility crisis didn't appear out of nowhere; it's best to be wary of overly clever statistics; hard to know what you don't know.

This paper discusses a major issue that I think deserves way more attention from the academically literate crowd. This is the kind of knowledge that should be taught in secondary school, at least in the modern age with papers at everyone's fingertips. A more approachable summary is in narrative form at https://www.talyarkoni.org/blog/2016/06/11/the-great-minds-journal-clu… > “But there’s a problem: statistical control–at least the way people typically do it–is a measurement-level technique. Meaning, when you control for the rate of alcohol use in a regression of cancer on bacon, you’re not really controlling for alcohol use. What you’re actually controlling for is just one particular operationalization of alcohol use–which probably doesn’t cover the entire construct, and is also usually measured with some error.” I strongly suggest reading the comment on that page from one of the authors as well.

I know that this article is fiction, but there's a common misconception about the Dunning-Kruger effect presented in the story that is worth understanding. People think that Dunning and Kruger showed that less competent people think that they're more competent than more competent people think they are, but this is not what Dunning and Kruger showed, if you'll excuse the complicated sentence construction (the article linked below has a graph and explains this a bit better). Actually, Dunning and Kruger simply showed that a person's estimate of their own skill tends to be closer to the mean, plus a little than their actual skill level. The Dunning-Kruger effect has probably been a bit distorted because we like the idea that thinking you are good at something is evidence that you are not, in fact, good at that thing. See also "what the Dunning-Kruger effect is and isn’t" - https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-eff…

Reminds me a bit of this blog post by Tal Yarkoni, https://www.talyarkoni.org/blog/2018/08/18/if-we-already-understood-th… That is perhaps more consistent with the "we are just really complicated neural networks argument".

Feels like this should have 2017 in the title. This is just a light overview of what the Dunning-Kruger effect is (or was claimed to be), Pat sounds extremely fictional and conveniently useful to help describe "the effect" (which - if you search HN you'll find multiple links to a 2010[0] article challenging DKE, even the of-submitted Wikipedia page[1] expresses doubt). [0]: https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-eff… [1]: https://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect

For the other side "No, it’s not The Incentives—it’s you" https://www.talyarkoni.org/blog/2018/10/02/no-its-not-the-incentives-i…

I link this[1] every time I see it mentioned, because the standard depiction of it bears no resemblance to what's in their paper. [1]: https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-eff…

I think there's a better explanation at https://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-eff… which also discusses some possible alternative explanations.

I'm not sure people are trying to fix academic publishing so much as burn Elsevier to the ground. Whether or how they are replaced is a separate question. https://www.talyarkoni.org/blog/2016/12/12/why-i-still-wont-review-for…

Law-abiding is by no means synonymous with good, or even neutral in a moral sense. Obligatory reposting of old HN content (especially under the section "Is Elsevier really so evil?"): "Why I still won’t review for or publish with Elsevier–and think you shouldn’t either" http://www.talyarkoni.org/blog/2016/12/12/why-i-still-wont-review-for-… I'm also discovering that law-abiding is as much a reputation as a technicality. Companies get away with breaking the law all the time. They just aren't universally or publicly punished for it.

> It's natural. It's Dunning Kruger effect. People become very confident when they first learn a little and consider themselves experts. The Dunning Kruger effect doesn't support what you claimed [0][1]. (Provided that I read your comment correctly). [0] https://danluu.com/dunning-kruger/ [1] http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-effe…

This post has a lot of details about it: http://www.talyarkoni.org/blog/2016/12/12/why-i-still-wont-review-for-…

If you read about the Dunning-Kruger effect, it is essential to also read its' criticism, very aptly summarized here: http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-effe… (posted here numerous times; the truth is, well, we all suck at heuristic self-assessment, regardless of our skill, sorry). It's always easy to write off other people as "incompetent", blame Dunning-Kruger, and just ignore what they say and think they know. A lot of physicians tend to do that, and, frankly speaking, it just comes off as arrogance and does not help confidence at all.

Here you have an exact explanation of what Dunning-Kruger effect is. http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-effe…

tl;dr "So the bias is definitively not that incompetent people think they’re better than competent people. Rather, it’s that incompetent people think they’re much better than they actually are. But they typically still don’t think they’re quite as good as people who, you know, actually are good." and graphic: http://www.talyarkoni.org/blog/wp-content/uploads/2010/07/dunning_krug…

I liked Tal Yarkoni's blog post on Dunning-Kruger http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-effe… - even David Dunning posted a comment

It's also, unfortunately, deeply harmful to humanity. http://www.talyarkoni.org/blog/2014/06/28/in-defense-of-facebook/ > by far the most likely outcome of the backlash Facebook is currently experience is that, in future, its leadership will be less likely to allow its data scientists to publish their findings in the scientific literature

http://www.talyarkoni.org/blog/2010/07/07/what-the-dunning-kruger-effe… A glance at the graph above should explain everything. Though many assume the slope of the "perception graph" is monotonically negative, the slope is actually positive.

I don't use R. I use [pandas] / [statsmodels] in Python. I don't do particularly special stats though. I read this [blog] earlier and as a result I don't think I'll bother to learn R unless I have to. [pandas]: http://pandas.pydata.org/ [statsmodels]: http://statsmodels.sourceforge.net/ [blog]: http://www.talyarkoni.org/blog/2013/11/18/the-homogenization-of-scient…