Simply Statistics

My daily fix: Run by biostatisticians Jeff Leek, Roger Peng, and Rafa Irizarry, Simply Statistics (http://simplystatistics.org/) it's more than stats - and it's a must read. R-bloggers (http://www.r-bloggers.com/) is the hub for everything R.

I think Rafael Irizarry put it best over a decade ago -- while historically there was a feud between self-declared "frequentists" and "Bayesians", people doing statistics in the modern era aren't interested in playing sides, but use a combination of techniques originating in both camps: https://simplystatistics.org/posts/2014-10-13-as-an-applied-statistici…

Further examples from Simply Statistics. http://simplystatistics.org/2012/11/26/the-statisticians-at-fox-news-u…

Ill mention a few I tend to go back for sheer joy. I'm a data scientist, so the ones I really like tend to be statistics focused since (I don't do DL): - Statistical Models for Earthquake Occurrences and Residual Analysis for Point Processes [1] Brought to me by [this](https://simplystatistics.org/posts/2017-09-04-deep-dive-ogata/) blogpost a while ago, for me it shows what a really good statistical analysis based on a model developed for the task at hand looks like. Great stuff. - Conducting highly principled data science: A statistician’s job and joy [2] Meng is such a good writter. This one really puts into perspective the job of good principles when tackling any kind of task that a data scientist will face. Principles, theoretical principles, are what avoids messing up a task. 1. https://amstat.tandfonline.com/doi/abs/10.1080/01621459.1988.10478560 2. https://www.sciencedirect.com/science/article/pii/S0167715218300981

Andrew Beam's post offered very persuasive evidence that Jeff Leek's intuition (deep learning yields poor performance with small sample sizes) is incorrect. The error bars and the consistent trend of higher accuracy with a properly implemented deep learning model, particularly with smaller sample sizes, is devastating to Leek's original post. I think this is a fantastic example of the speed and self-correcting nature of science in the internet-age. As an aside, @simplystats blocked me on Twitter, which I assume is in response to this tweet: https://twitter.com/ErickRScott/status/871586233599893505 and it seems that I'm likely not the only one blocked: https://twitter.com/jtleek/status/871693250947624961 What's most concerning about @simplystats blocking activity is the chilling effect it has on discourse between differing perspectives. I've tried to come up with a rationale for why highlighting the most recent evidence in reply to someone who sympathized with Leek's original post (btw, @thomasp85 liked the tweet) is grounds for blocking , but I can't come up with a reasonable idea. Further aside, is irq11 Rafael Irizarry? Update: after emailing the members of @simplystats they have removed the block on my account and offered a reasonable explanation. SimplyStats is a force for good in the world (https://simplystatistics.org/courses/) and I look forward to their future contributions.

Part II With good stuff in ML you are describing a world I've seen no solid evidence of first hand. Maybe that's to be expected: Long I did well learning math in part because I picked some really good books; due to some lucky parts of my background, that was easy to do. Then I tried this in statistics and picked a junk book -- eventually figured that out. I didn't have much respect for or progress in statistics before I had a high end background in probability from a star student of E. Cinlar at Princeton from Neveu's book, Neveu a star student of Loeve at Berkeley. Then in stochastic processes I got some more respect from work of J. Doob (prof of Halmos who then became an assistant to von Neumann, etc.) and the material on probability in the back of the Halmos, Measure Theory. The Halmos work on sufficient statistics and unbiased estimation also deserved respect. I've seen no such quality from ML. You suggest that there is such quality out there, but what I saw, e.g., Ng's on-line lectures on ML, the idea of convolutional neural networks for image recognition, doesn't look nearly as solid as what I did in probability or nearly as interesting as some of what you point to. E.g., just yesterday I went to https://simplystatistics.org/ a Web site of three profs in bio-stat, and read some of the material of one of them, Jeff Leek, looked at the TOC and some of the lectures in his Coursera on-line course, some of the research problems he's working on, some of what he…

I hear you express this sentiment often. Could you explain your thoughts on this? If anything, I think the stories are the problem. Cool hacks and interesting science often fail to get the 2-3 upvotes they need to escape the new page. On page 2 of "New", I've found the following fairly decent stories that died recently: http://simplystatistics.org/2014/03/20/the-8020-rule-of-statistical-me… http://googlecloudplatform.blogspot.co.uk/2014/03/cassandra-hits-one-m… Yet both of these are vastly more useful than Julie Horvath's "omfg, I had a workplace conflict" or "Yay basic income, no facts, no reasoning, yay" (currently 380 pts, 441 comments, was #1 for a while today). I'd suggest that it's far more important to improve the stories than to improve the comments. The comments on a story like "JDK 8 Release Notes" or "Circuit Breaker" (about a distributed computing design pattern) are actually quite good.

Palantir does really well for what they do, but this article is as true today as it was in 2016 in terms of what they are https://simplystatistics.org/2016/05/11/palantir-struggles/ They're almost just a consultancy.

That particular issue can also be solved by setting options(stringsAsFactors = FALSE). Anyway, the default behaviour was the sensible thing to do at the time: https://simplystatistics.org/2015/07/24/stringsasfactors-an-unauthoriz…

A lot of researchers would disagree with the statement "That’s not how science works: You don’t sign a contract to not change anything in your protocol as the research evolves" You've got to complete the test using the protocol and hypotheses established at the outset. If you decide a change in protocol is required you wrap up the original test, report on the results, and then start a new test. Otherwise you end up with some combination of the issues described in A menagerie of messed up data analyses and how to avoid them[1] [1] http://simplystatistics.org/2016/02/01/a-menagerie-of-messed-up-data-a…

I think one of the core problems is summed up down near the end: the idea of statistics being taught by an apprenticeship model. There's a slightly more fleshed out version of this idea, also by Dr. Leek, over at SimplyStatistics[1] It might be my own statistics-porous media filter, but it seems like this issue is finally getting the attention it deserves. Hopefully this discussion will lead to bigger and better things. [1]http://simplystatistics.org/2015/04/29/data-analysis-subcultures/

As an academic statistician running the largest data science program in the world (10,000+ have completed a class) I think that CS/Data Science only pose a threat to statistics if we don't adapt. I wrote about it here. http://simplystatistics.org/2013/04/15/data-science-only-poses-a-threa…

You should also read the responses to that article, such as Ioannidis's, which is scathing. They're all open-access, thankfully, and you can find them here: http://simplystatistics.org/2013/09/25/is-most-science-false-the-titan… Jager and Leek put their code on GitHub, so the commenters were able to review their code and tinker with it to see what would happen.

Off topic, but since you mentioned jgrahamc's article in Nature, interestingly, this was what I read last night on Simply Statistics: http://simplystatistics.org/2013/01/23/statisticians-and-computer-scie… It's a similar issue. I think statisticians are taking constructive steps to correct their path, since you know, ML is the new sexy thing. Bioinformatics could take a much longer time to self-correct though. Although, as I mentioned in an earlier comment, Fred seems to be in a prime position to disrupt the bioinformatics field since he seems to know all the problems that afflict it

Growth Mr. Money Moustache [http://www.mrmoneymustache.com/] Study Hacks, Cal Newport [http://calnewport.com/blog/] Tim Ferriss [https://tim.blog/] Oliver Emberton [oliveremberton.com] Intellectual Farnam Street, Shane Parrish [farnamstreetblog.com/blog] Less Wrong [http://lesswrong.com/] [lesserwrong.com] Raw Thought, Aaron Swartz [aaronsw.com/weblog] Slate Star Codex [slatestarcodex.com] Edge [http://edge.org/] Melting Asphalt, Kevin Simler [http://www.meltingasphalt.com/] Essays, Paul Graham [paulgraham.com/articles.html] Minding Our Way, Nate Soares [http://mindingourway.com/] Ribbonfarm, Venkatesh Rao [ribbonfarm.com] Overcoming Bias, Robin Hanson [overcomingbias.com] Shtetl-Optimized, Scott Aaronson [http://www.scottaaronson.com/blog/] Raikoth, Scott Alexander [http://web.archive.org/web/20140220082152/http://raikoth.net:80/] Heterodox Academy [https://heterodoxacademy.org/] Technology Hacker News [https://news.ycombinator.com/] Worrydream, Bret Victor [http://worrydream.com/] Naval Ravikant [https://startupboy.com/] Unenumerated, Nick Szabo [http://unenumerated.blogspot.com/] Stratchery [http://stratchery.com/] Statistics DataTau [http://www.datatau.com/] FiveThirtyEight [fivethirtyeight.com] Simply Statistics [https://simplystatistics.org/] Chris Olah [colah.github.io] Economics Marginal Revolution [marginalrevolution.com] Project Syndicate [project-syndicate.org]

Speaking as an analytics architect ... You'll be a lot better off spending your mental energy thinking about the outcomes you want to achieve (user engagement, upselling, growth, etc) and the types of analysis you'll need to understand what changes you need to make to produce those outcomes. Protip: this is actually really hard, and people underestimate it by orders of magnitude. A blog post by Roger Peng (with indirect commentary from John Tukey) ... https://simplystatistics.org/2019/04/17/tukey-design-thinking-and-bett… One other immediate tip is to start thinking about correlating your telemetry with user surveys - again, strongly focusing on outcomes and the controllable aspects of those outcomes. Don't let the data lead the discuisson; decide on the question you're asking, and the implications of all of the possible answers to that question (clearly yes, clearly no, mixed, etc) before you ask it. Then engineer the lightest weight system possible to ingest, process, store, analyze, and visualize that data. For me, that would just be: 1. Log data in whatever logging tool you like. Persist the raw stuff forever in a cheap data lake. 2. Batch at some fixed interval into a staging area of a relational DB. 3. Transform it with stored procedures for now (while you figure out what the right transforms are) into a flat fact table. 4. Visualize in Superset or PowerBI or even plain old Excel. Once you've got the patterns of analysis at least fundamentally right…

1. AVC : http://avc.com/ 2. Feld Thoughts : http://www.feld.com/ 3. Benedict Evans : http://ben-evans.com/benedictevans/ 4. andrewchen (Growth at Uber): http://andrewchen.co/ 5. Ben Horowitz: https://a16z.com/author/ben-horowitz/ 6. Accel: http://theaccelblog.squarespace.com/blog/ 7. Both Sides of the Table：https://bothsidesofthetable.com/ 8. cdixon blog：https://medium.com/@cdixon/ 9. Coding Horror：https://blog.codinghorror.com/ 10.Erica Swallow: https://ericaswallow.com/ 11.First Round Capital Review: http://firstround.com/ 12.Guy Kawasaki：https://guykawasaki.com/ 13.Hunter Walk: https://hunterwalk.com/ 14.Joi Ito : https://joi.ito.com/weblog/ 15.Marco.org: https://marco.org/ 16.Paul Graham: http://www.paulgraham.com/ 17.Sam Altman: http://blog.samaltman.com/ 18.Simply Statistics: http://simplystatistics.org/ 19.Ben Thompson: https://stratechery.com/ 20.The morning paper: https://blog.acolyer.org/

I like this summary of the evidence that most published research is false[1]: -"One of the hottest topics in science has two main conclusions: Most published research is false There is a reproducibility crisis in science The first claim is often stated in a slightly different way: that most results of scientific experiments do not replicate." - [Ioannidis'] "Paper: Why most published research findings are false. Main idea: People use hypothesis testing to determine if specific scientific discoveries are significant. This significance calculation is used as a screening mechanism in the scientific literature. Under assumptions about the way people perform these tests and report them it is possible to construct a universe where most published findings are false positive results. Important drawback: The paper contains no real data, it is purely based on conjecture and simulation." - then it summarises in the same way 7 other papers, including the Many Labs results. -"I do think that the reviewed papers are important contributions because they draw attention to real concerns about the modern scientific process. Namely We need more statistical literacy We need more computational literacy We need to require code be published We need mechanisms of peer review that deal with code We need a culture that doesn't use reproducibility as a weapon We need increased transparency in review and evaluation of papers" - the final paragraph: "The Many Labs results…

This, I think, is the study the first poster meant, Ioannidis 2005, 'Why Most Published Research Findings Are False': http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pme… It's quite a famous paper. Many rebuttals, comments, blog-posts etc, have been published, have a look at the comments on the PLOS site, I especially like this blog post summarizing more research: http://simplystatistics.org/2013/09/25/is-most-science-false-the-titan… From the same author, here's a small overview of the discussion at the end of 2013: http://simplystatistics.org/2013/12/16/a-summary-of-the-evidence-that-… Quote on the Ioannidis paper: >Under assumptions about the way people perform these tests and report them it is possible to construct a universe where most published findings are false positive results. Important drawback: The paper contains no real data, it is purely based on conjecture and simulation.

R and matlab are better, but those tools also have issues integrating into production depending on what you are doing. It's not so much the exact tool you use, but just having a little forethought about how your creation is going to interact with a production system. A lot of people feel programming is undervalued in academia. For instance Hadley Wickham creator of ggplot2 probably hasn't gotten the recognition he deserves. With a prevailing attitude such as that is it any wonder academic code has such a poor reputation? Whickham notes that he thinks tides are changing. I agree that it is, as a part of the datascience phenomenon. As part of the change you are going to see a few more macbooks, some cloud servers, maybe a guy with glasses talking about version control and software design. It is not all garbage, I hope you keep an open mind. Q:Do you feel that the academic culture has caught up with and supports non-traditional academic contributions (e.g. R packages instead of papers)? A:It’s hard to tell. I think it’s getting better, but it’s still hard to get recognition that software development is an intellectual activity in the same way that developing a new mathematical theorem is.[1] 1. http://simplystatistics.org/2012/05/11/ha/

I was very surprised when I first learned that results could be published without first having been reproduced. A lot of psychology journals have been burned by that over the years, http://chronicle.com/blogs/percolator/is-psychology-about-to-come-undo… http://simplystatistics.org/post/21326470429/replication-psychology-an… http://scholarlykitchen.sspnet.org/2012/04/26/reproducibility-an-attem… and as a consequence, just within the last year or so, the most prestigious psychology journals are beginning to ask authors of papers on new experimental research to show replication across at least two distinct data sets. This, in turn, is driving more collaboration among researchers at different research centers. It's a long overdue development, one in accord with best practice in science, http://norvig.com/experiment-design.html and I hope this practice spreads to most disciplines that publish new research reports in the major journals of the discipline. (Although the requirement for prepublication replication is not explicitly mentioned in the publication guidelines of Psychological Bulletin, http://www.apa.org/pubs/journals/bul/index.aspx which is the most prestigious psychology journal published in the United States, I have been told at the University of Minnesota behavior genetics "journal club" that replication across more than one data set is becoming a tacit requirement for…