Frank Harrell

Frank Harrell writes a lot of great stuff and his answers on the Cross Validated Stack Exchange site are worth just reading even if you didn't think you wanted to ask the question they reply to. His blog, http://www.fharrell.com, also contains interesting posts.

It's not about revealing information, it's about how you model the problem. Say you're using logistic regression. If you upsample the minority class (never downsample the majority class because that's statistically inadmissible), then your classifier might get better at modeling the decision boundary. However, that comes at the cost of losing the best part of logistic regression: probability prediction and calibration. Sometimes the trade-off is worth it, sometimes not. Depends on the type of problem. For more details I suggest reading Frank Harrell: https://www.fharrell.com/post/classification/

Both the Brier score and log loss are proper scoring rules (i.e. optimized when the predicted probabilities are the true outcome probabilities), and the choice between the two seems to have minimal impact on the conclusions that can be drawn (https://pubsonline.informs.org/doi/abs/10.1287/deca.2013.0280). I covered the Brier score in the post as I thought it would be easier to digest for a general audience. As Frank Harrell wrote on his blog (https://www.fharrell.com/post/class-damage/), one advantage of the Brier score could be its interpretability and the ability to break it decompose it into discrimination and calibration components.

"Bayesian inference on the other hand is a specific type of statistical model where the aim is to model distributions, not just the output variable (which is what ML is traditionally focused on)." - What distribution are you referring to? One of the advantages of the Bayesian approach (in the context of models and not of probability, it is not a model, it is a way of estimating the values of the parameters of a model) is that it provides a proper statistical distribution—and not a distribution based on theoretical formulas that require certain assumption to be true to have certain properties—of parameters and model predictions. You can read more at https://www.fharrell.com/post/stat-ml/ (Frank Harrell is a top statistician who was once a frequentist and now is a bayesian. He writes also on the differences between ML and SM and how to choose between the two)

Some relevant links: https://www.nature.com/articles/d41586-019-00874-8 https://www.fharrell.com/post/pval-litany/ https://www.scientificamerican.com/article/the-significant-problem-of-…

Assuming of course that p-values are not problematic [1]: "The reign of the p-value is over: what alternative analyses could we employ to fill the power vacuum?" https://royalsocietypublishing.org/doi/10.1098/rsbl.2019.0174 [2]: "A Litany of Problems With p-values" https://www.fharrell.com/post/pval-litany/

Lately, I've been enjoying a lot of posts in this blog right here: https://eugeneyan.com/writing My company just hired some new VPs from Amazon, and the writings in that blog convey a bit of the Amazon WOW. It's pretty insightful stuff. Also, many posts from Julia Evans :D More concretely, here are a few blog posts I go back to occasionally: https://doordash.engineering/2019/02/20/experiment-rigor-for-switchbac… https://www.unofficialgoogledatascience.com/2019/12/humans-in-loop-for… https://www.unofficialgoogledatascience.com/2016/10/practical-advice-f… https://eugeneyan.com/writing/ml-design-docs/ https://eugeneyan.com/writing/what-i-do-before-a-data-science-project-… https://www.fharrell.com/post/addvalue/ https://www.fharrell.com/post/stat-ml2/

Link to the study: http://www.bmj.com/content/357/bmj.j2353 PDF: http://www.bmj.com/content/bmj/357/bmj.j2353.full.pdf Criticism by Frank Harrell: > A very high-profile paper was published in BMJ on 2017-06-06: Moderate alcohol consumption as risk factor for adverse brain outcomes and cognitive decline: longitudinal cohort study by Anya Topiwala et al. > The authors had a golden opportunity to estimate the dose-response relationship between amount of alcohol consumed and quantitative brain changes. Instead the authors squandered the data by doing analyzes that either assumed that responses are linear in alcohol consumption or worse, by splitting consumption into 6 heterogeneous intervals when in fact consumption was shown in their Figure 3 to have a nice continuous distribution. > How much more informative (and statistically powerful) it would have been to fit a quadratic or a restricted cubic spline function to consumption to estimate the continuous dose-response curve. Article: http://www.fharrell.com/2017/04/statistical-errors-in-medica...