Normal Deviate

Bayesian methods have a lot going for them, but you need some way of checking whether the methods are giving sensible answers. Andrew Gelman (well know professor, blogger, and author of Bayesian Data Analysis) relies on simulation, which is a pretty Frequentist thing to do! Bayesian and Frequentist methods are not nearly as at odds as posts like this suggest. Frequentism is mostly about how methods should be evaluated. Bayesianism is mostly about how to incorporate different sources of information. You can assess the Frequentist properties of Bayesian methods! Larry Wasserman wrote a great post about this topic here: https://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequen…

I'm glad you're being open minded about responses. While I completely agree with you on the massive backlog of problems that could be solved with very simple data analysis (this is the reason I am in the field), your thoughts about data modeling techniques are very much off the mark and sound like something read from a dated statistics textbook. First, on logistic regression: -I can tell you with certainty that logistic regression is not vastly underused, and is consistently among the most popular machine learning algorithms [1]. From personal experience I think it's likely the most powerful model there is. -People are excited about Bayesian models and NN because they work. Actually, most logistic regression in practice is Bayesian by nature (regularization much?). More so, logistic regression is actually a piece of what powers NNs, and it's incorrect to directly compare the two. To put it more blunt, these are drastic improvements to logistic regression that allow the technique to solve greater problems. -You are trivializing the amount of training needed to properly apply a classifier by calling it simple. Second, there are plenty of basic questions in the data field to answer [2]: -Doing anything with large amounts of data is still complicated and hard -Unifying multiple data sets -General model comparison -Most modeling assumptions and as a result conclusions reached from them are wrong -Communicating results to people. This is the most underestimated. One of…

Fanboy. I have read half of E.T. Jaynes' Probability Theory: the Logic of Science so far. I have read the link you speak of. I have already responded here: http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequent… I don't think my LessWrong link confuses Bayes' Rule with Bayesian statistics. Why do you think Eliezer responded 1/2 to the brain teaser? He does not say it, but I'm pretty sure he just assumed that a mathematician who has 2 boys is twice as likely to spontaneously say "I have at least one boy", compared to a mathematician who has only one boy. The disagreement between his inference (which he did not know was "Bayesian" at the time) and "Orthodox" statistics didn't come from Bayes' Rule. It came from the use of a non-uniform prior to begin with. Which Frequentism rejects, because it's "subjective". So, instead of using this highly relevant prior information, it just uses a uniform prior. (By the way, this is nuts. A scientist should never throw away relevant information.) Both methods then use Bayes' Rule. They just start with different priors: best guess and objective-looking, respectively. (I'm still wondering why anyone would use antything but one's best guess.)

It is important to note that this is a study of people who happened to be stronger, not people who were randomly caused to be stronger (ie by randomly assigning workout routines). Larry Wassermann has a great write up on the distinction with regards to inferring causation vs association here: http://normaldeviate.wordpress.com/2012/06/18/48/ It is good that they attempted to control for things like cardiovascular fitness but on there are also confounding factors like genetics and selection bias that are harder to look at (ie maybe the genetic variants that make it easier to build fast twitch muscles are what has the protective effect and increased strength training won't help someone without those genes). That being said I highly endorse bouldering and rock climbing in general as a way to build strength and move towards a healthy life style for people who don't enjoy typical gym workouts.

If you want to know: do Higgs Bosons exist, you can go frequentist or bayesian: Wasserman: p-value is fine for Higgs experiment: https://normaldeviate.wordpress.com/2012/07/11/the-higgs-boson-and-the… Tom Campbell-Ricketts: Bayes is better even for Higgs experiment https://maximum-entropy-blog.blogspot.com/2012/07/higgs-boson-at-5-sig… My take: frequentist p-value is simpler but Bayes is what you really want.

See these blog posts: http://normaldeviate.wordpress.com/2013/09/01/is-bayesian-inference-a-… http://normaldeviate.wordpress.com/2013/05/05/aaronson-colt-bayesians-… The first is discussing whether people treat Bayesian stats as a religion. The second includes a quote from someone in the frequentist religion (you can tell because they make a jibe about the "Bayesian religion"). So there are people on both sides, thankfully dwindling in number, who really do get their pants in a twist about the formalism one chooses for doing stats. Note that the author of the OP is not one of them, as he lays out in the first sentence of his post.

Just out of curiosity, what kinds of statistical problems do you tend to solve, or are you just a fanboy? I saw you posted a link to something on lesswrong, which is confusing an application of Bayes Rule with Bayesian statistics. Maybe you should read what an actual statistician has to say on the matter: http://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequent… instead of a Harry Potter fanfic author.

Pet peeve: Conditioning on a non-random variable doesn't make sense (i.e. P(data | H_0))[1]. [1] http://normaldeviate.wordpress.com/2013/03/14/double-misunderstandings…

Sort of. You don't need identical replications of the same experiment, just long run probabilities for any application of the method. See example two here: https://normaldeviate.wordpress.com/2012/11/17/what-is-bayesianfrequen… (The author is a stats professor at CMU.) Quoting: "The plot shows the first 50 simulations. In the first simulation I picked some distribution {F_1}. Let {\theta_1} be the median of {F_1}. I generated {n=100} observations from {F_1} and then constructed the interval. The confidence interval is the first vertical line. The true value is the dot. For the second simulation, I chose a different distribution {F_2}. Then I generated the data and constructed the interval. I did this many times, each time using a different distribution with a different true median. The blue interval shows the one time that the confidence interval did not trap the median. I did this 10,000 times (only 50 are shown). The interval covered the true value 94.33 % of the time. I wanted to show this plot because, when some texts show confidence interval simulations like this they use the same distribution for each trial. This is unnecessary and it gives the false impression that you need to repeat the same experiment in order to discuss coverage."

Simpson's Paradox is scary if you've not seen it before, but you should not stop here and instead proceed immediately on to omitted variable bias and the conversation here (http://normaldeviate.wordpress.com/2013/06/20/simpsons-paradox-explain… In short, Simpson's paradox occurs because probability distributions and causal claims are distinct things which behave differently. It's nothing more than a particular, insidious example of correlation not implying causation. It's perfectly possible for a probability distribution to have a "contradictory" shape, but perfectly impossible for logical statements about the world to be contradictory. The resolution is that you shouldn't let your probability distributions turn into logical statements without analyzing your causal assumptions. This will lead you to whether or not excluding a variable is omitted variable bias (and whether including an improper one will lead to included variable bias, which is rarely recognized).