I received the following message from a statistician working in industry:
I am studying your paper, A Weakly Informative Default Prior Distribution for Logistic and Other Regression Models. I am not clear why the Bayesian approaches with some priors can usually handle the issue of nonidentifiability or can get stable estimates of parameters in model fit, while the frequentist approaches cannot.
My reply:
1. The term “frequentist approach” is pretty general. “Frequentist” refers to an approach for evaluating inferences, not a method for creating estimates. In particular, any Bayes estimate can be viewed as a frequentist inference if you feel like evaluating its frequency properties. In logistic regression, maximum likelihood has some big problems that are solved with penalized likelihood–equivalently, Bayesian inference. A frequentist can feel free to consider the prior as a penalty function rather than a probability distribution of parameters.
2. The reason our approach works well is that we are adding information. In a logistic regression with separation, there is a lack of information in the likeilhood, and the prior distribution helps out by ruling out unrealistic possibilities.
3. There are settings where our Bayesian method will mess up. For example, if the true logistic regression coefficient is -20, and you have a moderate sample size, our estimate will be much closer to zero (while the maximum likelihood estimate will be minus infinity, which for some purposes might be an acceptable estimate).
Probably I should write more about this sometime. Various questions along those lines arose during my recent talk at Cambridge.
I would think plotting the log likelihood and log prior and noticing how they add to the log posterior (for a parameter of interest) would make what "we are adding information" on – very obvious.
But I have no idea how often people do this privately or in publications.
There does seem to be a reluctance or at least a lack of effort/interest in actually doing this – sometimes the addition would be obvious to many but not always.
Why is it not important to display the inference mechanics as clearly as possible?
I guess I should take a sample of published papers and count…
K?
My comment concerns the use of words "frequentist" and "frequency properties". I think that "frequentist" refers to any approach using only frequency interpretation of probability. Analysing the frequency properties of Bayesian estimates does not require you to deny Bayesian probabilities. Bayesian inference can be used to analyse frequency properties, too. Bayesians can use some useful tools first proposed in frequentist setting for frequency analysis, and even improve them, modeling also the epistemic uncertainty.