Calibration accuracy for binary-data prediction

Jose von Roth writes,

I am creating a logistic model on 230 cases (4 categorical explanatory variables; about 25% of the cases are 1s, and 75% are 0s in the dependent variable). And I get accuracy of 65%. As a further validation, I boostraped the 230 cases sample 1000 times (with replacement), and ran the obtained model through those 1000 samples, getting accuracies in the range of 57% to 68%. Is that a approvable validation method? Or is bootstrapping “without” replacement and less cases better? Or is this kind of validation in general wrong? (Problem is that I have no test sample).

I’m a little confused here. How can you get an accuracy of 65% when simply predicting 0 all the time gives an accuracy of 75%! This doesn’t sound like such a great model…

1 thought on “Calibration accuracy for binary-data prediction

  1. why not use ROC score (aka AUC, aka mann-whitney U), on cross validated data, and, if you like, assign a p-value to it by comparing with the ROC scores you get when some of the features (or even the labels) are permuted?

Comments are closed.