Secret weapon with rare events

Gregory Eady writes:

I’m working on a paper examining the effect of superpower alliance on a binary DV (war). I hypothesize that the size of the effect is much higher during the Cold War than it is afterwards. I’m going to run a Chow test to check whether this effect differs significantly between 1960-1989 and 1990-2007 (Scott Long also has a method using predicted probabilities), but I’d also like to show the trend graphically, and thought that your “Secret Weapon” would be useful here. I wonder if there is anything I should be concerned about when doing this with a (rare-events) logistic regression. I was thinking to graph the coefficients in 5-year periods, moving a single year at a time (1960-64, 1961-65, 1962-66, and so on), reporting the coefficient in the graph for the middle year of each 5-year range).

My reply:

I don’t know nuthin bout no Chow test but, sure, I’d think the secret weapon would work. If you’re analyzing 5-year periods, it might be cleaner just to keep the periods disjoint. Set the boundaries of these periods in a reasonable way (if necessary using periods of unequal lengths so that your intervals don’t straddle important potential change points). I suppose in this case you could do 1960-64, 65-69, …, and this would break at 1989/90 so it would be fine. If you’re really running into rare events, though, you might want 10-year periods rather than 5-year.

2 thoughts on “Secret weapon with rare events

  1. You might find useful this paper on an application of the Chow test in a judicial politics context, to see the sort of problems it can have in practice. In short, a Chow test can be severely overconfident. I cannot say, of course, if your data will be similarly vulnerable to the problems we found. Even when there is no actual change between periods, a Chow test can find a change significant at 5% far more than 5% of the time (up to nearly all the time, in our data). We (Kelly Rader and I) argue in that paper and in <a>this follow-up paper that the clustered nature of the data is causing the problem (the Chow test assumes fully independent observations). We argue for the use of a randomization test as either a substitute or a robustness check.

Comments are closed.