Kaiser writes,
Have a question about using regression to analyze randomized block experiments. Lets say there are two blocks (male/female) and two treatments (A/B). And lets say the male/female proportion is determined by what is in the population; and lets say the treatments are unbalanced such that males can mostly treatment A and females get mostly treatment B.
So in this situation, I’d use least square means from regression estimates to compare main effects and interaction effects of the treatment. But implicitly, when looking at treatment effect, the regression would control for gender by basically equally weighting male and female. I’m fine with this for comparing treatment means.
What gets confusing (to me) is when I now want to predict the overall impact of treatment A (respectively B) on the entire population. What I typically do is to get regression predictions (least squares mean) for each gender block for each treatment. Then I apply the population proportion of male/female to predict the overall effect of A (and of B separately).
If I do that, then the predicted treatment effects would be different from the least squares means used in the hypothesis testing (the difference between equal weight versus unequal weights). It certainly causes difficulty in explaining why different sets of estimates are used. Is that a problem? If not, how would you explain this to lay people?
My reply: See the title above. The short answer is that the effects for men and women might be different, so the inference for the total effect will depend on what mix you’re interested in. In this case, the inference for the mix is more of a “prediction” question than a “scientific” question, so it’s no surprise that the hypothesis test results can change, depending on the proportions of the two sexes in the population.
To explain to lay people, you could try connecting to the mathematically equivalent problem of age adjustment for health statistics, where you compute the risk for each age group and then reweight based on some standard population values (so that you don’t, for example, say that Florida is the most dangerous state in America, just because it’s filled with oldsters).
Another simple way of explaining this to lay people:
The model gives you a prediction for each of these individual cases:
male A
male B
female A
female B
so, to get the overall population effect, we simply look at the proportion of the population who will fall into each of these 4 buckets.
This is similar to Andrew's last paragraph, but I find "age adjusting" to be a mysterious concept to some people. They tend to think it's a more sophisticated/magical technique than it is.
It probably only seems more sophicated/magical than it is because of how statistics is often taught. If you only teach people how to use statistical software and then "interpret" the output, that is all they will be able to do!
Learning "this is the estimated treatment effect after adjusting for the effect of age" is not very englightening if you don't even understand how you adjusted for the effect of age!
Anyway, back to Kaiser's question, if you mostly give males treatment A, and females treatment B, how are you going to be able to disentangle the effect of gender from the treatment effect?
grad student: that's why I said "mostly" not exclusively. it's an unbalanced design. not the best, I know but in practice, there are non-statistical reasons why we have to design it like that.