Mike Shores writes,
I am a junior at Roslyn High School (Roslyn, NY). I am currently enrolled in my schools Independent Social Science Research Program, where I am studying the impact of teachers’ political biases on essay grading. However, I have run into some statistical problems and am wondering if you would be able to help me.
In my experiment there are two independent variables. Teachers were assigned to read one of two essays (liberal or conservative) and to report their personal political orientation (liberal or conservative) on a 6-point scale. I was interested in how these two factors affected the grade teachers assigned the essay.
I initially ran a two-way ANOVA and to do so I defined participants as liberal if rated themselves from 1 to 3 or conservative if they rated themselves as a 4 to 6. In doing so, however, I wonder if I’m giving up some statistical power and so I am trying to figure out if there is a better way to analyze the data.
Another possibility I thought of was to analyze the data from the liberal and conservative essays separately and run correlations between their political beliefs and the grade they gave the essay. I was wondering if you had any suggestions about how I should proceed.
My reply:
First of all, I’m impressed that you got the teachers to agree to grade extra papers for you!
My quick answer is that you’d be better off calculating averages and standard errors, and making scatterplots, rather than doing anova. Probably the best thing would be 2 scatterplots, one for the conservative essays and one for the liberal essays, plotting teacher’s self-evalutation vs. grade given to the paper. You can then compute the correlation for each of the scatterplots (and you can check if each of these is statistically significantly different from 0).
Or you can run a regression, predicting the grade of the essay given 3
variables:
x1 = the teachers’ poliical orientation
x2 = a variable that is +1 if the essay is conservative and -1 if liberal
x3 = an interaction of x1 and x2 (i.e., x1*x2)
There are probably other good ways to analyze the data too. But, yeah, you shouldn’t simplify the 1-6 scale to a binary scale. that’s just throwing away information.
If anybody has other thoughts, feel free to pass them along.
Scatterplots often don't work well for discrete data – you might also want to try jittered scatterplots, mosaic plots, fluctuation diagrams etc.
You also should figure out a way to separate the grading activity from the self-reported political orientation. Otherwise, teachers might figure out what you are up to. Perhaps embed your political orientation question in the context of a bunch of other questions so that the teachers think your focus is on something other than political views. Perhaps what they say they focus on in a paper–grammer, organization, neatness, spelling, or …. People do not like to believe that political orientation affects their judgments–especially in assigning grades, an activity in which they want to believe they are objective.
The non-ANOVA statistical analyses mentioned are great; but if the design is such that your hypothesized effect is negated, the statistics won't really matter.
Also, are you working with anyone on ethical treatment of research participants?
I agree with the concern about reporter bias because of them figuriung out what you are up to. Some methodology stuff (asking them to assign their preference after they grade, having them grade one essay only-requires larger sample size, using proxy variables) might help. I'm not a statistician, though, just a blog reader internet surfer. But I bet this is a common problem and someone could help figure out the right methodology to take care of this concern.
yes, the self report bias may be a big problem here, although you did not specify the research design used to gather your essay and ideology data. However, it doesn't sound as if you are trying to publish research, rather just demonstrating mastery of inferential stats. Go with either one of Dr. Gelman's suggestions and be sure to equivocate your results.