Doug writes:
Probability sampling is a great invention, but rhetoric has overtaken reality here. Both of the probability samples in this study had large amounts of nonresponse, so that the real selection probability–i.e., the probability of being selected by the surveyor and the respondent choosing to participate–is not known. Usually a fairly simple nonresponse model is adequate, but the accuracy of the estimates depends on the validity of the model, as it does for non-probability samples. Nonresponse is a form of self-selection. All of us who work with non-probability samples should spend our efforts trying to improve the modeling and methods for dealing with the problem, instead of pretending it doesn’t exist.
Good stuff. Read the whole thing. Doug was, along with me and several others, an advisor on the recent report on National Election Study weighing.
Yes, this is definitely worth reading, particularly for those who may be unfamiliar with the issues involved.
Weights of 70 (when the average weight would be 1) — capped at 5! Situations like this are just ugly.
Most schools now conduct their student teach evaluations online.
I went to the office of institutional research at my school and attempted to get datasets for pre-post internet surveying.
The idea was to evaluate changes in the non-response. I was assured by the director of research that there was no significant difference between the census style surveys they used to conduct vs the new 'optional' online surveys that the school (and most schools) now use.
After two weeks, he told me that the data magically no longer existed…
Today, most schools don't take into account the self-selection that occurs with online teacher evaluation surveys — that's a darn shame….