According to Josh Millet, the notorious Rorschach inkplots have been posted on the web, leading to much teeth-gnashing among psychologists, who worry that they can’t use the test anymore now that civilians can get their hands on the images ahead of time.
For example, here’s a hint for Card IV (see below): “The human or animal content seen in the card is almost invariably classified as male rather than female, and the qualities expressed by the subject may indicate attitudes toward men and authority.”
So, if they show you this one on a pre-employment test, better play it safe and say that the big figure looks trustworthy and that you’d never, ever steal paperclips from it.
Oh, and when Card II comes up, maybe you should just play it safe and not mention blood at all.
More general concerns
I’m not particularly worried about the Rorschach test since it’s pretty much a joke–you can read into it whatever you want–but, as Millet points out, similar issues would arise, for example, if someone stole a bunch of SAT questions and posted them. It would compromise the test’s integrity. Millet points out that this problem could be solved if you were to release thousands and thousands of potential SAT questions: nobody could memorize all of these, it would be easier to just learn the material.
I’ve had the plan for many years to do this for introductory statistics classes: to have, say, 200 questions for the final exam, give out the questions to all the students, and explain ahead of time that the actual exam will be a stratified sample from the list. This would encourage students to study the material but not in a way that they could usefully “game the system.” I haven’t done this yet–it’s a lot of work!–but I’m still planning to do so.
My introductory stats professor (many years ago!) actually did something like this. He assigned a lot of homework problems each week, all from the textbook. The midterm and final were each just a sample of 3-4 of those homework problems. It was really effective — at both making sure students didn't cram and steadily worked out problems on their own during the course of the class, and at reducing a lot pointless exam-related anxiety.
The professor of my first graduate course in microeconomics did this with a packet of around 350 questions. The problem was that the questions were mostly one-offs, involving some kind of clever trick rather than the application of a general principle. What you learned solving one question didn't really help with any of the others.
In the end, quite a few students simply memorized the entire list. (Never underestimate the resolve of a first-year PhD student!) With one or two exceptions, those were the only people who did well in the course. I still like the idea though. They key is designing the right questions.
Instead of "teaching to the test", now we will have "studying to the test". The students will organize into teams and create model answers to all 200 questions… and if this list is not updated each year, you can bet that the model answers will be passed from one class to the next. I don't see how this is better than the typical exam.
ETS does this for the GRE writing sections. Their website gives you several hundred potential issue and argument topics.
Dr Gelman,
I've been introduced to your work through my graduate work in statistics in the US. I am in a different world and environment now and was wondering if you have any thoughts on how one would go about market mix modeling ?
The problem essentially reduces to finding the optimal mix of media mix to increase return (of investment). Linear regression does not work and the simplest of non-linear models do not either.
Essentially, the problem is not only about finding the right model for the data but also about finding the optimal mix of media after which one sees no increase in returns — sales plateau.
Just figured I'd sound you on this problem. There seems to be no straight forward way to go after this – by which I mean intuitive and one that translates well to business.
LR.
My variation on this approach is to publish exams from the previous semester (without solutions) as study guides. I write 2 or 3 new questions each semester, and combine those with my 8-year test bank. I have high hopes that some of the slicker Greeks or ROTCs will start building test files in a bid to outwit me (by studying!), but I have yet to see that materialize. Students' memories are at best Markov chains, dependent only on what happened last semester.
I agree with Frank about one-offs; when I find an old question that's just a bit too cute, I toss it out.
Back on topic: that Rorschach Card IV is a close-up silhoutte of Santa on a Harley, comin' right at ya!
You get at a tension between good and cheap test design and politics.
We want to resuse good and high quality items. But they are expensive to produe and validate. So, we want to be able to resuse them. We also need to reuse some items — even if mostly use new ones each year — to do the best job of linking scores from year to year.
On the other hand, we want to show people how valid out test is, so we need to release some items. Transparentcy is a big part of politics.
Of course, in reality, most tests are not really valididated and are never linked. Instead, teachers are all levels — teachers untrained in test design or item writing — create tests for their own use that LOOK valid. (We call that "facial validity.") And then we say that the class didn't get it as well this year, or we didn't teach it as well, when it might simply has been that this year's test was harder.
Ahhh….the issues of relying on tests, as we all do, without doing anything to train the poeple who write most tests in how to do it.
@lr (Marketing Mix Modeling)
Start with Dominique Hanssens, LJ Parsons, RL Schultz Market Response Models 2nd ed 2001.
In particular, see pages 104-105 on SCAN*PRO model. This (and similar models from, say, IRI) is your basic, industrial-strength model that's been run thousands of times. This isn't a Bayesian approach, but a good grounding for the general practice.
For a Bayesean approach, you can then try Rossi, Allenby and McCullough's Bayesean Statistics and Marketing.
http://www.amazon.com/Bayesian-Statistics-Marketi…
The authors used to offer a short course using this book as a text; not sure if they still do.
If you need a US data set to try this out on, there's a 65 gigabyte, 6 year academic database available for a handling charge from IRI. Information here:
http://us.infores.com/Insights/Academics/tabid/17…
Andrew, I've been enjoying your blog a while and was delighted to read your idea for overwhelming students with the questions that will be on the exam. This is a strategy for open assessment that I've been promoting as part of the QCommons project, a fully open assessment bank.
Recently I added support for groups and I invite you to join and run a Statistics group. Once there are enough questions, the fact that the questions to the final were already out on the web won't be an issue.
The QCommons project is at http://qcommons.org/ and I blog about it and related work at http://openeducationresearch.org/
-Turadg