Don Coffin writes:
A colleague of mine and I are doing a presentation for new faculty on a number of topics related to teaching. Our charge is to identify interesting issues and to find research-based information for them about how to approach things. So, what I wondered is, do you know of any published research dealing with the sort of issues about structuring a course and final exam in the ways you talk about in this blog post? Some poking around in the usual places hasn’t turned anything up yet.
I don’t really know the psychometrics literature but I imagine that some good stuff has been written on principles of test design. There are probably some good papers from back in the 1920s. Can anyone supply some references?
In the 1951 edition of Edcational Measurement, Lindquist had a chapter that talks about virtually all ofthe debates we have in education about standardized testing.
Is that what you mean?
Obviously, teat design goes back pretty far, you are right. But most psychometrics today is far more focused on theath than the design. Content and the relationship btween constructs and measures is not of much concern to psychometricians today.
And so, those old issues are still out there.
A full citation.
Lindquist, E. F. (1951). Preliminary Considerations in Objective Test Construction. In Lindquist, E. F. (ed), Educational Measurement, pp. 119-158. Washington, DC: American Council on Education.
The final paragraph of the chapter:
The author's primary purpose in this chapter, therefore, has not been so much to suggest answers to the questions raised as simply to raise the questions, or to indicate the general direction of the thinking which should precede the selection of any specific test construction project. If the answer that have .been suggested may at times have seemed dogmatic in character, this may itself contribute to the central purpose of stimulating more critical consideration of the questions raised.
More practically speaking, there are interesting things in the following article about the relationship between content, teaching and final exams, even with an experimental design!
Carrell, S. & West, J. (2010). Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors.
That's the two that come to mind, one from back in the day and the other more tied to Dan Coffin's question.
Bob Mislevy, Linda Steinberg and I worked out a series of principle for test construction that scale well from classroom assessment to large-scale and high-stakes assessments. There are basically three steps.
1) Decide on what are the knowledge, skills and abilities you want to know whether or not the students have.
2) Figure out what you could observe about the student's performance that would provide evidence about whether or not the student's has the aspects of proficiency identified in Step 1.
3) Figure out how to structure tasks such that the students provide that evidence.
We have written quite a lot about this subject. A good starting point is this ETS Research Report:
http://www.ets.org/research/policy_research_repor…
That paper is also available as a research report from CRESST at UCLA. A number of other reports are available at both the CRESST and ETS sites, search for author "Mislevy" for the greatest number of hits.
I dont't think if i got that right, but i think, for medical education, the answer would be to just set up learing goals/objectives acording to a blueprint (e.g. The revised Bloom Taxanomy). And in a second step to construct exams according to that blueprint. Of course, the objectives should be known to the students.
Furthermore – "assessment drives learing" – one could consider to not just map one exam to one lecture but to build an assessment tool for entire the curriculum. "progress testing" is one way to do that.
I like Bob Mislevy, he's always got something interesting to say.
The hardest part of test construction is construct validity – making sure you're measuring what you really want to measure. That's steps 2&3 in Russell's list.
The book How People Learn has some very interesting things to say, and has a chapter or two on assessment.
The research you want is in the field of Learning Sciences not psychometrics.