In an introductory course, when does learning occur?

Now that September has arrived, it’s time for us to think teaching. Here’s something from Andrew Heckler and Eleanor Sayre. Heckler writes:

The article describes a project studying the performance of university level students taking an intro physics course. Every week for ten weeks we took 1/10th of the students (randomly selected only once) and gave them the same set of questions relevant to the course. This allowed us to plot the evolution of average performance in the class during the quarter. We can then determine when learning occurs: For example, do they learn the material in a relevant lecture or lab or homework? Since we had about 350 students taking the course, we could get some reasonable stats.

In particular, you might be interested in Figure 10 (page 774) which shows student performance day-by-day on a particular question. The performance does not change directly after lecture, but rather only when the homework was due. [emphasis added] We could not find any other studies that have taken data like this, and it has nice potential to measure average effects of instruction.

Note also Figure 9 which show a dramatic *decrease* in student performance–almost certainly due to interference from learning a related topic.

I love this kind of thing. The results are not a huge surprise, but what’s important to me about this kind of study is the active measurement that’s involved, which can be difficult to set up but, once it’s there, allows the opportunity to discover things about teaching and learning that I think would be nearly impossible to find out through our usual informal processes of evaluation. Some time I’m hoping to do this sort of project with our new introductory statistics course. (Not this semester, though; right now we’re still busy trying to get it all working.)

9 thoughts on “In an introductory course, when does learning occur?

  1. A few thoughts, without having read the paper.

    Presumably there are different types of learners. It would be nice to tailor instruction to the mix of learners.

    Perhaps the authors could tease out different types of learners by breaking out results by declared major at the time of the class, and eventual major upon graduation. Assuming the have names on the evaluations they gave, that should be possible with a little help from the registrar.

  2. Although I don't find the answers that surprising, there are a couple of points I'd like to make…

    1)
    It would be interesting to adjust for quality of the students. Was it only the good students (amongst those randomly selected) who took the test around the homework period? The poorer students may have been too exhausted/ discouraged about the subject around homework time to take the test or just ran out of time, because of the homework, to do both.

    2)
    Lectures provide information. Homework poses questions about the information. The test poses questions about the information.

    Maybe the improvement at homework time was because there was more training in answering questions and perhaps training in recognising which question came from which topic.

    If the test was to verbally recount the information then perhaps the students would have been better at doing so at the time of the lecture rather than at homework time i.e. how the test is styled may not accurately reflect mastery of the information.

  3. FH: Jason's right — "learning style" correlates poorly (if at all) with actual learning, even given instruction tailored to professed favorite style.

    Megan: On point 1: We have tested that. The short answer is no. There is no effect of test day, either of day of week (i.e. for homework due dates) or day of quarter (i.e. for the psych 100 effect) on whether students are "good" students or "bad" students, as measured by their final grade. Similarly, there's no effect of the hour of the day — students who come just after lecture are the same as students who come in the late afternoon. Students were tested in our research lab, and all students had one hour to complete all tests.

    On a slightly different, but more exciting question: Do high- and low-performing students learn differently? And if so, how? These results are suggestive but not conclusive. If you test 400 students in 10 bins, and then divide those bins into high- and low-performing students, your statistics get pretty small. However, it appears that high-performing students in the course tend to learn more material on some topics. On other topics, the low-performing students actually degrade while the high-performing students learn, for a net zero effect. This is as expected, but here's the kicker: afterwards, most students tend to relax towards their initial state. So the high-performing students forget more than the low-performing students. Andrew Heckler has moved on to other projects, but I am now testing this effect (and others — stay tuned!) at Rochester Institute of Technology with Scott Franklin.

    On point 2: We conducted interviews with most of the students. If the "verbal reasoning" hypothesis is correct, then they would have done better in interviews that they would on paper-based tests. We do not have any evidence to support that hypothesis. Also, I note that college students are pretty good at answering questions already, so if we were to see an effect, it would have to be quite small.

  4. Eleanor: I know it’s just a quick blog post, but does seem as if lack of evidence is being confused with evidence of lack of effect – "no effect of test day"

    For the benefit of students, supplying rough confidence or credible intervals would prevent anyone from accusing you of doing that – they show what effect sizes can safely be ruled out.

    But more importantly I don't believe conclusions should be based on individual as of yet un-replicated studies. If there has been (or will be another half dozen or so studies on essentially the same question) what did they all say jointly – were they contradictory?

    K?
    p.s. it took the clinical research community from about 1980 to 2000+ to engrain this in most of their researchers

  5. K: Forgive me if I was too casual in a blog post. There's no correlation between test day, weekday, or hour and students' final grade in the course (the best operational definition of "student goodness"), a statement which is true over five courses totaling about 2k students. For a more detailed analysis, please read the papers or contact me directly.

    As for replication studies, the OP notes correctly that these kind of measurements are costly to set up and thus not common. Some of the experiments have been replicated in five courses at two institutions. Others less. As I mentioned before, this is an ongoing research program. If you have classes suitable for replication studies, we are currently soliciting beta testers for a less-costly implementation of the basic methodology. Please email me for details.

  6. Eleanor: I believe its just fine to be casual in a plot post.

    I should have tried harder to make sure my comments were taken to be about what was in and not in the post rather than your research.

    I would still like to see some indication of the size of the effects supported by any given study – i.e. an interval because as Andrew often points out – its never exactly zero.

    And this is rhetorical comment rather than I request for you to provide further information – I almost think thats unfair to blog posters.

    But as for costs, I am guessing the expected cost of implementing a false conclusion (these happen to all of us at at least a minmum rate) would far outweigh the expected cost of replication studies (it did in clinical research when I studied that in the 1980,s).

    And its great to hear that is ongoing research and you are facilitating others in replicating it.

    K?

Comments are closed.