Peer pressure, selection, and educational reform

Partly in response to my blog on the Harlem Children’s Zone study, Mark Palko wrote this:

Talk of education reform always makes me [Palko] deeply nervous. Part of the anxiety comes having spent a number of years behind the podium and having seen the disparity between the claims and the reality of previous reforms. The rest comes from being a statistician and knowing what things like convergence can do to data.

Convergent behavior violates the assumption of independent observations used in most simple analyses, but educational studies commonly, perhaps even routinely ignore the complex ways that social norming can cause the nesting of student performance data.

In other words, educational research is often based of the idea that teenagers do not respond to peer pressure. . . .

and this:

When you isolate a group of students, they will quickly arrive at a consensus of what constitutes normal behavior. It is a complex and somewhat unpredictable process driven by personalities and random connections and any number of outside factors. You can however, exercise a great deal of control over the outcome by restricting the make-up of the group. . . .

Dobbie and Fryer address the question of self-selection, “[R]esults from any lottery sample may lack external validity. The counterfactual we identify is for students who are already interested in charter schools. The effect of being offered admission to HCZ for these students may be different than for other types of students.” In other words, they can’t conclude from the data how well students would do at the Promise Academies if, for instance, their parents weren’t engaged and supportive (a group effective eliminated by the application process).

But there’s another question, one with tremendous policy implications, that the paper does not address: how well would the students who were accepted to HCZ have done if they were given the same amount of instruction * as they would have received from HCZ using public school teachers while being isolated from the general population? (There was a control group of lottery losers but there is no evidence that they were kept together as a group.)

Why is this question so important? Because we are thinking about spending an enormous amount of time, effort and money on a major overhaul of the education system when we don’t have the data to tell us if what we’ll spend will wasted or, worse yet, if we are to some extent playing a zero sum game.

Social norming can work both ways. If you remove all of the students whose parents are willing and able to go through the application process, the norms of acceptable behavior for those left behind will move in an ugly direction and the kids who started out with the greatest disadvantages would be left to bear the burden.

But we can answer these questions and make decisions based on solid, statistically sound data. Educational reform is not like climate change where observational data is our only reasonable option. Randomized trials are an option in most cases; they are not that difficult or expensive.

I don’t really have anything to add–I’ve pretty much said whatever I have to say on the topic–but I thought this was an interesting perspective worth sharing.

2 thoughts on “Peer pressure, selection, and educational reform

  1. Parental/home influence and peer effects.

    Parental/home influence and peer effects.

    Parental/home influence and peer effects.

    Parental/home influence and peer effects.

    Ignore those and you are not actually studying education or the real processes and mechanisms that lead to desirable educational outcomes.

Comments are closed.