Jim Madden writes:

I have been developing interactive graphical software for visualizing hierarchical linear models of large sets of student performance data that I have been allowed access to by the Louisiana Department of Education. This is essentially pre/post test data, with student demographic information (birthday, sex, race, ses, disability status) and school associated with each record. (Actually, I can construct student trajectories for several years, but I have not tried using this capability yet.) My goal is to make the modeling more transparent to audiences that are not trained in statistics, and in particular, I am trying to design the graphics so that nuances and uncertainties are apparent to naive viewers.

The general strategy I employ is to make the illustrations as literal as possible. Of course, in a typical data set (N ~ 30,000), one cannot fit everything into a single image. Therefore, I allow viewers to navigate the data locally at each level. Districts appear in the base graphic. District performance is indicated in a scatter plot; district icons give indicators of district properties (size, composition, and aggregate statistical properties). By clicking on districts, viewers can expand them into within-district data on schools; schools in turn can be expanded to show the actual student-score scatter-plots. Relevant model parameters can be viewed as regression lines associated with the level being viewed, and significance is indicated by background coloration. Mouseover allows the identification of the images; specific schools or districts can be selected for viewing from an index.

I confess to not having explored other software for hierarchical modeling, so I don’t know if such graphical capabilities are already available. If so, then perhaps you would be good enough to apprise me.

The next phase of the research will test the interpretations of audiences to be sure that the graphics is not producing misleading images and that naive viewers are not leaping to unwarranted conclusions.

The New York Times recently ran an editorial praising Louisiana for initiative to improve education.

The editorial singles out an “evaluation system that judges teacher-preparation programs based on how much their graduates improve student performances in important areas, including reading, math and science.” There are no references or names given, so it’s unlikely that readers will easily find the recent report that the editorial mentions. In fact, it was prepared by my colleague and friend, George Noell. You can download his reports at the public web site.

He claims to be able to detect influence of teacher-training on student performance that are on the order of 0.1 standard deviations. (On average, the scores of students of experienced teachers exceed the scores of peers with inexperienced teachers by as much as 10% of a standard deviation.) The superior teacher preparation programs have an effect that is somewhat smaller.

Of course I support studying school performance with the best statistical tools available. I have reservations, however, about the relevance of George’s work. Performance goals that the state has set entail improving some large target populations by half a standard deviation or more. The results that George presents do not tell us if the programs that he has found to be performing well are having any effect on these populations. To achieve the goals we desire, I am not sure that we need to be paying attention to the effects that George is detecting. (To make my meaning clearer by analogy: We might find that wine drinkers live on average 0.5 years longer. But this is not relevant to the problem of youth killed by gang violence.) If you look at George’s work, I’d be interested in your reactions.

I’m actually a real-algebraic geometer who has strayed into educational statistics because of my involvement in education initiatives more broadly. But my interest in statistics has been growing. I’m writing to you because I am interested in getting feedback/reaction . . .

My general goal is to make educational data sets more transparent and more informative. I want to provide a way of looking at them that is not like “peering through a drinking straw”. I want viewers to be able to see the “lay of the land” and understand how the global an local statistical properties of a vast data set can inform strategic policies that have meaningful social aims.

This all seems interesting and important to me. When writing ARM, we developed some ideas about displaying multilevel inferences (see chapters 12-14, for starters), but I think much more can and should be done. And the stuff we’ve done so far could be automated; that would be a good start.