Suresh Krishna writes:

I am facing this multi-level modeling issue where the effects are small but visible in plots, and the question is about a good way to extract the information.

I have a hierarchical dataset of this form:

1 subject

Tested under 2 conditions: A and B

10 sessions in each condition

In each session, 2 kinds of tests: Test 1 and Test 2

200 independent repetitions of each test-type, with 200 Yes/No answers

So I think this is a 2 x 2 x 10 x 2 setup

What I want to know is whether the difference in percentage of yes answers between Test1 and Test2 is different for the 2 conditions A and B. i.e. Is there an “interaction” between the Effect of Condition and the Effect of Test.

I looked through Agresti and Pinheiro/Bates and couldn’t find an example covering this situation. I do not have your book handy here . . .

I would be really grateful if you could suggest a framework or procedure to address this question.

I considered:

Pool data from all the sessions for a given condition and test together, thus getting 2000 repetitions of Test1 and Test 2 in each condition. Now I have a 2x2x2 setup, which maps on to something in Agresti, but then I am ignoring within-session correlation information.

I could simply get a difference between Test1 and Test2 percentages for each session, and then compare the distribution of these differences in conditions A and B (with something like a t-test), but then I only have 10 points (one for each session) and so I guess I am throwing away a lot of information.

My reply: The simplest reasonable thing to do here is to analyze the condition A and condition B data separately. For each, you have 10 data points comparing Test 1 to Test 2: take the average differences with s.e. equal to the sd of the 10 differences divided by sqrt(10). You now have two estimates (one for A, one for B), each with a s.e. Take the difference, and combine the s.e.’s in the usual way. You could also get the equivalent result by fitting a multilevel model.

My article on Anova might be helpful for understanding this sort of thing. I discuss there why it’s not really throwing away data to consider these as experiments with 10 data points. Basically, when you average the data to get the 10 data points, your sd is going down because of the averaging.