Circular analysis in systems neuroscience - with particular attention to cross-subject correlation mapping
Nikolaus Kriegeskorte
Laboratory of Brain and Cognition, Section on Functional Imaging Methods
National Institute of Mental Health

A neuroscientific experiment typically generates a large amount of data, of which only a small fraction is analyzed in detail and presented in a publication. However, selection among noisy measurements can render circular an otherwise appropriate analysis and invalidate results. In the first half of the talk, I will argue that systems neuroscience needs to adjust some widespread practices in order to avoid the circularity that can arise from selection. In particular, "double dipping" - the use of the same data set for selection and selective analysis - will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis. To demonstrate the problem, we applied widely used analyses to noise data known not to contain the experimental effects in question. Spurious effects can appear in the context of both univariate activation analysis and multivariate pattern-information analysis. I will suggest a policy for avoiding circularity. In the second half of the talk, I will specifically address the issue of selection bias in cross-subject correlation-mapping studies of the type widespread in social neuroscience. Correlation estimates are noisy in this specific scenario unless the number of subjects is large. This leads to large error margins on correlation estimates (which are typically not reported), large biases when a selective correlation analysis is performed on the same data as used to define a region by correlation mapping, and substantial false-positives rates when correction for multiple testing is inadequate. As a solution to these problems, I suggest rigorous multiple-testing correction, a leave-one-subject-out approach for independent ROI-effect estimation, and reporting of error margins on correlation estimates.


Back to workshop main page.