“Voodoo” social cognitive neuroscience correlations

This article (careful pdf) is poised to shake up the world of social neuroscience. It focuses on the reported correlations between activation in given voxels while the subject is under certain testing conditions and something else, usually either personality data or the activation of the same voxels under different conditions. And it argues persuasively that the statistical methods used to make these correlations are often misleading.

Previous attempts to determine typical voxel correlation levels from fMRI studies have indicated that test-retest correlations (a measure of reliablity) will not often be above 0.7. The test-retest correlation for the Big Five factors of personality are around 0.7-0.8, for which the authors select an upper bound of 0.8.

So if the relabilities of personality and voxel data are about 0.8 and 0.7, then under perfect conditions the highest correlation that could be found between the two would be the square root of 0.8 times 0.7, or 0.74. The authors then conducted a meta analysis of the available studies and noted in a histogram that more 50 statistically significant results with correlations greater than 0.75 were found in the literature.

The authors smelled something fishy and they looked into the methods of how some of these studies were conducted, which is an interesting discussion. Many of the authors appear to have been quite forthright about their methods, suggesting that they may not be entirely culpable for the overzealous correlations.

Reference

Vul E, Harris C, Winkeilman P, Pashler H. 2008 Voodoo correlations in social neuroscience. Perspectives on Psychological Science, In press. Available as pdf here.