Convergence and divergence in the neural organization of object responses to pictures and words

Konkle, Wang, Peelen, Caramazza, Bi

Visual information is transformed from early sensory formats into increasingly abstract representations of its content. We probed this abstraction by exploring the convergence in the neural responses to pictures of objects and their spoken names, taking a broad look at several major semantic divisions between object categories. Our aim was to explore which neural regions show reliable responses that are specifi c to visual pictures, specifi c to auditory words, or common between these two modalities, using a data-driven clustering approach. Using fMRI, we measured neural response patterns to objects from 18 broad semantic categories, presented as both pictures and auditory words, in 16 participants. We used a clustering technique to group together voxels by their response profi le similarity over these 18 categories, in a way that is agnostic to where the voxels are located and whether they refl ect responses from the visual or auditory modality. They key advantage of this procedure is that it simultaneously discovers common and unique structure across both modalities without presupposing any regional boundaries in advance. This analysis identifi ed several regions with similar neural profi les to pictures and words (parahippocampal, transverse occipital sulcus, retrosplenial cortex), which primarily had a response preference for inanimate categories of objects. In contrast, other neural regions were only reliably modulated by pictorial stimuli (lateral occipital, fusiform), and these regions primarily had a response preference to pictures of animate entities. Taken together, these results demonstrate a surprising and currently unexplained link between the neural organization of broad object domains and activations by different modalities.




Konkle, T., Wang, X., Peelen, M., Caramazza, A., Bi, Y.(2015). Convergence and divergence in the neural organization of object responses to pictures and words. Talk presented at the annual meeting of the Vision Sciences Society, May 15-20, St. Pete Beach, FL.