Neural Codes for Read-out of Object Information from Neuronal Population Activity in the Macaque Inferior Temporal Cortex

Chou Hung, Gabriel Kreiman, Tomaso Poggio & James DiCarlo

The inferior temporal (IT) cortex plays a crucial role in visual object recognition in primates [1].  In the effort to understand the complex brain computations leading to object recognition, it is key to characterize what information is represented in inferotemporal cortex and what its neural code is.  In recent years, evidence has accumulated about the responses of single neurons in IT cortex of monkeys [1] and, more recently with fMRI techniques, about much coarser activity in homologue areas in humans.

We quantitatively studied the properties of the neural code in IT and its robustness to different stimulus transformation as well different sources of noise.  We use a statistical classifier (SVM) to read out the information about object identity or class from a set of 77 possible stimuli from multi-unit spiking activity, single unit spiking activity and local field potentials.  By using different inputs to the classifier we could directly compare different coding schemes in a quantitative manner. By training the classifier in a set of stimuli and testing its performance on a different set, we could test the generalization power of the representation to different transformations and changes.

We observed that the activity of small populations of IT neurons contains detailed information about object identity and type.  We characterized the selectivity and invariance properties of the representation and analyze its underlying neural code, showing (i) that short intervals, containing few spikes, from 12.5 to 50 ms in duration contain large amounts of visual information, (ii) that the representation is robust to changes in scale and position of the images, (iii) that we can read out stimulus scale and position, (iv) that 'identification' and 'classification' may not be separate processes in terms of generalization power but appears to be part of a continuum with the only difference being the 'similarity' of the stimuli, (v) that the representation is robust to significant neuron drop-out.  Our analysis of population patterns of activity in IT directly supports the notion that such codes can support rapid visual recognition.  Thus, we showed that population-based measures of recognition performance offer new insights into the spatiotemporal codes supporting visual recognition.

Acknowledgments

This report describes research done at the Center for Biological & Computational Learning, which is in the McGovern Institute for Brain Research at MIT, as well as in the Dept. of Brain & Cognitive Sciences, and which is affiliated with the Computer Sciences & Artificial Intelligence Laboratory (CSAIL).

This research was sponsored by grants from: Office of Naval Research (DARPA) Contract No. MDA972-04-1-0037, Office of Naval Research (DARPA) Contract No. N00014-02-1-0915, National Science Foundation (ITR/SYS) Contract No. IIS-0112991, National Science Foundation (ITR) Contract No. IIS-0209289, National Science Foundation-NIH (CRCNS) Contract No. EIA-0218693, National Science Foundation-NIH (CRCNS) Contract No. EIA-0218506, and National Institutes of Health (Conte) Contract No. 1 P20 MH66239-01A1.

Additional support was provided by: Central Research Institute of Electric Power Industry (CRIEPI), Daimler-Chrysler AG, Compaq/Digital Equipment Corporation, Eastman Kodak Company, Honda R&D Co., Ltd., Industrial Technology Research Institute (ITRI), Komatsu Ltd., Eugene McDermott Foundation, Merrill-Lynch, NEC Fund, Oxygen, Siemens Corporate Research, Inc., Sony, Sumitomo Metal Industries, and Toyota Motor Corporation.

References

[1] Logothetis, N. K. and D. L. Sheinberg (1996). "Visual Object Recognition." Annual Review of Neuro-science 19: 577-621.