Research Abstracts - 2006

Elucidating the Role of Top-Down Processing in the Brain Using Neural Readout Techniques and Computational Models

Ethan Meyers, Gabriel Kreiman & Lior Wolf

The Problem

In primate visual cortex, feedback connections from higher visual areas to lower visual areas outnumber feed-forward connections in a ratio of 10 to 1 [1]. Yet despite the abundance of these feedback connections, the functional role that these connections play in vision remains largely unknown. Several theorists have assigned speculative roles to feedback connections[2, 3], however, such proposals have usually been philosophical in nature, and have lacked any substantial evidence in terms of either neural data or in terms of working computational models. In contrast, many of the properties of the neurons involved in the feed-forward pathway are known [4, 5]. Also, recent computational models, such as the Standard Model proposed by Riesenhuber and Poggio [6], can both match the neurophysiological properties of neurons found in the feed-forward visual pathway, and visual features derived from such models have been shown to achieve high levels of performance in computer vision tasks [7].

In the project, we propose to try to gain a better understanding the role of these feedback connections in visual processing by taking a dual approach of both developing new methods of analyzing neural data and by exploring possible computational models in which feedback connections aid computations useful for vision.

Approach

To try to find patterns in neural data that can elucidate the role of these feedback connections, we propose to further develop the readout methods used in Hung et al.[8]. In particular, we intend to use machine learning classifiers, such as support vector machines, to categorize which stimuli are being represented in different cortical areas of Rhesus Macaque monkeys, while the monkeys engages in a delayed match to sample experiment similar to those used by Freedman et al. [9]. By decoding patterns in neural firing in the inferiotemporal (IT) and prefrontal (PFC) cortex for object and category specific signals during the fixation, sample, delay, and test periods, we intend to determine which time periods contain object and/or category specific information, which will hopefully give us insight into when feedback processing might be occurring. We are also interested in exploring new methods that look for synchronized activity between PFC and IT which might further enhance these readout techniques.

In terms of computational modeling, we intend to explore how feedback information coming from higher areas can help improve the performance of computer vision systems in specific tasks. In particular, we would like to add feedback processing to Risenhuber and Poggio’s Standard Model and see if and how such processing can help achieve higher classification rates on object recognition tasks. We are also interested in exploring other tasks in which feedback processing might be useful, such as finding the specific location of objects in space, and searching for harder to identify objects in highly cluttered scenes.

Preliminary Results

Preliminary results using neural readout on data collected in a delayed match to sample paradigm published by Freedman et al. [9] shows that there is a strong stimulus specific signal in IT during the period that the stimulus is present, and a weaker stimulus specific signal during the delay period (see figure 1 below). It is harder to tell whether signals coming from PFC are stimulus specific based on our current analysis, but future analyses should give us a more definite answer. Looking at category specific signals and synchronous firing between PFC and IT should also help us to better interpret what these areas are representing and how they are interacting.

Preliminary results from computational models have shown that adding feedback to hierarchical classification systems does improve their performance. In recent experiments, we compared five different hierarchical classification schemes in four object classification tasks [10]. Figure 2 shows the different hierarchies used and the results from the Caltech 101 object dataset. As can be seen, the type of architecture used can have a large influence on performance, and in particular, the feedback scheme which we called ‘semantic concatenation’ tended to consistently obtain some of the best results.

Acknowledgments

This report describes research done at the Center for Biological & Computational Learning, which is in the McGovern Institute for Brain Research at MIT, as well as in the Dept. of Brain & Cognitive Sciences, and which is affiliated with the Computer Sciences & Artificial Intelligence Laboratory (CSAIL).

This research was sponsored by grants from: Office of Naval Research (DARPA) Contract No. MDA972-04-1-0037, Office of Naval Research (DARPA) Contract No. N00014-02-1-0915, National Science Foundation (ITR/SYS) Contract No. IIS-0112991, National Science Foundation (ITR) Contract No. IIS-0209289, National Science Foundation-NIH (CRCNS) Contract No. EIA-0218693, National Science Foundation-NIH (CRCNS) Contract No. EIA-0218506, and National Institutes of Health (Conte) Contract No. 1 P20 MH66239-01A1.

Additional support was provided by: Central Research Institute of Electric Power Industry (CRIEPI), Daimler-Chrysler AG, Compaq/Digital Equipment Corporation, Eastman Kodak Company, Honda R&D Co., Ltd., Industrial Technology Research Institute (ITRI), Komatsu Ltd., Eugene McDermott Foundation, Merrill-Lynch, NEC Fund, Oxygen, Siemens Corporate Research, Inc., Sony, Sumitomo Metal Industries, and Toyota Motor Corporation.

References

[1] J. Bullier. Integrated model of visual processing. Brain Research Reviews, 36:96-107, 2001

[2] S. Hochstein and m. Ahissar. View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron , 36(5):791-804, 2002.

[3] S. Ullman. Sequence seeking and counter streams: a computational model for bidirectional information flow in the visual cortex. In Cerebral Cortex, 5: 1-11, 1995.

[4] D Hubel, T. Wiesel. Receptive fields and functional architecture of monkey striate cortex. J. Physiology 195(1):215-43, 1968.

[5] K. Tanaka. Columns for complex visual object features in the inferotemporal cortex: clustering of cells with similar but slightly different stimulus selectivities. Cerebral Cortex 13:90-99, 2003.

[6] M. Riesenhuber, T. Poggio. Hierarchical models of object recognition in cortex. Nature Neuroscience, 2(11):1019-25.

[7] T. Serre, L. Wolf, T. Poggio. Object recognition with features inspired by visual cortex. In IEEE Computer Vision and Pattern Recognition, 2:994-1000, 2005.

[8] C. Hung, G. Kreiman, T. Poggio and J.J. DiCarlo. Fast readout of object identity from Macaque inferior temporal cortex. Science, Vol. 310, 863-866, 2005.

[9] Freedman, D.J., M. Riesenhuber, T. Poggio, and E.K. Miller. Comparison of primate prefrontal and inferior temporal cortices during visual categorization. Journal of Neuroscience, 23, 5235-5246, 2003.

[10] L. Wolf, S. Bileschi, E. Meyers. Perception strategies in hierarchical vision systems. To Appear in IEEE Computer Vision and Pattern Recognition, 2006.