CSAIL Publications and Digital Archive header
bullet Technical Reports bullet Work Products bullet Research Abstracts bullet Historical Collections bullet

link to publications.csail.mit.edu link to www.csail.mit.edu horizontal line


Research Abstracts - 2006
horizontal line

horizontal line

vertical line
vertical line

Elucidating the Role of Top-Down Processing in the Brain Using Neural Readout Techniques and Computational Models

Ethan Meyers, Gabriel Kreiman & Lior Wolf

The Problem

In primate visual cortex, feedback connections from higher visual areas to lower visual areas outnumber feed-forward connections in a ratio of 10 to 1 [1]. Yet despite the abundance of these feedback connections, the functional role that these connections play in vision remains largely unknown. Several theorists have assigned speculative roles to feedback connections[2, 3], however, such proposals have usually been philosophical in nature, and have lacked any substantial evidence in terms of either neural data or in terms of working computational models. In contrast, many of the properties of the neurons involved in the feed-forward pathway are known [4, 5]. Also, recent computational models, such as the Standard Model proposed by Riesenhuber and Poggio [6], can both match the neurophysiological properties of neurons found in the feed-forward visual pathway, and visual features derived from such models have been shown to achieve high levels of performance in computer vision tasks [7].

In the project, we propose to try to gain a better understanding the role of these feedback connections in visual processing by taking a dual approach of both developing new methods of analyzing neural data and by exploring possible computational models in which feedback connections aid computations useful for vision.


To try to find patterns in neural data that can elucidate the role of these feedback connections, we propose to further develop the readout methods used in Hung et al.[8]. In particular, we intend to use machine learning classifiers, such as support vector machines, to categorize which stimuli are being represented in different cortical areas of Rhesus Macaque monkeys, while the monkeys engages in a delayed match to sample experiment similar to those used by Freedman et al. [9]. By decoding patterns in neural firing in the inferiotemporal (IT) and prefrontal (PFC) cortex for object and category specific signals during the fixation, sample, delay, and test periods, we intend to determine which time periods contain object and/or category specific information, which will hopefully give us insight into when feedback processing might be occurring. We are also interested in exploring new methods that look for synchronized activity between PFC and IT which might further enhance these readout techniques.

In terms of computational modeling, we intend to explore how feedback information coming from higher areas can help improve the performance of computer vision systems in specific tasks. In particular, we would like to add feedback processing to Risenhuber and Poggio’s Standard Model and see if and how such processing can help achieve higher classification rates on object recognition tasks. We are also interested in exploring other tasks in which feedback processing might be useful, such as finding the specific location of objects in space, and searching for harder to identify objects in highly cluttered scenes.

Preliminary Results

Preliminary results using neural readout on data collected in a delayed match to sample paradigm published by Freedman et al. [9] shows that there is a strong stimulus specific signal in IT during the period that the stimulus is present, and a weaker stimulus specific signal during the delay period (see figure 1 below). It is harder to tell whether signals coming from PFC are stimulus specific based on our current analysis, but future analyses should give us a more definite answer. Looking at category specific signals and synchronous firing between PFC and IT should also help us to better interpret what these areas are representing and how they are interacting.

preliminary readout results from IT and PFC

Figure 1. Classification accuracy of 54 stimuli based on recordings from IT (blue line) and PFC (dashed red line) over time. The fixation period is from 0-500ms; the sample period is from 600-1100ms; the delay period is from 1100-2100; the test is from 2100-2600ms. Chance performance is approximately 2%.

Preliminary results from computational models have shown that adding feedback to hierarchical classification systems does improve their performance. In recent experiments, we compared five different hierarchical classification schemes in four object classification tasks [10]. Figure 2 shows the different hierarchies used and the results from the Caltech 101 object dataset. As can be seen, the type of architecture used can have a large influence on performance, and in particular, the feedback scheme which we called ‘semantic concatenation’ tended to consistently obtain some of the best results.

An example of several feed-forward and feedback hierarchies used in object recognition experiments. Results from the different hierarchies on the Caltech 101 dataset

Figure 2. Left: Different feed-forward and feedback hierarchical schemes used to test how effective feedback is in object recognition tasks. Note that while all these schemes are drawn to look like feed-forward systems so that they are easier to compare, in fact schemes (d) and (e) contain feedback processing (for more details see 10). Right: Results from running these hierarchies on the Caltech 101 object dataset.


This report describes research done at the Center for Biological & Computational Learning, which is in the McGovern Institute for Brain Research at MIT, as well as in the Dept. of Brain & Cognitive Sciences, and which is affiliated with the Computer Sciences & Artificial Intelligence Laboratory (CSAIL).

This research was sponsored by grants from: Office of Naval Research (DARPA) Contract No. MDA972-04-1-0037, Office of Naval Research (DARPA) Contract No. N00014-02-1-0915, National Science Foundation (ITR/SYS) Contract No. IIS-0112991, National Science Foundation (ITR) Contract No. IIS-0209289, National Science Foundation-NIH (CRCNS) Contract No. EIA-0218693, National Science Foundation-NIH (CRCNS) Contract No. EIA-0218506, and National Institutes of Health (Conte) Contract No. 1 P20 MH66239-01A1.

Additional support was provided by: Central Research Institute of Electric Power Industry (CRIEPI), Daimler-Chrysler AG, Compaq/Digital Equipment Corporation, Eastman Kodak Company, Honda R&D Co., Ltd., Industrial Technology Research Institute (ITRI), Komatsu Ltd., Eugene McDermott Foundation, Merrill-Lynch, NEC Fund, Oxygen, Siemens Corporate Research, Inc., Sony, Sumitomo Metal Industries, and Toyota Motor Corporation.


[1] J. Bullier. Integrated model of visual processing. Brain Research Reviews, 36:96-107, 2001

[2] S. Hochstein and m. Ahissar. View from the top: Hierarchies and reverse hierarchies in the visual system. Neuron , 36(5):791-804, 2002.

[3] S. Ullman. Sequence seeking and counter streams: a computational model for bidirectional information flow in the visual cortex. In Cerebral Cortex, 5: 1-11, 1995.

[4] D Hubel, T. Wiesel. Receptive fields and functional architecture of monkey striate cortex. J. Physiology 195(1):215-43, 1968.

[5] K. Tanaka. Columns for complex visual object features in the inferotemporal cortex: clustering of cells with similar but slightly different stimulus selectivities. Cerebral Cortex 13:90-99, 2003.

[6] M. Riesenhuber, T. Poggio. Hierarchical models of object recognition in cortex. Nature Neuroscience, 2(11):1019-25.

[7] T. Serre, L. Wolf, T. Poggio. Object recognition with features inspired by visual cortex. In IEEE Computer Vision and Pattern Recognition, 2:994-1000, 2005.

[8] C. Hung, G. Kreiman, T. Poggio and J.J. DiCarlo. Fast readout of object identity from Macaque inferior temporal cortex. Science, Vol. 310, 863-866, 2005.

[9] Freedman, D.J., M. Riesenhuber, T. Poggio, and E.K. Miller. Comparison of primate prefrontal and inferior temporal cortices during visual categorization. Journal of Neuroscience, 23, 5235-5246, 2003.

[10] L. Wolf, S. Bileschi, E. Meyers. Perception strategies in hierarchical vision systems. To Appear in IEEE Computer Vision and Pattern Recognition, 2006.

vertical line
vertical line
horizontal line

MIT logo Computer Science and Artificial Intelligence Laboratory (CSAIL)
The Stata Center, Building 32 - 32 Vassar Street - Cambridge, MA 02139 - USA
tel:+1-617-253-0073 - publications@csail.mit.edu