Gain Control with Normalization in the Standard Model

Minjoon Kouh & Tomaso Poggio

The Problem

Across cortex, especially in the sensory areas, many neurons respond strongly to some stimuli, but weakly to others, as if they were tuned to some optimal features or some particular input patterns. For examples, olfactory neurons respond to particular mixtures of molecules, or odors. Auditory neurons of a songbird are tuned to sound patterns. Neurons along the ventral pathway of primate visual cortex are known to be responsive to different visual features. In primary visual cortex V1, neurons show Gaussian-like tuning in multiple dimensions, such as orientation, spatial frequency, direction, and velocity [1]. V4 neurons show tuned responses to different types of gratings or contour features [2,3]. IT neurons are responsive to a particular view of a face or other objects [4]. However, the underlying neural mechanism of such tuning operation remains a puzzle.

Motivation

It was observed [5,6] that the Gaussian function (based on Euclidean distance) is closely related to the normalization and the weighted sum by the following mathematical relationship:

In other words, a bell-shaped function like Gaussian can be expressed as a normalized scalar product.

Gain control circuits by normalization, therefore, may underlie the "mysterious" Gaussian-like tuning of cortical cells. The advantage of considering normalized scalar product is its biological plausibility. Weighted sum can be easily performed by synaptic weights, and the normalization can be implemented by gain control circuits (possibly using feedforward or lateral shunting inhibition). Sigmoid-like nonlinearity, which is found in the pre- and post-synaptic responses, can control the magnitude and the sharpness of tuning.

Previous Work

The standard model, a quantitative model of the first few hundred milliseconds of primate visual perception [10] is based on many widely accepted ideas and observations about the architecture of primate visual cortex, and it reproduces many observed shape tuning properties of the neurons along the ventral pathway.

This computational model captures two major trends observed along the ventral pathway, both of which are essential for achieving object recognition. The neurons are progressively tuned to more complex and specific shapes (specificity), and their responses are increasingly invariant to such transformations as translation and scaling (invariance). The model is based on two key operations performed at various levels in the hierarchical architecture: Gaussian tuning to optimal stimuli to provide selectivity and maxi-mum operation to provide invariance.

It has been show that the Gaussian function can be well approximated with a normalized scalar product followed by a sigmoid nonlinearity [11]. Interestingly, another principle operation in the model, computation of maximum, can also be approximated with a similar functional form [12]. Our preliminary results show that the essential properties of the standard model are intact when the Gaussian and the maximum functions are replaced by the normalized scalar product.

Current Work

We will continue to investigate the biological plausibility and validity of the normalization operation in the context of cortical tuning. The gain control by normalization will be applied to different layers of the standard model [2005 CSAIL Abstract Cadieu et al.], and its performance will systematically be checked and tested for consistency with past experiments.

There are many interesting aspects of gain control operation. We will explore how the synaptic weights, or the tuned activation patterns, can be learned with reinforcement learning, or stochastic gradient descent [13], [2005 CSAIL Abstract Serre et al.]. More detailed and specific biophysical mechanisms of normalization will be studied [2005 CSAIL Abstract Knoblich et al.]. Lastly, we will also investigate the behavior of different tuning operations when more than one stimuli are presented (clutter condition), trying to simulate the old [14] and the new [2005 CBCL Abstract Zoccolan et al.] experimental results.

Impact

The advantage of using the normalized scalar product as a tuning operation is its biophysical plausibility. Unlike the computation of Euclidean distance or a Gaussian function, both normalization and scalar product operations can be readily implemented with a network of neurons. This project is a part of CBCL's continuing efforts of validating and enhancing the standard model with biophysically plausible architecture and the accumulating evidences from the experimental data.

Acknowledgments

This report describes research done at the Center for Biological & Computational Learning, which is in the McGovern Institute for Brain Research at MIT, as well as in the Dept. of Brain & Cognitive Sciences, and which is affiliated with the Computer Sciences & Artificial Intelligence Laboratory (CSAIL).

This research was sponsored by grants from: Office of Naval Research (DARPA) Contract No. MDA972-04-1-0037, Office of Naval Research (DARPA) Contract No. N00014-02-1-0915, National Science Foundation (ITR/SYS) Contract No. IIS-0112991, National Science Foundation (ITR) Contract No. IIS-0209289, National Science Foundation-NIH (CRCNS) Contract No. EIA-0218693, National Science Foundation-NIH (CRCNS) Contract No. EIA-0218506, and National Institutes of Health (Conte) Contract No. 1 P20 MH66239-01A1.

Additional support was provided by: Central Research Institute of Electric Power Industry (CRIEPI), Daimler-Chrysler AG, Compaq/Digital Equipment Corporation, Eastman Kodak Company, Honda R&D Co., Ltd., Industrial Technology Research Institute (ITRI), Komatsu Ltd., Eugene McDermott Foundation, Merrill-Lynch, NEC Fund, Oxygen, Siemens Corporate Research, Inc., Sony, Sumitomo Metal Industries, and Toyota Motor Corporation.

References

[1] D. Hubel and T. Wiesel. Receptive fields and function architecture of monkey striate cortex. Journal of Physiology, Vol. 195, pp. 215--243, 1968.

[2] J. Gallant et al. Neural responses to polar, hyperbolic, and Cartesian gratings in area V4 of the Macaque monkey. Journal of Neurophysiology, Vol. 76, pp. 2718--2739, 1996.

[3] A. Pasupathy and C. Connor. Shape representation in area V4: Position--specific tuning for boundary conformation. Journal of Neurophysiology, Vol. 86, pp. 2505--2519, 2001.

[4] N. Logothetis et al. Shape representation in the inferior temporal cortex of monkeys. Current Biology, Vol. 5, pp. 552--563, 1995.

[5] M. Maruyama et al. A connection between GRBF ane MLP. MIT AI Memo 1291, 1992.

[6] T. Poggio and E. Bizzi. Generalization in Vision and Motor Control. Nature, Vol. 431, pp. 768--774, 2004.

[7] W. Reichardt et al. Figure-ground discrimination by relative movement in the visual system of the fly - II: Towards the neural circuitry. Biological Cybernetics, Vol. 46, pp. 1--30, 1983.

[8] D. Heeger. Modeling simple-cell direction selectivity with normalized, half-squared, linear operators. Journal of Neurophysiology, Vol. 70, pp. 1885--1898, 1993.

[9] M. Carandini and D. Heeger. Summation and division by neurons in primate visual cortex. Science, Vol. 264, pp. 1333--1336, 1994.

[10] M. Riesenhuber and T. Poggio. Hierarchical models of object recognition in cortex. Nature Neuroscience, Vol. 2, pp. 1019--1025, 1999.

[11] M. Kouh and T. Poggio. A general mechanism for tuning: Gain control circuits and synapses underlie tuning of cortical neurons. MIT AI Memo 031, 2004.

[12] A. Yu et al. Biophysiologically plausible implementations of the maximum operation. Neural Computation, Vol. 14, pp. 2857--2881, 2002.

[13] H. S. Seung. Learning in Spiking Neural Networks by Reinforcement of Stochastic Synaptic Transmission. Neuron, Vol. 40, pp. 1063-1073, 2003.

[14] J. Reynolds et al. Competitive Mechanisms Subserve Attention in Macaque Areas V2 and V4. Journal of Neuroscience, Vol. 19, pp. 1736-1753, 1999.