Research Abstracts - 2007

Material Perception

Lavanya Sharan,Yuanzhen Li & Edward H. Adelson

Figure 1: Perceiving the properties of the materials in this image is vital to interpreting it.

What:

We find it easy to distinguish a plastic spoon from a stainless steel spoon or a wooden spoon. This ability is known as material perception. It is different from object recognition, which is the ability to distinguish a spoon from say a fork. While object recognition has been widely studied, material perception has received less attention. We are trying to understand how humans perceive material properties and how such ability can be achieved by machines. 

Why:

The world that we see around us is composed of objects as well as materials. We can discern subtle changes in color, texture, gloss and translucency and we employ this sensitivity daily e.g. judging if food is cooked (nicely browned onions) or if a product on eBay is worth bidding (shiny paint on a bicycle).  If we can understand how humans assess visual qualities of materials and surfaces, we can build machines that can do the same. An immediate application would be in the field of computer graphics, where realistic depictions of hair, skin, clothes etc. are desired. Such knowledge would also benefit product design; aesthetically appealing products like really glossy lip gloss or natural looking leather can be manufactured. In terms of practical tasks, a domestic robot with material recognition capability can tell the difference between spilled flour and spilled yoghurt and use appropriate cleaning tools for each material.

How Images of two dissimilar objects made of the same material differ greatly on a pixel by pixel basis (making techniques like template matching inapplicable), yet both convey the impression of the material. The problem of material recognition is related to texture analysis however there is an important difference. It is often assumed that textures are generated by a stationary stochastic processes but this is not true for samples of material (e.g., a chrome sphere) Nevertheless, the field of texture offers some useful ideas. Recent work in texture has shown that pixel and wavelet statistics are good texture descriptors. We are adapting these descriptors to deal with problems in material perception.

Dror et al [1] started with the simplified situation where they examined spheres of homogeneous reflectance in unknown illumination. They used image based textural statistics in a machine learning framework to classify the spheres as shiny, matte, white, gray etc. Fleming et al [2] made progress on the human vision aspect of this problem, by demonstrating that humans can estimate surface reflectance of objects in the absence of context, as long as the illumination conditions are representative of those found in the real world.

Progress:

We photographed many real world materials under various artificial illuminations. We find that pixel statistics like the moments and percentiles of the intensity histogram are correlated with the surface reflectance. We filtered the images with center surround and oriented filters in a multi-scale decomposition and observed that the moment and percentiles of the histograms of the filtered images are diagnostic of reflectance as well. We find that these simple image statistics perform as well as human subjects at reflectance classification and estimation tasks. Moreover if we manipulate these statistics of an image of a real-world surface the perceived reflectance of that surface changes. These findings, taken together, suggest that our set of image statistics capture perceptually relevant information.

Future:

We would like to validate our results on larger and more challenging image data sets. We also want to extend beyond reflectance estimation to other aspects of material perception – glossiness perception, translucency perception etc.

Research Support:

This work is supported by NTT, as part of a collaboration with Dr. Shin’ya Nishida and Dr. Isamu Motoyoshi at NTT CS Labs, Atsugi, Japan [7]. Support has also come from NSF, NIH and Unilever Research.

Figure 2: Simple image statistics like the moments or percentiles of the luminance histogram are predictive of the diffuse reflectance of a surface. Here we demonstrate that manipulating these statistics of an image produces a perceptual change in the reflectance. The top row shows a pair of surfaces, the left one is white modeling clay and one on the right is black stucco. Both images are normalized to have the same mean luminance and their luminance histograms are plotted to their right in red and blue respectively. Now if we force the luminance histogram of the white surface to be the same as that of the black surface and vice-versa, we obtain the pair of images in the bottom row. The histograms of the manipulated images are plotted to the right of the images in black and magenta. Note how the white clay surface now looks darker and shinier and the black stucco surface looks less glossy and lighter.

Figure 3:  Image statistics agree with human observers. Here o   bserver ratings for all surfaces are plotted against the model ratings. The model is a support vector regression method that uses 4 features mainly standard deviation and (90^th - 10^th) percentile of the filtered outputs. Both the model and the observers rate a surface along the diffuse reflectance dimension, from 0 (black) to 1 (white). The four plots show the data for four different observers. If the model behaves as an average observer we would expect all points to lie on or near the diagonal black line with slope 1. Our data demonstrate that is indeed so. The model tends to agree with observers and the degree of agreement is comparable to the agreement between observers.

References

[1] Ron O. Dror, Edward H. Adelson and Alan S. Willsky. Recognition of surface reflectance properties from a single image under unknown real-world illumination. In Proceedings of the Workshop on Identifying Objects Across Variation in Lighting at CVPR 2001, Hawaii, December 2001.

[2] Roland W. Fleming, Ron O. Dror and Edward H. Adelson. Real-world illumination and the perception of surface reflectance properties.  Journal of Vision, 3:347-368, 2003.

[3] Isamu Motoyoshi, Shin’ya Nishida and Edward H. Adelson. Image statistics as a determinant of reflectance perception. Journal of Vision Volume 5, Number 8, Abstract 569. (http://journalofvision.org/5/8/569/) May 2005.

[4] Edward H. Adelson, Yuanzhen Li and Lavanya Sharan. Image statistics for material perception. Journal of Vision Volume 4, Number 8, Abstract 123. (http://journalofvision.org/4/8/123/) May 2004.

[5] Lavanya Sharan. Image statistics and the perception of surface reflectance. S. M. thesis, Massachussetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2005.          

[6] Lavanya Sharan, Yuanzhen Li and Edward H. Adelson. Image statistics and reflectance estimation. Journal of Vision Volume 5, Number 8, Abstract 375. (http://journalofvision.org/5/8/375/) May 2005.

[7] Isamu Motoyoshi, Shin'ya Nishida, Lavanya Sharan and Edward H. Adelson. Image statistics and the perception of surface qualities Nature (in press) 2007.