April 18, 2007

Elucidation of Neural Mechanisms Underlying Material Perception
From Surface Textures
Perception of glossiness and lightness discovered to originate from the
skewed distribution of the luminance of an image

Nippon Telegraph and Telephone Corporation (NTT; headquartered in Chiyoda-ku, Tokyo, Japan; president and CEO: Norio Wada) in collaboration with the Massachusetts Institute of Technology has elucidated how human brain makes use of the skewed distribution of the luminance of the image when viewing the glossiness and lightness*1 of a surface, from the experiments with natural images and illusions. This implies that the visual perception of materials, which until now has been mostly a mystery, is based on relatively simple neural information processing.
We found a high correlation between the human perception of glossiness and lightness and the skewness of the image's luminance histogram*2, and that the skewness of the luminance histogram can be easily calculated by low-level neural mechanisms already known. We further discovered a new illusion called the "skewness aftereffect," which demonstrates the existence of the proposed neural mechanisms in the brain. This study opens the door to a new "neuroscience of material perception." Furthermore, it is expected to contribute to the development of technological standards for high-resolution graphics transmission and generation.
Our findings are published in the April 18 electronic edition of the British journal "Nature."

1. Summary of Experiments

1.1 High correlations of the skewness of the luminance histogram with the physical glossiness/reflectance, and with the perceived glossiness/lightness
First, we made patches of stucco-like material with varying the amount of reflectance and glossiness. We photographed these objects and analyzed their images. We found that the glossier a dark surface is, the more positively skewed its luminance histogram (slope extending to high levels of luminance) (Figure 1).
Next, we showed these stucco images at a constant mean luminance to human observers and asked them to rate the lightness or glossiness of each surface. We discovered that the more positively skewed the luminance histogram of the image, the higher the observers rated the image's glossiness and the lower they rated the image's lightness.
That is, there are high correlations among the physical glossiness/reflectance, the skewness of the luminance histogram and the perceived glossiness/lightness.

1.2 Changes in the perceived glossiness/lightness in accordance with the skewness of the luminance histogram
Next, we manipulated the luminance histograms of some images of surfaces, and studied how changes in glossiness and lightness are perceived. We found that the perception clearly changed in response to the skewness of the luminance histograms. This experiment strongly suggests the possibility that human brain uses the statistical properties of an image as exemplified by the skewness of the luminance histogram in perceiving surface properties such as glossiness and lightness.

1.3 Proposal of possible neural computation of luminance skewness
We examined known physiology and discovered low-level visual mechanisms that can support the easy extraction of a histogram's skewness. On-center and off-center neurons are widely found in neural substrates for early visual processing, such as the retina, Lateral Geniculate Nuclei (LGN), and the primary visual cortex. We found that the balance of the response strength of these two types of neurons corresponds well to the skewness of the luminance histogram (Figure 2).

1.4 Discovery of an illusion that supports the existence of the proposed neural mechanism
Finally, we discovered an illusory phenomenon that precisely supports the perception of glossiness and lightness as a calculation of simple statistical values by these low-level neural mechanisms. After prolonged observation of images with positive skewness, subjects rated the glossiness of other images as lower than they are, and the lightness as higher (Figure 3). Images with negative skewness produced the opposite results. Such "skewness aftereffects" are perfectly consistent with the neural model described above.

2. Background and Significance of Research
We can easily distinguish materials from differences in their reflectance properties, such as glossiness, lightness, and transparency. However, estimating the reflectance property of a surface from a single image is an impossibly difficult computational problem, and how human brain solves such a hard problem has been a mystery.
Much research has been made on the physical aspects of surface image formation, and it is revealed that surface image qualities could be the products of extremely complex optical processes . For example, certain surfaces having transparency, such as the human skin, are too complex to correctly reproduce their images even with the latest computer graphics technology. Such complexities may give the impression that the perception of textures requires extremely high-level and complex neural processing. However, our research showed that, contrary to expectations, the perception is done by low-level visual processing mechanisms. Our discoveries, while partial, are among the first papers to elucidate the mechanism of surface reflectance perception, and open the door to the "neuroscience of material perception."
Surface reflectance properties are critical factors that lend richness, reality, and beauty to the world we see. Our finding that the perception of complex surface reflectance can be predicted and controlled by simple manipulations of two-dimensional images is expected to have particularly significant applications. For example, further expansion of our discoveries may make it possible to allow full control of an image's texture by adding simple processing functions.

3. Future Plans
Besides glossiness, we are working on elucidating the perceptual mechanisms of material perception such as skin and metals which have transparent and metallic properties. By understanding the cognitive functions of human brain, we at NTT seek to elucidate fundamental principles leading to such applications as new information displays and compression technologies, thus contributing to the realization of a rich information distribution society.

*1 Lightness: Strict technical term that refers not to the brightness of an image or the intensity of light, but instead to a surface's whiteness/blackness (reflectance ratio).
*2 Luminance histogram: A graph showing the frequency of the appearance of varying degrees of brightness (luminance) in a pixel. See Figure 1.

Figure 1: Images of surfaces with different glossiness and lightness, and their luminance histograms.
- Figure 2: Neural network that calculates the skewness of the luminance histogram.
- Figure 3: Adaptation stimulus and test stimulus producing the skewness aftereffect.

For further information, contact:
Kishine, Iizuka
Planning Division
NTT Science and Core Technology Laboratory Group
Tel: 046-240-5152


Copyright (c) 2007 Nippon telegraph and telephone corporation