The reason for not matching across individual gray levels arises from the belief that image features should correspond to physically realisable attributes of the real world scene being viewed. Gray levels do not correspond in this way; because of the complexities of the radiometric model, a particular grey level value may be a function of totally different scene characteristics. As a simple example, an object of low reflectance illuminated by an intense light source may give a similar pixel value to an object of high reflectance illuminated by a low intensity light source. In general, the same pixel grey level will occur in several different image locations; given a particular grey-level pixel value in one image it is impossible to deduce the position of of it's counterpart in the other image.

As individual pixel grey-level comparison is not viable, the first common approach to the comparison of stereo images consists of considering a small window of intensity values as a feature, and searching for a similar intensity window in the second image. This is effectively 2D correlation of the image window in the first image by superimposing it at successive positions in the other image to find the best match. Unless optimised, correlation proves computationally exhaustive. It also suffers when images taken at wide viewpoints are used, since the allowable disparity, and hence search area is correspondingly greater. In addition, scene objects can change their shape quite dramatically when viewed from significantly different viewpoints. This can be rectified by reducing viewpoint separation although at the expense of depth accuracy. Nevertheless, window correlation is a viable alternative to the edge based approach considered here, and may produce more dense depth data under favourable conditions.