The Computer Vision Lab at GET
University of Paderborn Department of Electrical Engineering GET - Grundlagen der ElektroTechnik Associated Member of the Heinz Nixdorf Institute (HNI)

Detection of Occlusions

Without the occurrence of occluded regions in the images, stereo matching is a one to one mapping of the two images. In general, however, there may be several objects in the scene with different distances in relation to the cameras which cause discontinuity in disparity and occlusions near intensity edges defining the boundaries of different object surfaces. Constraints such as uniqueness, smoothness or ordering of the disparity, which are utilized to simplify the matching process are invalid assumptions in occluded regions. If occlusions are not specially treated in the matching process, they may be incorrectly matched with regions in the other image. Although occlusions are one of the essential reasons for wrong matches in stereo analysis, there are only a few approaches which treat them explicitly. One way to avoid correspondence errors in occluded areas is a bidirectional or dual matching process [1-4,7]. In this approach matching is carried out from the left to the right and from the right to the left image in two separate, but identical processes. Occluded areas, which are indicated by the mismatch between the two disparity maps, are marked in so called occlusion maps and they are excluded from further calculations.

In the following approach occlusion detection is demonstrated on a correlation-based technique [5,6]. In the first step the stereo images are filtered by oriented Gabor filters in order to extract horizontal changes in intensity. With the convolution of the product of the left filter response r_l(x) and a spatially shifted complex conjugate version of the right filter response r_r(x) with a small real valued window w(x), we obtain a local, complex-valued measurement of the similarity between the filtered images from the right image to the left image. This measurement is normalized to the local energy of the filter responses:

where the disparity d = [d₁, d₂]^T acts as a two-dimensional spatial displacement of the right filter response and x = [x₁ x₂]^T are the image coordinates. The similarity measurement which is done in the opposite direction is then given by:

The peaks in the real part of the similarity measurement act as candidate disparities d between the two images with pixel accuracy. The imaginary part of the similarity measurement can be used to improve the disparity estimstion to subpixel accuracy [5,6]. The following example shows a bidirectional similarity measurement for selected points in a random dot stereogram.

Random dot stereogram with bidirectional similarity measurements

The figure above shows a random dot stereogram, in which a square area marked by the yellow frame is inserted in the images with a relative horizontal displacement of ten pixels. The image point marked by the red point in the left image does not occur in the right image. The similarity measurement, which is carried out from left to right at this point shows a peak in the real part, which corresponds to the position marked by the green point in the right image. Due to the high similarity measurement of this match this would lead to a wrong disparity estimation in a matching process carried out in only one direction. The real part of the similarity measure carried out from right to left must show the same peak at the disparity with the opposite sign at the corresponding position in the right image. But in this measurement there is another, higher peak, which corresponds to the match with the correct image area in the left image marked by the green point. The mismatch between the two similarity measurements can be exploited to detect occluded image areas.
This technique is based on the assumption that the match with features or pixel intensities in occluded areas is not as good as the match with the correct regions in the matching process, which is carried out in the other direction. Due to interocular differences, this need not be generally true. Furthermore, the bidirectional matching process is computationally twice as expensive as a matching process carried out in only one direction.
A new implicit strategy for occlusion detection without a bidirectional matching process is given in [5,6]. In this approach a self-organizing process is used to disambiguate the correspondence problem and to suppress matches with occluded image areas. Hereby the following symmetry property of the similarity measurement is exploited to avoid an explicit bidirectional matching strategy.

This identity is used to establish a special coupling structure in the self-organizing process by which the occlusion problem in stereo can be treated in a robust and effective way. The following figure shows some stereo images and their disparity maps in which occlusions (red areas) are detected by this approach.

Natural stereograms and their disparity maps This stereo approach is used to localize objects in the DEMON project.

Last change: 9.12.1998 (R.Trapp)