The Saliency of Curves
Michael Lindenbaum

The human visual system (HVS) is capable of filtering images and finding the important visual events so that its limited computational resources may be focused on them and used efficiently. This discrimination between the important parts of the image, denoted ``figure'', and the less important parts, denoted ``background'', is done before the objects in the image are identified, and using general rules (or cues) indicating what is likely to be important [Wer23]. Presented, for example, with a binary image containing points and/or curves (such as those resulting from edge detection), it turns out that this perceptual process prefers to choose for figure, a subset of points lying on some long, smooth and dense curve. Saliency methods in computer vision are algorithms which get an image and provide such preference  in the form of a saliency map, where prefered locations get higher value.

The term was coined by Shaashua and Ullman  who also suggested an algorithm [SU88]. They specified  a particular measure (the saliency of a curve) that is a particular quantification of the desirable smoothness and length properties, and relies on curvature estimates and gaps. Every image point may be associated with a saliency measure as well, which is just the highest saliency of all curves starting at this point. They have shown that indeed, the image subsets, associated with high saliency are those considered as more important by common human subjective judgment.  One important advantage of their algorithm is that its implementation may be formulated as dynamic programming and consequently may be carried out as an iterative process running on a network of simple processors getting only local information. This makes the theory attractive not only as an algorithm but also as a  computational theory (i.e. a reasonable grounded explanation to visual processes in the HVS) because the proposed process is consistent with common neural mechanisms.

The algorithm itself is based on the following step

Sk+1i = MAXj [Ei + cij Skj ]
where
Skj  - the saliency of the j-th image point after the k-th iteration
cij   - A coefficient which is larger if the i-th and the j-th image points are likely to be on the same smooth curve (low curvature and no gaps make  cij larger)
Ei   - A term which is higher if there is an edge point at the i-th image point.
The maximum MAXj [.] is taken over all neighbors of the i-th image point.

The algorithm repeats this iteration many times, in parallel over all the image, until the saliency values converge everywhere. At this point the endpoints of the long and smooth curves will have a high saliency value.

Typical results on synthetic images:

saliency map

A line segments image (left) and its Shaashua Ullman  saliency map (middle). Thresholding the saliency we get indeed the "closed" curve,  considered by most people as the most important image part (right)*

When applied over real images the results are less than perfect:
 

saliency map

The original "lizzard" image (left), its edge image (second left), the saliency map (second right) and the thresholded saliency (right) which gives more or less the "right" contours **.


The saliency concept was followed in several other publications. A partial list

References:
  1. [AB98] - Alter, T.D. and Basri, R., Extracting Salient Curves from Images: An Analysis of the Saliency  Network, IJCV 27(1), pp. 51-69, 1998.
  2. [GM96] - Guy, G. and Medioni, G.G. , Inferring Global Perceptual Contours from Local Features, IJCV 20(1), pp. 113-133, 1996,
  3. [HH93] - Herault, L., and Horaud, R., Figure-Ground Discrimination: A Combinatorial Approach,PAMI 15(9),  pp. 899-914, 1993 .
  4. [LB00]  - Lindenbaum, M., Berengolts, A., A Probabilistic Interpretation of the Saliency Network, ECCV2000, 2000.
  5. [SU88] - Shaashua, A. and Ullman, S., Structural Saliency: The Detection of Globally Salient Structures Using Locally Connected Network, ICCV88, pp. 321-327, 1988.
  6. [We50] - Wertheimer, M., Laws Of Organization in Perceptual Forms, in A Source Book of Gestalt Psychology, W.D. Ellis. Ed., pp. 71-88, 1950.
  7. [WT98]  -Williams, L. and Thornber, K., A Comparison of Measures for Detecting Natural Shapes in Cluttered Backgrounds, ECCV98, 1998.
  8. [WJ95] - Williams, L.R. and Jacobs, D., Stochastic Completion Fields: A Neural Model of Illusory Contour Shape and Salience, ICCV95, pp. 408-415, 1995.

  9.  
* Thanks to Ronen Basri for contributing these images.
** Implementation based on the work of Lindenbaum and Berengolz [LB]

Michael Lindenbaum

Last modified: Wed Dec 6 15:01:02 IST 2000