The Saliency of Curves
Michael Lindenbaum
The human visual system (HVS) is capable of filtering images and finding
the important visual events so that its limited computational resources
may be focused on them and used efficiently. This discrimination between
the important parts of the image, denoted ``figure'', and the less important
parts, denoted ``background'', is done before the objects in the image
are identified, and using general rules (or cues) indicating what is likely
to be important [Wer23]. Presented, for example, with a binary image containing
points and/or curves (such as those resulting from edge detection), it
turns out that this perceptual process prefers to choose for figure, a
subset of points lying on some long, smooth and dense curve. Saliency methods
in computer vision are algorithms which get an image and provide such preference
in the form of a saliency map, where prefered locations get higher value.
The term was coined by Shaashua and Ullman who also suggested
an algorithm [SU88]. They specified a particular measure (the saliency
of a curve) that is a particular quantification of the desirable smoothness
and length properties, and relies on curvature estimates and gaps. Every
image point may be associated with a saliency measure as well, which is
just the highest saliency of all curves starting at this point. They have
shown that indeed, the image subsets, associated with high saliency are
those considered as more important by common human subjective judgment.
One important advantage of their algorithm is that its implementation may
be formulated as dynamic programming and consequently may be carried out
as an iterative process running on a network of simple processors getting
only local information. This makes the theory attractive not only as an
algorithm but also as a computational theory (i.e. a reasonable grounded
explanation to visual processes in the HVS) because the proposed process
is consistent with common neural mechanisms.
The algorithm itself is based on the following step
Sk+1i = MAXj [Ei
+
cij Skj ]
where
Skj - the saliency of the
j-th image point after the k-th iteration
cij - A coefficient which is larger
if the i-th and the j-th image points are likely to be on the same smooth
curve (low curvature and no gaps make cij larger)
Ei - A term which is higher if
there is an edge point at the i-th image point.
The maximum MAXj [.] is taken over
all neighbors of the i-th image point.
The algorithm repeats this iteration many times, in parallel
over all the image, until the saliency values converge everywhere. At this
point the endpoints of the long and smooth curves will have a high saliency
value.
Typical results on synthetic images:
A line segments image (left) and its Shaashua Ullman saliency
map (middle). Thresholding the saliency we get indeed the "closed" curve,
considered by most people as the most important image part (right)*
When applied over real images the results are less than perfect:
The original "lizzard" image (left), its edge image (second left), the
saliency map (second right) and the thresholded saliency (right) which
gives more or less the "right" contours **.
The saliency concept was followed in several other publications. A partial
list
-
The SU saliency measure was re-analyzed, revealing some inherent deficiencies,
such that the maximal saliency may not appear on the visually most salient
curve and the scale variance of the saliency calculation [AB98].
-
A probabilistic variation on the US saliency, suggests to look at the affinity
between two image features (measured by US depending on curvature, gaps,
etc) as a probability that these two feature are connected. With these
probabilistic interpretation, the distribution of curve lengths is
considered and the US saliency is interpreted as the expected of
the curve. Besides, other types of saliency which are not geometry (curvature)
based are derived [LB00].
-
Another type of saliency is non-iterative and is based on the consistency
of locally supporting votes. This method, denoted "tensor voting" was applied
for detecting 3D surfaces as well [GM96].
-
Some papers, aiming to model the HVS ability to complete occluded boundary
(using stochastic models for particle motion), produce saliency processes
as well. [WJ95]. A survey on different saliency methods is described
in [WT98].
-
Work on Figure from Ground discrimination such as [HH93], do not explicitly
address the saliency issue but, implicitly, calculate a (binary) saliency
as well.
References:
-
[AB98] - Alter, T.D. and Basri, R., Extracting Salient Curves from Images:
An Analysis of the Saliency Network, IJCV 27(1), pp. 51-69, 1998.
-
[GM96] - Guy, G. and Medioni, G.G. , Inferring Global Perceptual Contours
from Local Features, IJCV 20(1), pp. 113-133, 1996,
-
[HH93] - Herault, L., and Horaud, R., Figure-Ground Discrimination: A Combinatorial
Approach,PAMI 15(9), pp. 899-914, 1993 .
-
[LB00] - Lindenbaum, M., Berengolts, A., A Probabilistic Interpretation
of the Saliency Network, ECCV2000, 2000.
-
[SU88] - Shaashua, A. and Ullman, S., Structural Saliency: The Detection
of Globally Salient Structures Using Locally Connected Network, ICCV88,
pp. 321-327, 1988.
-
[We50] - Wertheimer, M., Laws Of Organization in Perceptual Forms, in A
Source Book of Gestalt Psychology, W.D. Ellis. Ed., pp. 71-88, 1950.
-
[WT98] -Williams, L. and Thornber, K., A Comparison of Measures for
Detecting Natural Shapes in Cluttered Backgrounds, ECCV98, 1998.
-
[WJ95] - Williams, L.R. and Jacobs, D., Stochastic Completion Fields: A
Neural Model of Illusory Contour Shape and Salience, ICCV95, pp. 408-415,
1995.
* Thanks to Ronen Basri for contributing these images.
** Implementation based on the work of Lindenbaum and Berengolz [LB]
Michael Lindenbaum
Last
modified: Wed Dec 6 15:01:02 IST 2000