Lihi Zelnik
Temporal factorization [5] is the dual approach to the more familiar multi-body factorization. The traditional multi-body factorization approaches (e.g., [1,2,3]) provide spatial clustering/segmentation by grouping together points moving with consistent motions. This is done by grouping columns of the correspondence matrix of [4]. Temporal factorization [5] is the dual approach to factorization, i.e., obtaining temporal clustering/segmentation by grouping together frames capturing consistent shapes. This is done by clustering the rows of the correspondence matrix of [4] instead of its columns. Temporal cuts are thus detected at non-rigid changes in the shape of the scene/object. For example, in a sequence showing a face which appears serious at some frames, and is smiling in other frames, all the "serious expression" frames will be grouped together and separated from all the "smile" frames which will be classified as a second group, even though the head may meanwhile undergo various random motions.
Results of factorization applied to a sequence
taken from the movie "Brave Heart". The actor (Mel Gibson) is serious at
first and then smiles while moving his head independently from his
expression throughout the sequence. Optical flow was estimated
relative to the first frame and the clustering was applied
directly to it. We set the number of clusters to 2.
This is a sample frame from the first detected temporal cluster,
which shows the actor smiling:
This is a sample frame from the second detected temporal
cluster which shows the actor serious:
Results of temporal factorization (into 2 clusters)
applied to a video clip taken from
the movie "Lord of the Rings - Fellowship of the Ring". The clip
shows two hobbits first calm and then screaming. The shape-based
temporal factorization detected the cut between the two expressions and
grouped together all the "calm" frames separately from all the "scream" frames.
An example calm frame is:
An example calm "scream" frame is:
Videos can be found
here.