High Resolution Tracking of Non-Rigid 3D Motion Using Harmonic Maps

In this section, we present a fully automatic method for high resolution, non-rigid dense 3D point tracking [9]. Harmonic maps were used in [11] to do surface matching, albeit focusing on rigid transformations. Given the source manifold $M$ and the target manifold $D$, only the boundary condition $u\vert _{\partial M} : \partial M \rightarrow \partial D$ was used to constrain and uniquely determine the harmonic map $u: M \rightarrow D$. For applications like high resolution facial tracking though, we need to account for non-rigid deformations, with a high level of accuracy. To this end, we introduce additional feature correspondence constraints, in addition to the boundary constraint in our implementation of harmonic maps. We select a set of motion-representative feature corners (for example, for facial expression tracking, we select corners of eyes, lips, eye brows etc.) and establish inter-frame correspondences using commonly used techniques (for example, hierarchical matching used in [10]). We can then integrate these correspondence constraints with the boundary condition to calculate harmonic maps, which not only account for global rigid motion, but also subtle non-rigid deformations and hence achieve high accuracy registration and tracking.

The algorithm is illustrated in Figure 1 by considering the example of a synthetic surface $S$ undergoing non-rigid deformation.

Figure 1: Illustration of Harmonic Map: A Synthetic Example. (a) $S_o$: Initial configuration of surface. (b) $S_t$: Surface after non-rigid deformation. (c) $D_o$: Harmonic Map of $S_o$ with the hard boundary constraints only. (d) $D_t$: Harmonic map of $S_t$ with the hard boundary constraints only. We can notice that although $D_o$ and $D_t$ conform to each other around the boundary, the interior non-rigid deformation is still unaccounted for. (e) $D'_o$: Harmonic map of $S_o$ with the 'tip of the nose' as an additional feature-correspondence constraint. We can see that imposing correspondence constraints aligns $D'_o$ and $D_t$ better, resulting in accurate registration.

(a) (b)

(c) (d) (e)

High quality dense point clouds of facial geometry moving at video speeds are acquired using a phase-shifting based structured light ranging technique [6]. To use such data for temporal study of the subtle dynamics in expressions, an efficient non-rigid 3D motion tracking algorithm is needed to establish inter-frame correspondences. Because our dynamic range sequences are acquired at a high frame rate (30 Hz), we can assume that the local deformation between two adjacent frames is small. To register two frames, we align their respective harmonic maps as closely as possible by imposing the suitable boundary and feature constraints. The motivation to do so is to establish a common parametric domain for the two surfaces, which, coupled with the above mentioned property, allows to recover 3D registration between the two frames.In our case, the harmonic maps are diffeomorphisms, that is one to one and on-to, and hence lend themselves as a natural choice for surface parameterization in tracking applications.

The outline of the non-rigid tracking algorithm is given as follows:

• Data Preparation and Initialization:
  Identify the boundary and fit a coarse generic face mesh model to the first frame using Free-Form Deformation (FFD).
• Coarse Registration:
  1. Rough alignment: Globally align the 3D face scans of the adjacent frames using the standard Iterative Closest Points (ICP) techniques.
  2. Since we have the boundary of the first frame, identify the boundary for all the subsequent frames automatically given the global alignment achieved with the previous frame. Calculate the initial harmonic maps onto 2D disks using the boundary condition.
  3. For coarse level registration between successive frames, introduce more constraints on the harmonic map using feature point correspondence constraints, where features are detected using standard methods like corner detection.
• Iterative Refinement:
  1. Iteratively augment the list of constraints for the harmonic map with the local feature correspondences obtained using commonly used techniques (for example, hierarchical matching used in [10]).
  2. Repeat the previous step to progressively refine the harmonic map until the difference between the new source harmonic maps and the target harmonic maps recedes below a pre-defined threshold.
  3. Overlay the new source and the target harmonic map disks to establish dense registration, and hence recover the deformation parameters for the generic face mesh model between two consecutive frames.
  4. Continue this process over the whole sequence to achieve high resolution tracking.

The accuracy of the proposed tracking algorithm is demonstrated through experiments on real data. We performed tracking on four subjects performing various expressions for a total of twelve sequences of 250-300 frames each (at 30Hz). Each frame contains approximately 80K 3D points, whereas the generic face mesh contains 8K nodes. The tracking results are available as video clips at http://www.cs.sunysb.edu/~ial/expressionModeling.html, including opening and closing of the mouth (female subject) or strongly asymmetric smile (male subject). Our technique tracks very accurately even in the case of topology change and severe `folding' of the data. (See Figure 2)

Figure 2: Snapshots from a Tracking Sequence of Subject A: a) Initial data frame. b) Initial tracked frame. c) Data at the expression peak. d) Tracked data at the peak. e) Close-up at the peak.

(a) (b) (c) (d) (e)