Face Transfer is a method for mapping videorecorded performances
of one individual to facial animations of another. It extracts
visemes (speech-related mouth articulations), expressions,
and three-dimensional (3D) pose from monocular video or film
footage. These parameters are then used to generate and drive a
detailed 3D textured face mesh for a target identity, which can be
seamlessly rendered back into target footage. The underlying face
model automatically adjusts for how the target performs facial expressions
and visemes. The performance data can be easily edited
to change the visemes, expressions, pose, or even the identity of
the target—the attributes are separably controllable. This supports
a wide variety of video rewrite and puppetry applications.
Face Transfer is based on a multilinear model of 3D face meshes
that separably parameterizes the space of geometric variations due
to different attributes (e.g., identity, expression, and viseme). Separability
means that each of these attributes can be independently
varied. A multilinear model can be estimated from a Cartesian
product of examples (identities × expressions × visemes) with
techniques from statistical analysis, but only after careful preprocessing
of the geometric data set to secure one-to-one correspondence,
to minimize cross-coupling artifacts, and to fill in any
missing examples. Face Transfer offers new solutions to these problems
and links the estimated model with a face-tracking algorithm
to extract pose, expression, and viseme parameters.
