Research Abstracts - 2007

Abstract

Acquiring 3D digital models of deforming objects is an open problem with applications in computer graphics, animation, simulation, as well as medicine. We seek to provide practical ways for acquiring the deforming geometry of human faces, bodies, skin or clothing from real world measurements such as images, motion or range data. Due to the under-constrained nature of this problem (for example, a frontal view of the human body does not constrain the deformation on the back), we assume an underlying model of geometry and deformation appropriate for the object of interest (e.g. a human body). This ”representative” model is manipulated within its constraints to match the measurements as best as possible.

For faces, the identity, expression, and speech parameters of a multilinear model are automatically extracted from video. For body motions, the joint angles of a rigid linked human body model are manipulated to match accelerometers, gyroscopes, and ultrasonic sensors placed on the subject’s body. For detailed skin/clothing motions, the rigid links will be replaced with moldable surfaces that will be deformed to match high-resolution high-speed range data. These methods will enable the collection of large high-quality human shape, motion and deformation databases, the impact of which would extend well beyond computer graphics.

Faces

The work on faces is published as “Face Transfer with Multilinear Models” [Vlasic et al. 2005]. The underlying model that enables the extraction and manipulation of 3D face geometry is a multilinear model, which is based on tensors and tensor decomposition. The main advantage of this model is that it can describe face variations with separable attributes, such as identity and expression. We show how to build such a model from incomplete data, as well as how to automatically estimate its parameters from video sequences. Some capabilities of this system are demonstrated in Figure 1.

Figure 1: Face Transfer with Multilinear Models gives animators decoupled control over facial attributes such as identity, expression, and viseme (speech-related mouth shapes). In this example, we combine the pose and identity from the first frame, a surprised expression from the second, and a viseme (midway between ”oo” and ”ee”) from the third. The resulting composite is blended back into the original frame.

Body motion

We have built an affordable, self-contained, wearable inertio-acoustic motion capture system that can acquire human body motions in natural surroundings. We use a skeletal model for the human body, consisting of rigid links (bones) that are connected at joints. The parameters of that model (joint angles) are computed to match the accelerometer (acceleration), gyroscope (angular velocity), and ultrasonic time-of-flight (distance) measurements for the sensors distributed on the subject’s body. The prototype performs well on complex and subtle motions, as illustrated in Figure 2. This work is currently under review for a computer graphics journal.

Figure 2: Our system can capture motions at almost any place for extended time periods. The subtle motions of hands and feet are clearly visible (right), including the interactions between the feet, pedals, hands and handle bars (left).

Skin/Clothing

As an end goal of this project, we would like to be able to acquire the full geometry of a human body as it moves and deforms. Although this work is in its infancy, we plan to use an enhanced model of a human body, similar to the rigid linked body we used for motion capture, but with a capability to deform. For a rich source of measurements, we envision using high-resolution high-framerate range data. We are currently in the process of building such a range scanner, as well as developing methods for tracking complex deforming shapes like skin and cloth.

References

[1] Peter Sand, Leonard McMillan, Jovan Popović. Continuous Capture of Skin Deformation. ACM Transactions on Graphics 22(3), 2003, pages 578-586.

[2] Robert W. Sumner, Jovan Popović. Deformation transfer for triangle meshes. ACM Transactions on Graphics 23(3) 2004, pages 399-405.

[3] Daniel Vlasic, Matthew Brand, Hanspeter Pfister, Jovan Popović. Face Transfer with Multilinear Models. ACM Transactions on Graphics 24(3), 2005, pages 426-433.

[4] Robert W. Sumner, Matthias Zwicker, Craig Gotsman, Jovan Popović. Mesh-based Inverse Kinematics. ACM Transactions on Graphics 24(3), 2005, pages 488-495.