Realistic 3D facial animation parameters

In this project, a robust, accurate and inexpensive approach to estimate 3D facial motion from multi-view video is proposed, where two mirrors located near one's cheeks can reflect the side views of markers on one's face. Nice properties of mirrored images are utilized to simplify the proposed tracking algorithm significantly, while a Kalman filter is employed to reduce the noise and to predict the occluded markers positions. More than 50 markers on one's face are continuously tracked in video of 30 frames per second. The estimated 3D facial motion data has been practically applied to our facial animation system. In addition, the dataset of facial motion can also be applied to the analysis of co-articulation effects, facial expressions, and audio-visual hybrid recognition system.

Keywords: facial animation, motion capture, motion tracking, 3D position estimation, mirrored images, stereo computer vision, Kalman filter

Demo videos of synthesized results

Viseme "O" on the generic model (MPEG1, about 7.2MB)

¡@

Acknowledgement: We would like to appreciate Nathalie. P. Valles, Yves Lapire, Dominique Fohr, Michel Pitermann and other researchers of speech group of Loria, France. They help our experiment in French and provide knowledge of French visemes. Especially, we would like to thank Michel Pitermann, since we use his face as one of the face model exhibited in this paper and he also gave us a lot of valuable suggestions for the paper. Besides, we would also like to thank Professor Mary Flanagan of University of Oregon. She helped us revise the language style.

Reference:

  1. I-Chen Lin, Jeng-Sheng Yeh, Ming Ouhyoung, "Extracting 3D facial animation parameters from multiview video clips ," IEEE Computer Graphics and Applications, vol. 22, no. 6, pp. 72-80, Nov.-Dec. 2002. (SCI)

  2. I-Chen Lin, Jeng-Sheng Yeh, Ming Ouhyoung, "Realistic 3D Facial Animation Parameters from Mirror-Reflected Multi-View Video", Proc. Computer Animation 2001 (CA 2001)(IEEE ISBN 0-7803-7237-9), IEEE Computer Society, pp. 2-11, Seoul, Korea, Nov. 2001.

Authors' note:

In March 6, 2003, one of the author, I-chen Lin, found that "the part of the proposed [mirror orientation estimation method]" is equivalent to the [epipole estimation with a symmetry plane constraint] proposed by D.Q. Huynh, "Affine Reconstruction from Monocular Vision in the Presence of A Symmetry plane", Proc. ICCV 1999 pp. 476-482. Eventhough, the two works took different points of views in the beginning.

Dr. Huynh's work focused on the problem solving of 3D position estimation in the situation of a symmetry plane and discussed the advantage of non-linear computation. On the other hand, our work not only estimate 3D positions but also take advantage of the perfectly-synchornized property between multiple mirrored views to track 3D facial motion. Furthermore, we did three computer simulations to manifest the outstanding benefits of 3D position and motion estimation from mirror-reflected video clips compared to traditional two-view algorithms; we also discussed the advantages and disadvantages between the proposed method and two view approaches.

Go back to "I-Chen's project webpage (English)"

¡@