Face2Face: Real-time Face Capture and Reenactment of RGB Videos

We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion. To this end, we first address the under-constrained problem of facial identity recovery from monocular video by non-rigid model-based bundling. At run time, we track facial expressions of both source and target video using a dense photometric consistency measure. Reenactment is then achieved by fast and efficient deformation transfer between source and target. The mouth interior that best matches the re-targeted expression is retrieved from the target sequence and warped to produce an accurate fit. Finally, we convincingly re-render the synthesized target face on top of the corresponding video stream such that it seamlessly blends with the real-world illumination. We demonstrate our method in a live setup, where Youtube videos are reenacted in real time.

IMPORTANT NOTE: This demo video is purely research-focused and we would like to clarify the goals and intent of our work. Our aim is to demonstrate the capabilities of modern computer vision and graphics technology, and convey it in an approachable and fun way. We want to emphasize that computer-generated videos have been part in feature-film movies for over 30 years. Virtually every high-end movie production contains a significant percentage of synthetically-generated content (from Lord of the Rings to Benjamin Button). These results are hard to distinguish from reality and it often goes unnoticed that the content is not real. The novelty and contribution of our work is that we can edit pre-recorded videos in real-time on a commodity PC. Please also note that our efforts include the detection of edits in video footage in order to verify a clip’s authenticity. Hopefully, you enjoy watching our video, and we hope to provide a positive takeaway 🙂

Motivation

Our primary goal is to create a mathematical model of our world. Such a model enables computers to reconstruct, understand, and interact with it. Computer Vision tries to obtain this model from image data. The reconstruction of the surrounding is very important nowadays. New technologies like Virtual Reality or Augmented Reality rely on such data. The better the quality of the representation is, the better these new technologies will work. Beside scanning solid objects with e.g. a Kinect depth sensor, we also concentrate our work on the harder problem of reconstructing deformable objects in uncontrolled environments. The face tracking problem belongs to that class. Most existing real-time face trackers are based on sparse features and thus capture only a coarse face model. Our approach tries to use all available information in the captured input, i.e. every pixel. That’s why it is called a dense face tracker. Our method follows the principle of analysis-by-synthesis. Thus, we determine the parameters of our face model by minimizing the error between the input image and a synthesized face image. Our resulting synthesized model is so close to the input that it is hard to distinguish between the synthesized and the real face.

Use Cases

There are many possible use cases for our dense face tracking technique. The most prominent use case is the usage in post-production systems in the film industry. There are already techniques that transfer the expressions of a human to a virtual avatar. This is also possible with our technique, despite the fact that we can even transfer the expressions / mouth movement to another human actor without the need of special hardware and long scanning procedures. Thus, we can easily alter the face of an actor after capturing. E.g. one can alter the lighting, the skin or modify the expression. We also think that our technique paves the way to live simultan dubbing in a teleconferencing scenario. A similar field where such a dense face tracking could be beneficial, is the game industry. Here, the motion of a player can be transferred to a virtual avatar of the same person in an online multiplayer game, resulting in a more realistic output.
Our project stems originally from another important field, i.e. medical research. There we analysed the healing of patients that suffer from a cleft lip and palate disorder. Head tracking is also important for other medical purposes, i.e. tracking during surgeries etc.. Beside that, our technique can be adapted to track other body parts or organs. Which is one of the most challenging tasks in modern medicine and is for example needed to damage as less as possible healthy matter during a therapy. Also many psychologist are interested in our system to be used to treat people that suffer from psychical diseases. There is also social psychological research that analysis the bias of which person is more trustworthy.
Fraud detection: As we reconstruct the face parameters as well as the lighting conditions, our approach can be used to detect inconsistencies. For example the expressions of a person and also the transition between them are unique. Thus, a fraud can be detected by analyzing the tracked expressions in a video sequence and comparing them to a reference video sequence. Which is similar to the analysis of handwritten text (graphoanalysis). In the field of Man-Machine Interaction detection and tracking of movements is the key-component. Thus, accurate tracking pushes the development of these applications.