Rendering with Style: Combining Traditional and Neural Approaches for High-Quality Face Rendering

Abstract

In this work we propose to combine incomplete, high-quality renderings showing only facial skin with recent methods for neural rendering of faces, in order to automatically and seamlessly create photo-realistic full-head portrait renders from captured data without the need for artist intervention. Our method begins with traditional face rendering, where the skin is rendered with the desired appearance, expression, viewpoint, and illumination. These skin renders are then projected into the latent space of a pre-trained neural network that can generate arbitrary photo-real face images (StyleGAN2). The result is a sequence of realistic face images that match the identity and appearance of the 3D character at the skin level, but is completed naturally with synthesized hair, eyes, inner mouth and surroundings.

Project Page

Bibtex:

@article{10.1145/3478513.3480509,
author = {Chandran, Prashanth and Winberg, Sebastian and Zoss, Gaspard and Riviere, J\'{e}r\'{e}my and Gross, Markus and Gotardo, Paulo and Bradley, Derek},
title = {Rendering with Style: Combining Traditional and Neural Approaches for High-Quality Face Rendering},
year = {2021},
issue_date = {December 2021},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
volume = {40},
number = {6},
issn = {0730-0301},
url = {https://doi.org/10.1145/3478513.3480509},
doi = {10.1145/3478513.3480509},
journal = {ACM Trans. Graph.},
month = {dec},
articleno = {223},
}