Artist-Friendly Relightable and Animatable Neural Heads

Abstract

An increasingly common approach for creating photorealistic digital avatars is through the use of volumetric neural fields. The original neural radiance field (NeRF) allowed for impressive novel view synthesis of static heads when trained on a set of multi-view images, and follow up methods showed that these neural representations can be extended to dynamic avatars. Recently, new variants also surpassed the usual drawback of baked-in illumination in neural representations, showing that static neural avatars can be relit in any environment. In this work we simultaneously tackle both the motion and illumination problem, proposing a new method for relightable and animatable neural heads. Our method builds on a proven dynamic avatar approach based on a mixture of volumetric primitives, combined with a recently proposed lightweight hardware setup for relightable neural fields, and includes a novel architecture that allows relighting dynamic neural avatars performing unseen expressions in any environment, even with nearfield illumination and viewpoints.

Project Page

Bibtex:

@INPROCEEDINGS {Xu2024,
    author = { Xu, Yingyan and Chandran, Prashanth and Weiss, Sebastian and Gross, Markus and Zoss, Gaspard and Bradley, Derek },
    booktitle = { 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) },
    title = ,
    year = {2024},
    pages = {2457-2467},
    doi = {10.1109/CVPR52733.2024.00238},
    publisher = {IEEE Computer Society},
    address = {Los Alamitos, CA, USA},
    month =June
}