LumiGAN: Relightable 3D GANs | 3DV 2024

ABSTRACT

Unsupervised learning of 3D human faces from unstructured 2D image data is an active research area. While recent works have achieved an impressive level of photorealism, they commonly lack control of lighting, which prevents the generated assets from being deployed in novel environments. To this end, we introduce LumiGAN, an unconditional Generative Adversarial Network (GAN) for 3D human faces with a physically based lighting module that enables relighting under novel illumination at inference time. Unlike prior work, LumiGAN can create realistic shadow effects using an efficient visibility formulation that is learned in a self-supervised manner. LumiGAN generates plausible physical properties for relightable faces, including surface normals, diffuse albedo, and specular tint without any ground truth data. In addition to relightability, we demonstrate significantly improved geometry generation compared to state-of-the-art non-relightable 3D GANs and notably better photorealism than existing relightable GANs.

FILES

• Technical paper (link to arxiv)
• Project website (link)

CITATION

B. Deng, Y. Wang, G. Wetzstein, LumiGAN: Unconditional Generation of Relightable 3D Human Faces, 3DV 2024

@inproceedings{Deng2024:lumigan,
author = {Boyang Deng and Yifan Wang and Gordon Wetzstein},
title = {LumiGAN: Unconditional Generation of Relightable 3D Human Faces},
booktitle = {3DV},
year = {2024}
}

LumiGAN framework

Method Overview. LumiGAN integrates a physically based lighting module to a 3D GAN to generate relightable 3D faces. The StyleGAN-inspired generator generates triplane features from a latent code, which are then mapped to volume density, diffuse albedo, specular tint, and visibility. Given an environment map projected to spherical harmonics coefficients, the physically based lighting module uses these generated properties to render the lit image, particularly accounting for physically plausible shadows. The rendered lit image, transformed from linear color space to sRGB, is passed into a super-resolution block with other features to produce the final high-resolution image.