ReliTalk - Official PyTorch Implementation

This repository provides the official PyTorch implementation for the following paper:

ReliTalk: Relightable Talking Portrait Generation from a Single Video
Haonan Qiu, Zhaoxi Chen, Yuming Jiang, Hang Zhou, Wayne Wu, Xiangyu Fan, Lei Yang, and Ziwei Liu

From MMLab@NTU affliated with S-Lab, Nanyang Technological University and SenseTime Research.

[Project Page] | [Paper] | [Demo Video]

Generated Samples

Relighting

Audio Driven Relighting

Datasets

Video Data: HDTF

Getting Started

• Clone this repo: git clone --recursive git@github.com:arthur-qiu/ReliTalk.git
• Create a conda environment conda env create -f environment.yml and activate conda activate IMavatar
• We use libmise to extract 3D meshes, build libmise by running cd code; python setup.py install
• Download FLAME model, choose FLAME 2020 and unzip it, copy 'generic_model.pkl' into ./code/flame/FLAME2020

Preparing Dataset

Prepare the dataset following intructions in ./preprocess/README.md.

Link the dataset folder to ./data/datasets. Link the experiment output folder to ./data/experiments.

Relighting

Train for rough normal

cd code
python scripts/exp_runner.py --conf ./confs/IMavatar_supervised_Obama.conf

Generate rough normal

python scripts/exp_runner.py --conf ./confs/IMavatar_supervised_Obama_test.conf --is_eval --checkpoint [epoch1]

Train for reflectance decomposition

mv ../data/experiments/Obama/IMavatar/Obama_train/eval/Obama_eval/epoch_[epoch1]/normal ../data/datasets/Obama/Obama/Obama_eval/

mv ../data/experiments/Obama/IMavatar/Obama_train/eval/Obama_train/epoch_[epoch1]/normal ../data/datasets/Obama/Obama/Obama_train/

mv ../data/experiments/Obama ../data/experiments/Obama_store

python scripts/exp_runner_relight.py --conf ./confs/IMavatar_supervised_Obama_light.conf

Inference for relighting

python scripts/exp_runner_relight.py --conf ./confs/IMavatar_supervised_Obama_test_light.conf --is_eval --checkpoint [epoch2]

We are organizing the code and will release it soon.

Acknowledgments

This code borrows heavily from IMavatar.