ReliTalk - Official PyTorch Implementation

This repository provides the official PyTorch implementation for the following paper:

ReliTalk: Relightable Talking Portrait Generation from a Single Video
Haonan Qiu, Zhaoxi Chen, Yuming Jiang, Hang Zhou, Wayne Wu, Xiangyu Fan, Lei Yang, and Ziwei Liu

From MMLab@NTU affliated with S-Lab, Nanyang Technological University and SenseTime Research.

[Project Page] | [Paper] | [Demo Video]

Datasets

Video Data: HDTF

Getting Started

• Clone this repo: git clone --recursive git@github.com:arthur-qiu/ReliTalk.git
• Create a conda environment conda env create -f environment.yml and activate conda activate IMavatar
• We use libmise to extract 3D meshes, build libmise by running cd code; python setup.py install
• Download FLAME model, choose FLAME 2020 and unzip it, copy 'generic_model.pkl' into ./code/flame/FLAME2020

Preparing Dataset

Prepare the dataset following intructions in ./preprocess/README.md.

Link the dataset folder to ./data/datasets. Link the experiment output folder to ./data/experiments.

Training

Inference

We are organizing the code and will release it soon.

Acknowledgments

This code borrows heavily from IMavatar.