Skip to content

VANeRF: 3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands

Notifications You must be signed in to change notification settings

XuanHuang0/VANeRF

Repository files navigation

VANeRF: 3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands (AAAI 2024)

This is an official implementation of "3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands".

Installation

  1. Please install python dependencies specified in requirements.txt:

    conda create -n vanerf python=3.9
    conda activate vanerf
    pip install -r requirements.txt
    
  2. Register and download MANO data. Put MANO_LEFT.pkl and MANO_RIGHT.pkl in folder $ROOT/smplx/models/mano.

Data preparation

  1. Download InterHand2.6M dataset and unzip it. (Noted: we used the v1.0_5fps version and H+M subset) The directory structure of $ROOT/InterHand2.6M is expected as follows:

    InterHand2.6M    
    │   ├── annotations 
    │   │   ├── skeleton.txt
    │   │   ├── subject.txt  
    │   │   ├── test    
    │   │   ├── train  
    │   │   └── val  
    │   └── images   
    │       ├── test 
    │       ├── train
    │       └── val 
    
  2. Process the dataset by :

    python data_process/dataset_process.py
    

Evaluation

  1. Download the pretrained model and put it to $ROOT/EXPERIMENTS/vanerf/ckpts/model.ckpt.

  2. Run evaluation:

    #small view variation
    python train.py --config ./configs/vanerf.json --run_val --model_ckpt ./EXPERIMENTS/vanerf/ckpts/model.ckpt
    #big view variation
    python train.py --config ./configs/vanerf_bvv.json --run_val --model_ckpt ./EXPERIMENTS/vanerf/ckpts/model.ckpt
    

    Results will be stored in folder $ROOT/EXPERIMENTS/vanerf/.

  3. Visualize the dynamic results:

    python render_dynamic.py --config ./configs/vanerf.json --model_ckpt ./EXPERIMENTS/vanerf/ckpts/model.ckpt
    

Training on InterHand2.6M dataset

Execute train.py script to train the model on the InterHand2.6M dataset.

python train.py --config ./configs/vanerf.json --num_gpus 4

The output model would be store in $ROOT//EXPERIMENTS/vanerf/ckpts.

Citation

If you find our code or paper useful, please consider citing:

@article{huang20243d,
  title={3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands},
  author={Huang, Xuan and Li, Hanhui and Yang, Zejun and Wang, Zhisheng and Liang, Xiaodan},
  journal={arXiv preprint arXiv:2401.00979},
  year={2024}
}

About

VANeRF: 3D Visibility-aware Generalizable Neural Radiance Fields for Interacting Hands

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages