DAP-NeRF

Dynamic Appearance Particle Neural Radiance Field
Ancheng Lin, Yusheng Xiang, Jun Li, Mukesh Prasad
Paper (arxiv)

DAP-NeRF introduces particle-based representation to model the motions of visual elements in a dynamic 3D scene. DAP-NeRF consists of superposition of a static field and a dynamic field. The dynamic field is quantised as a collection of appearance particles, which carries the visual information of a small dynamic element in the scene and is equipped with a motion model. All components, including the static field, the visual features and motion models of the particles, are learned from monocular videos without any prior geometric knowledge of the scene.

Notes

TODO: Clean codes thoroughly.

10/6/2024: Codes realeased.

Requirements

pip install -r requirements.txt

Data Preparation

├── data
│   ├── d_nerf
│   │   ├── mutant
│   │   ├── standup 
│   │   ├── ... 
│   ├── vrig
│   ├── NHR
│   ├── ...

For synthetic dataset (d_nerf):
The dataset provided in D-NeRF is used. You can download the dataset from dropbox.

For HyperNeRF dataset (vrig):
The dataset provided in HyperNeRF is used. You can download scenes from Hypernerf Dataset and organize them as Nerfies.

For NHR dataset:
The dataset provided in NHR is used. The processed scenes can be downloaded from here.

Training & Evaluation

For Synthetic or HyperNeRF scenes, run

python run_dap.py --config configs/(dnerf|vrig)/xxx.py --render_test --render_video --eval_psnr --eval_ssim --eval_lpips_vgg

For NHR scenes, run

python run_nhr_dap.py --config configs/nhr/xxx.py --render_test --render_video --eval_psnr --eval_ssim --eval_lpips_vgg

Acknowledgements

This repository is partially based on TiNeuVox and nerfacc. Thanks for their awesome works.