🚀HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork

NeurIPS 2023

Bipasha Sen* ¹ Gaurav Singh* ¹, Aditya Agarwal* ¹, Rohith Agaram ¹, Madhava Krishna ¹, Srinath Sridhar ²

*denotes equal contribution, ¹ International Institute of Information Technology Hyderabad, ² Brown University

applications_teaser.mp4

This is the official implementation of the paper "HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork" accepted at NeurIPS 2023

👉 TODO

• Code Release
  • Training Code
  • Architecture modules, renderer, Meta MRHE
  • Pretrained Compression Checkpoint
• ...

CREATING THE ENVIRONMENT

Please follow the steps outlined in torch-ngp repository for creating the environment upto and including the Build extension subheading.

Note: Please build the extensions using the source code on this repository.

Dataset

Download the ABO Dataset. We use the images and the transforms from abo-benchmark-material.tar and the metadata file abo-listings.tar for training. Place them in a directory structure as follows:

dataset_root
├── ABO_rendered
│   ├── B00EUL2B16
│   ├── B00IFHPVEU
│   ...
│
└── ABO_listings
  └── listings
      └── metadata
          ├── listings_0.json.gz
          ...
          └── listings_f.json.gz

Training

To train a model on the ABO Chair dataset run the following command:

CUDA_VISIBLE_DEVICES=0 python main_nerf.py --path <dataset_root> --workspace <workspace dir> --bound 1.0 --scale 0.8 --dt_gamma 0 --class_choice CHAIR --load_ckpt

Compression Demo

Download the pretrained compression checkpoint from here for the CHAIR category and place it in your workspace dir as follows:

<Workspace dir>
└── checkpoints
    └── ngp_ep<>.pth

To render a specific NeRF from the codebook, run the following command:

CUDA_VISIBLE_DEVICES=0 python main_nerf.py --workspace <Workspace dir> --bound 1.0 --scale 0.6 --dt_gamma 0 --class_choice CHAIR --load_ckpt --test --test_index <index of codebook (max index 1037)>

👏 Acknowledgement

Some parts of the code are inspired and borrowed from torch-ngp (which we use as our backbone) and INR-V. We thank the authors for providing the source code.

📜 BibTeX

If you find HyP-NeRF useful in your work, consider citing us.

@article{hypnerf2023,
  title={HyP-NeRF: Learning Improved NeRF Priors using a HyperNetwork},
  author={Sen, Bipasha and Singh, Gaurav and Agarwal, Aditya and Agaram, Rohith and Krishna, K Madhava and Sridhar, Srinath},
  journal={NeurIPS},
  year={2023}
}