Get results different from the paper

[bluestyle97]

Hi, I'm playing with the code and observing that some results are different from the original paper. For example:

"A bag full of trash sitting on a old park bench" (paper/exp_vit16.py, 15000 steps):

"A bag full of trash sitting on a old park bench" (paper/imp_vit16.py, 15000 steps):

while in the paper, the results are (Figure 6):

"American muscle car palms and moon synthwave." (paper/exp_vit16.py, 15000 steps):

"American muscle car palms and moon synthwave." (paper/exp_imp16.py, 15000 steps):

while in the paper, the result is (Figure 1):

I did not change the config file nor the random seed (set as default 777). Do you know how this occurs? How can I obtain exactly the same results as in the paper?

[hanhung]

Hello, so the main issue is the grid_sample function in PyTorch. When using CUDA as the backend it is non-deterministic even when the same seed is used (this is mentioned in the docs: https://pytorch.org/docs/stable/generated/torch.nn.functional.grid_sample.html). This means that even if the same seed is set for two experiments you will not get the same results since we use the grid_sample function.

As for the prompts in the paper:

1. For the prompt: "A bag full of trash sitting on a old park bench". The training resolution is actually lower (168*168) for these figures in the paper and the scheduling is reduced to 10k iterations. I'll upload these missing configurations to the configs.
2. For the prompt: "American muscle car palms and moon synthwave.". It is trained using the OpenCLIP ViT-B/32 model with the 'laion2b_e16' checkpoint. You can change the guidance model using the instructions in the readme.

Let me know if you have any other questions.
Thanks.

[bluestyle97]

Thanks for your reply! I will try changing the guidance model as your suggested.

[bluestyle97]

By the way, have you tried using an explicit density grid + an implicit color network? I'm just curious about this because the original DVGO paper adopts this architecture. Why did you choose to use either explicit density+explicit rgb or implicit density + implicit rgb?

[hanhung]

Earlier on in the project we did try using the base DVGO model with CLIP guidance. However, we have not tried with the newest implemented features in the paper. We mainly chose this setting as we wanted to compare a purely explicit voxel grid approach with implicit voxel grids which follows the setting of the original NeRF and Mip-NeRF used in Dream Fields. From our early experiments with the base DVGO, there are also density artifacts in the generated object as in the explicit model in the paper. I think maybe an implicit density network + rgb color voxel grid might actually make more sense, but have yet to try it out.

[bluestyle97]

OK, thanks a lot!