Clarification Needed on Using `train_gaussian.py` for Scene Generation

[aruiplex]

Hi!

I am following the README instructions to prepare the training data, and I encountered some confusion while attempting to proceed with the train_gaussian.py script. Below is a detailed breakdown of the steps I’ve followed so far:

Steps Followed:

1. Rendered Images Creation:
   I used the Stanford Shapenet Renderer to generate rendered images for a single class (e.g., "chair") in the ShapeNetCore_renders directory. The structure of my directory for the "chair" class looks like this:

   ShapeNetCore_renders/
      └── chair/
          ├── 1a6f615e8b1b5ae4dbbc9440457e303e/
          │   └── models/
          │       ├── models_r_000.png
          │       ├── models_r_000_albedo0001.png
          │       ├── models_r_000_depth0001.png
          │       ├── models_r_000_normal0001.png
          │       ├── models_r_012.png
          │       └── ... # other different angle images
          ├── 1a8bbf2994788e2743e99e0cae970928/
          ├──1a74a83fa6d24b3cacd67ce2c72c02e/
          └── ... # other objects
   

2. Point Cloud Generation:
   I then ran the process_data/sample_points.py script to generate point clouds for all objects in the ShapeNetCore directory. For a specific object, the directory structure now looks like this:

   ShapeNetCore/03001627/1a6f615e8b1b5ae4dbbc9440457e303e/models/
   ├── model_normalized.json
   ├── model_normalized.mtl
   ├── model_normalized.obj
   ├── model_normalized.solid.binvox
   ├── model_normalized.surface.binvox
   └── points3d.ply  # Generated point cloud file
   

Issue:

I am now trying to use the train_gaussian.py script to generate a Gaussian scene, but I am unclear on how to proceed with this script. Specifically:

• How do I structure the input data for this script? Should I use the rendered images, the point cloud data, or both?
• Are there any specific configurations or pre-processing steps required before running train_gaussian.py?
• What should the output of the script look like (e.g., file format, directory structure)?
• Could you provide an example or clarification on how to execute the script for a single class (e.g., "chair")?

Any guidance or examples would be greatly appreciated!

Thank you for your help!

[aruiplex]

@weiqi-zhang @junshengzhou Please explain the dataset preparation steps to reproduce your work. Thanks.

[weiqi-zhang]

Hi, @aruiplex . Thank you for your attention to our work! The input data required by train_gaussian.py consists of three key components: Rendered Images, Point Cloud Data and Camera Pose Information(you should modify the rendering code to save transforms.json). To run the train_gaussian.py script, use the following command: python train_gaussian.py -s <path_to_datafolder>. The output of the script is a 3DGS, which is represented as a .ply file.

[aruiplex]

Hi! Thank you for your response, could you please share the render script to reproduce the dataset generation process? I want to reproduce to train this model. Or could you share the dataset to train the model? Thank you for your contribution to this community.

[zhuhu00]

Hi, @aruiplex . Thank you for your attention to our work! The input data required by train_gaussian.py consists of three key components: Rendered Images, Point Cloud Data and Camera Pose Information(you should modify the rendering code to save transforms.json). To run the train_gaussian.py script, use the following command: python train_gaussian.py -s <path_to_datafolder>. The output of the script is a 3DGS, which is represented as a .ply file.

Hi, Can you provide some more detailed answers, I have also rendered the png, and sampled the ply file, but this is in two folders, do I need to merge them together? Or how to run it, and the transform.json file, can you provide a solution on how to modify it, or can you update the readme. @weiqi-zhang @junshengzhou

[aruiplex]

Hi! Please let us reproduce and check your work @weiqi-zhang @junshengzhou.

[weiqi-zhang]

We have already uploaded the rendering code. Please use the provided code to render the mesh.

[aruiplex]

Hi! I used the script you provided, here is the error in the train_gaussian.py.

python train_gaussian.py -s /data/datasets/ShapeNetCore_3dgs/airplane/1a04e3eab45ca15dd86060f189eb133 
Optimizing 
Output folder: ./output/f95bfd7c-6 [02/01 21:46:07]
Tensorboard not available: not logging progress [02/01 21:46:07]
Found transforms_train.json file, assuming Blender data set! [02/01 21:46:07]
Reading Training Transforms [02/01 21:46:07]
Reading Test Transforms [02/01 21:46:10]
Loading Training Cameras [02/01 21:46:14]
Loading Test Cameras [02/01 21:46:15]
Number of points at initialisation :  100000 [02/01 21:46:15]
Training progress:   2%|███▍                                                                                                                                                                           | 590/30000 [00:02<01:58, 248.17it/s, Loss=0.1653027]Traceback (most recent call last):
  File "/data/Hypothesis/proposition/DiffGS/process_data/train_gaussian.py", line 228, in <module>
    training(lp.extract(args), op.extract(args), pp.extract(args), args.test_iterations, args.save_iterations, args.checkpoint_iterations, args.start_checkpoint, args.debug_from)
  File "/data/Hypothesis/proposition/DiffGS/process_data/train_gaussian.py", line 127, in training
    gaussians.densify_and_prune(opt.densify_grad_threshold, 0.005, scene.cameras_extent, size_threshold)
  File "/data/Hypothesis/proposition/DiffGS/process_data/scene/gaussian_model.py", line 395, in densify_and_prune
    self.densify_and_clone(grads, max_grad, extent)
  File "/data/Hypothesis/proposition/DiffGS/process_data/scene/gaussian_model.py", line 389, in densify_and_clone
    self.densification_postfix(new_xyz, new_features_dc, new_features_rest, new_opacities, new_scaling, new_rotation)
  File "/data/Hypothesis/proposition/DiffGS/process_data/scene/gaussian_model.py", line 340, in densification_postfix
    self._xyz = optimizable_tensors["xyz"]
                ~~~~~~~~~~~~~~~~~~~^^^^^^^
KeyError: 'xyz'
Training progress:   2%|███▌                                                                                                                                                                           | 600/30000 [00:02<02:09, 226.27it/s, Loss=0.1653027]

Please provide the complete training script and dataset script. @weiqi-zhang @junshengzhou

[aruiplex]

Why are you comment the line 156 and line 159 in the gaussian_model.py? @weiqi-zhang @junshengzhou

[aruiplex]

Also, the new_area.json in the convert_data.py is not classified. How can I get the new_area.json?

[CNGaoWenbo]

Same problem, did you guys solve this problem? thank you

[andrewsilver1997]

I don't think new_area.json is required here. I ran convert_data.py by commenting out new_area.json related code snippet.