Depth Prediction Mismatch Between demo.py and KITTI Benchmark Results

[adamkanjo]

Hello,

First, thank you for the amazing work. I've created a Dockerfile to run the demo.py script, but the results I'm getting differ noticeably from those shown on the KITTI Vision Benchmark website.

I've also downloaded the pretrained models as instructed, so I'm unsure why the depth predictions from demo.py don't match the benchmarked results. Am I missing something, or is there a specific configuration required to reproduce the benchmark performance?

demo.txt
Dockerfile.txt

Thank you in advance for your help.
Best regards,
Adam Kanjo

[Sharpiless]

Thank you for reaching out and for the detailed explanation.

I've just re-ran the code from the Git repository myself and encountered the same issue you're describing. I'm currently checking whether this is due to a change introduced during a recent code refactor. Thank you very much for bringing this to my attention—I'll investigate the root cause and update the code as soon as possible.

[Sharpiless]

Hi Adam,

I found the issue — turns out it was my mistake. While testing some training code after a refactor, I accidentally overwrote the pretrained checkpoint with a version that had only been trained for a few steps.

I've re-uploaded a checkpoint that performs also well. You should be able to use it for inference without issues.

Download it using:

wget https://github.com/Sharpiless/DMD3C/releases/download/pretrain-checkpoints/dmd3c_distillation_depth_anything_v2.pth

Thanks again for catching this! I will update the README later.

[Sharpiless]

Here is the python packages I'm using:
Package Version

---

absl-py 2.1.0
addict 2.4.0
aiofiles 23.2.1
altair 5.4.0
annotated-types 0.7.0
anyio 4.4.0
astor 0.8.1
asttokens 2.4.1
astunparse 1.6.3
attrs 24.2.0
beautifulsoup4 4.12.3
blinker 1.8.2
boto3 1.35.6
botocore 1.35.6
BpOps 0.0.0
Brotli 1.0.9
certifi 2022.12.7
charset-normalizer 2.1.1
click 8.1.7
cmake 3.25.0
coloredlogs 15.0.1
comm 0.2.2
ConfigArgParse 1.7
contourpy 1.3.0
cycler 0.12.1
Cython 3.0.11
dash 2.18.1
dash-core-components 2.0.0
dash-html-components 2.0.0
dash-table 5.0.0
debugpy 1.8.5
decorator 5.1.1
diffusers 0.30.0
diskcache 5.6.3
einops 0.8.0
exceptiongroup 1.2.2
executing 2.0.1
fastapi 0.112.1
fastjsonschema 2.20.0
ffmpy 0.4.0
filelock 3.15.4
fire 0.6.0
Flask 3.0.3
flatbuffers 25.2.10
FlowIO 1.3.0
fonttools 4.57.0
fsspec 2024.6.1
gmpy2 2.2.1
gradio 4.29.0
gradio_client 0.16.1
gradio_imageslider 0.0.20
grpcio 1.65.5
h11 0.14.0
h5py 3.11.0
httpcore 1.0.5
httpx 0.27.0
huggingface-hub 0.29.2
humanfriendly 10.0
hydra-core 1.4.0.dev1
idna 3.4
imageio 2.35.1
imgaug 0.4.0
importlib_metadata 8.4.0
importlib_resources 6.5.2
imutils 0.5.4
iopath 0.1.10
ipykernel 6.29.5
ipython 8.18.1
ipywidgets 8.1.5
itsdangerous 2.2.0
jedi 0.19.1
Jinja2 3.1.4
jmespath 1.0.1
joblib 1.4.2
jsonschema 4.23.0
jsonschema-specifications 2023.12.1
jupyter_client 8.6.2
jupyter_core 5.7.2
jupyterlab_widgets 3.0.13
kiwisolver 1.4.7
kornia 0.8.0
kornia_rs 0.1.8
lazy_loader 0.4
lightning-utilities 0.11.7
lit 15.0.7
llvmlite 0.43.0
lmdb 1.5.1
lxml 5.3.0
Markdown 3.7
markdown-it-py 3.0.0
MarkupSafe 2.1.5
matplotlib 3.9.4
matplotlib-inline 0.1.7
mdurl 0.1.2
mkl_fft 1.3.11
mkl_random 1.2.8
mkl-service 2.4.0
mmcv-full 1.7.2
mmengine 0.10.7
mpmath 1.3.0
narwhals 1.5.2
nbformat 5.10.4
nest-asyncio 1.6.0
networkx 3.2.1
numba 0.60.0
numpy 1.26.0
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.19.3
nvidia-nvjitlink-cu12 12.6.20
nvidia-nvtx-cu12 12.1.105
omegaconf 2.4.0.dev3
onnx 1.16.2
onnxruntime 1.19.2
open3d 0.18.0
opencv-contrib-python 4.10.0.84
opencv-python 4.8.0.76
opt-einsum 3.3.0
orjson 3.10.7
packaging 25.0
paddleocr 2.8.1
paddlepaddle-gpu 2.6.1
pandas 2.2.2
parso 0.8.4
pexpect 4.9.0
pillow 11.1.0
pip 23.3.1
platformdirs 4.2.2
plotly 5.24.1
portalocker 3.1.1
prompt_toolkit 3.0.47
protobuf 3.19.6
psutil 6.0.0
ptyprocess 0.7.0
pure_eval 0.2.3
py-cpuinfo 9.0.0
pyclipper 1.3.0.post5
pycocotools 2.0.8
pydantic 2.8.2
pydantic_core 2.20.1
pydub 0.25.1
Pygments 2.18.0
pyparsing 3.2.3
pyquaternion 0.9.9
PySocks 1.7.1
python-dateutil 2.9.0.post0
python-docx 1.1.2
python-multipart 0.0.9
pytorch3d 0.7.8
pytz 2024.1
PyYAML 6.0.2
pyzmq 26.2.0
rapidfuzz 3.9.6
referencing 0.35.1
regex 2024.11.6
requests 2.32.3
retrying 1.3.4
rich 13.7.1
rpds-py 0.20.0
ruff 0.6.1
s3transfer 0.10.2
safetensors 0.4.4
scikit-image 0.24.0
scikit-learn 1.5.1
scipy 1.13.1
seaborn 0.13.2
semantic-version 2.10.0
setuptools 68.2.2
shapely 2.0.6
shellingham 1.5.4
six 1.17.0
sniffio 1.3.1
soupsieve 2.6
stack-data 0.6.3
starlette 0.38.2
sympy 1.13.2
tenacity 9.0.0
tensorboard 2.17.1
tensorboard-data-server 0.7.2
termcolor 2.4.0
threadpoolctl 3.5.0
tifffile 2024.8.10
timm 0.9.16
tokenizers 0.13.3
tomli 2.2.1
tomlkit 0.12.0
torch 1.13.1+cu117
torchaudio 0.13.1+cu117
torchmetrics 0.11.4
torchvision 0.14.1+cu117
tornado 6.4.1
tqdm 4.66.4
traitlets 5.14.3
transformers 4.29.2
triton 2.0.0
typer 0.12.4
typing_extensions 4.12.2
tzdata 2024.1
ultralytics 8.2.100
ultralytics-thop 2.0.8
urllib3 2.2.3
uvicorn 0.30.6
wcwidth 0.2.13
websockets 11.0.3
Werkzeug 3.0.4
wheel 0.41.2
widgetsnbextension 4.0.13
xarray 2024.7.0
yapf 0.43.0
zipp 3.21.0

[adamkanjo]

Hi @Sharpiless

Thank you a lot for your support. I updated the name of the checkpoint here:

And then it worked fine:

---

I'd like to suggest the following steps to help others quickly test and run the project.
These could be useful to include in the README or setup docs.

1. Clone BP-Net

git clone https://github.com/kakaxi314/BP-Net.git

2. Clone and move DMD3C into BP-Net

git clone https://github.com/Sharpiless/DMD3C.git
cp -r DMD3C/* BP-Net/
rm -rf DMD3C

3. Patch bp_cuda_kernel.cu to fix deprecated .type() usage under folder BP-Net/exts/

sed -i 's/\.\s*type\s*(\s*)\s*/.scalar_type()/g' bp_cuda_kernel.cu

4. Download the pretrained checkpoint into the checkpoints/ folder

wget https://github.com/Sharpiless/DMD3C/releases/download/pretrain-checkpoints/dmd3c_distillation_depth_anything_v2.pth -P BP-Net/checkpoints/

5. Add the following to BP-Net/configs/config.yaml

chpt: dmd3c_distillation_depth_anything_v2.pth

Into

6. Update the model loading code in utils_infer.py

From this

Replace the block with:

if "chpt" in self.cfg:
    self.ddp_log(f'resume CHECKPOINTS')
    save_path = os.path.join('checkpoints')
    print("save_path", save_path)
    cp = torch.load(os.path.join(save_path, self.cfg.chpt), map_location=torch.device('cpu'))
    model.load_state_dict(cp['net'], strict=True)
    self.best_metric_ema = cp['best_metric_ema']
    print("Epoch:", cp["epoch"])
    del cp

7. Download a KITTI sample

Download synced + rectified data and calibration files from:
https://www.cvlibs.net/datasets/kitti/raw_data.php

Extract the files under the datas/:

8. Update the base variable in demo.py to point to your KITTI sample

Example:

base = "/workspace/BP-Net/datas/2011_09_26/2011_09_26_drive_0001_sync"

9. Download the Dockerfile and place it in the project root

📄 Dockerfile.txt

10. Build and run the Docker container

Follow the instructions provided in the Dockerfile comments to set everything up.

11. Run the demo

Once completed, the output files will be saved in the outputs/ folder.

Best regard
Adam Kanjo