Clarification on downstream applications

[SlamMate]

Accuracy Issue:

I use the matching pairs obtained from Roma and the depth from DepthAnythingv2 as inputs to compute the relative pose between two frames. Then, I apply the relative pose transformation to the absolute pose of the previous frame. However, the accuracy is still not as good as directly using OpenCV's RANSAC with EPnP. The parameters I used are entirely based on your library's madpose/examples/calibrated.py.

Depth Interpolation Query:

I also have a question regarding get_depth: Why not use bilinear interpolation to leverage Roma's subpixel accuracy in matching, but instead choose nearest neighbor matching?

Here is my code:

kpts0, kpts1 = roma_model.to_pixel_coordinates(matches, h1, w1, h2, w2)

# Find a fundamental matrix (or anything else of interest)
kpts0 = kpts0.detach().cpu().numpy().astype(np.float32)   # (N,2)
kpts1 = kpts1.detach().cpu().numpy().astype(np.float32)

depth_map0 = prev_3.depth_est.detach().cpu().numpy()
depth_map1 = viewpoint.depth_est.detach().cpu().numpy()

# K0, K1 (tensor to numpy)
K1 = prev_3.K.cpu().numpy()
K2 = viewpoint.K.cpu().numpy()

# kpts1 = kpts1.cpu().numpy()
# kpts2 = kpts2.cpu().numpy()
# Query the depth priors of the keypoints
depth0 = get_depths(prev_3.original_image, depth_map0, kpts0)
depth1 = get_depths(viewpoint.original_image, depth_map1, kpts1)

# Thresholds for reprojection and epipolar errors
reproj_pix_thres = 8.0
epipolar_pix_thres = 2.0

# Weight for epipolar error
epipolar_weight = 1.0

# RANSAC Options and Estimator Configs
options = madpose.HybridLORansacOptions()
options.min_num_iterations = 1000
options.final_least_squares = True
options.threshold_multiplier = 5.0
options.num_lo_steps = 4
options.squared_inlier_thresholds = [reproj_pix_thres**2, epipolar_pix_thres**2]
options.data_type_weights = [1.0, epipolar_weight]
options.random_seed = 0

est_config = madpose.EstimatorConfig()
est_config.min_depth_constraint = True
est_config.use_shift = True

min0 = float(depth_map0[depth_map0 > 0].min()) if (depth_map0 > 0).any() else 0.1
min1 = float(depth_map1[depth_map1 > 0].min()) if (depth_map1 > 0).any() else 0.1

pose_mad, stats_mad = madpose.HybridEstimatePoseScaleOffset(
    kpts0,
    kpts1,
    depth0,
    depth1,
    [min0, min1],
    K1,
    K2,
    options,
    est_config,
)

R_rel = pose_mad.R()  # 3x3, numpy
t_rel = pose_mad.t()  # (3,), numpy
s_est = pose_mad.scale
o1_est = pose_mad.offset0
o2_est = pose_mad.offset1

T_w2c0 = torch.eye(4, dtype=torch.float32)
T_w2c0[:3, :3] = prev_3.R
T_w2c0[:3, 3] = prev_3.T

# Create 4x4 transformation matrix T_c0->c1
T_c0_2_c1 = torch.eye(4, dtype=torch.float32)
T_c0_2_c1[:3, :3] = torch.from_numpy(R_rel)
T_c0_2_c1[:3, 3] = torch.from_numpy(t_rel)

# Matrix multiplication
T_w2c1 = T_c0_2_c1 @ T_w2c0
# Split back into R and t
new_R = T_w2c1[:3, :3]
new_t = T_w2c1[:3, 3]

[MarkYu98]

Hello, the reason of using nearest neighbor interpolation on depth maps is because depth values can have sudden changes (e.g. edges, boundary of objects) and bilinear interpolation are more likely to not capture these details. However, depending on the specific case, bilinear interpolation could also work well.

For your accuracy issue, possible reason is that Depth-Anything-v2 predicts affine-invariant disparity (inverse-depth), which does not align with our affine (shift-and-scale) modeling on relative depth (we have included some results with using the inverse of DA-v2 predictions in Table 8 in the appendix of the paper for your reference, though technically the inverse of affine-invariant disparity is not affine-invariant relative depth). I would suggest trying a different MDE model that outputs relative depth (or metric depth) instead of inverse depth.

[SlamMate]

Thank u for clarification, in previous communication, I indeed use the DAV2 metric depth large model. But I think the problem is a bit unclear. So I produce a report to show this problem.

Pose Estimation Issue with DAV2 Metric Depth

Overview

I'm using the DAV2 metric depth model in the DAV2 metric depth version for pose estimation on the fr1_desk sequence from the TUM dataset. Despite following the standard procedure, I encounter consistent failures on certain frames while maintaining a high inlier ratio.

Setup

• Dataset: TUM fr1_desk sequence
• Initial Pose: Ground truth poses
• Pose Update: Every two frames using madpose (based on ROMA and DAV2 metric depth)
• Pose Difference Calculation:
  [
  \text{combined_error} = \sqrt{\text{trans}^2 + \text{rot}^2}
  ]
  • prev_total_loss: Difference between previous estimated pose and ground truth
  • after_madpose_total_loss: Difference after madpose estimation
  • num_matches: Number of ROMA matches
  • Inliers: From madpose.HybridEstimatePoseScaleOffset

Observations

1. Consistent Frame Failures:

   • Certain frames consistently fail.
   • High inlier ratios persist even during failures.

2. RANSAC Threshold Adjustment:

   • Lowered reproj_pix_thres to 2.
   • Result: In some failed frames, inlier ratio dropped to ~0.5 or lower.

image

Possible Cause

• Small Baseline & Depth: The dataset has a small baseline and shallow depth, leading the algorithm to use excessively large scales to find an optimal solution.

Suggested Fix

• Introduce Scale Penalty: Adding a penalty for large scale factors might prevent the algorithm from over-scaling.
• Tune RANSAC Threshold: Further adjust the RANSAC threshold to better suit the dataset characteristics.

[SlamMate]

I also checked the scale estimation.