Cannot run ZoeDepth in Python 3.10+

[decamun]

As far as I can tell, there is no way to run ZoeDepth in Python 3.10+, because torch v2 introduces a breaking change and v1 cannot run above python 3.9.

My goal is to run this code in Python 3.11 and Pytorch 2.1.0. This is necessary for me because reverting to Python 3.9 is breaking for other parts of my codebase.

I can load the model using the method in #82:

(in model_io.py)

def load_state_dict(model, state_dict):
    """Load state_dict into model, handling DataParallel and DistributedDataParallel. Also checks for "model" key in state_dict.

    DataParallel prefixes state_dict keys with 'module.' when saving.
    If the model is not a DataParallel model but the state_dict is, then prefixes are removed.
    If the model is a DataParallel model but the state_dict is not, then prefixes are added.
    """
    state_dict = state_dict.get('model', state_dict)
    # if model is a DataParallel model, then state_dict keys are prefixed with 'module.'

    do_prefix = isinstance(
        model, (torch.nn.DataParallel, torch.nn.parallel.DistributedDataParallel))
    state = {}
    for k, v in state_dict.items():
        if k.startswith('module.') and not do_prefix:
            k = k[7:]

        if not k.startswith('module.') and do_prefix:
            k = 'module.' + k

        state[k] = v

    model.load_state_dict(state, strict=False) # modified
    for b in model.core.core.pretrained.model.blocks: # modified
        b.drop_path = torch.nn.Identity() # modified
    print("Loaded successfully")
    return model

But I still get the following error when trying to use the model.eval function:

  File "C:\Users\[decamun]/.cache\torch\hub\intel-isl_MiDaS_master\midas\backbones\beit.py", line 47, in _get_rel_pos_bias
    new_sub_table = F.interpolate(old_sub_table, size=(new_height, new_width), mode="bilinear")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\[decamun]\AppData\Local\anaconda3\envs\n15_b\Lib\site-packages\torch\nn\functional.py", line 3924, in interpolate
    raise TypeError(
TypeError: expected size to be one of int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int], but got size with types [<class 'numpy.intc'>, <class 'numpy.intc'>]

#62 fixes this for models constructed locally, but even locally instantiating zoedepthNK in 'infer' downloads the statedict from this repo.

From config_zoedepth_nk.json

"infer": {
        "train_midas": false,
        "pretrained_resource": "url::https://github.com/isl-org/ZoeDepth/releases/download/v1.0/ZoeD_M12_NK.pt",
        "use_pretrained_midas": false,
        "force_keep_ar": true
    },

We need an update to this repo (or a release in an independent repo) that incorporates the fixes in #62

[aayushi7priya]

By your code, the model is loading but it's showing error when i try to get the depth_map prediction. I am getting error in infer_pil function. Were you able to suceesfully get the depth map?

img_path = '/content/IMG_20231012_190249746.jpg' # Replace with your image file path
img = Image.open(img_path)

depth = model_zoe_nk.infer_pil(img)

colored_depth = colorize(depth)
fig, axs = plt.subplots(1,2, figsize=(15,7))
for ax, im, title in zip(axs, [img, colored_depth], ['Input', 'Predicted Depth']):
ax.imshow(im)
ax.axis('off')
ax.set_title(title)

[AnthoneoJ]

As far as I can tell, there is no way to run ZoeDepth in Python 3.10+, because torch v2 introduces a breaking change and v1 cannot run above python 3.9.

My goal is to run this code in Python 3.11 and Pytorch 2.1.0. This is necessary for me because reverting to Python 3.9 is breaking for other parts of my codebase.

I can load the model using the method in #82:

(in model_io.py)

def load_state_dict(model, state_dict):
    """Load state_dict into model, handling DataParallel and DistributedDataParallel. Also checks for "model" key in state_dict.

    DataParallel prefixes state_dict keys with 'module.' when saving.
    If the model is not a DataParallel model but the state_dict is, then prefixes are removed.
    If the model is a DataParallel model but the state_dict is not, then prefixes are added.
    """
    state_dict = state_dict.get('model', state_dict)
    # if model is a DataParallel model, then state_dict keys are prefixed with 'module.'

    do_prefix = isinstance(
        model, (torch.nn.DataParallel, torch.nn.parallel.DistributedDataParallel))
    state = {}
    for k, v in state_dict.items():
        if k.startswith('module.') and not do_prefix:
            k = k[7:]

        if not k.startswith('module.') and do_prefix:
            k = 'module.' + k

        state[k] = v

    model.load_state_dict(state, strict=False) # modified
    for b in model.core.core.pretrained.model.blocks: # modified
        b.drop_path = torch.nn.Identity() # modified
    print("Loaded successfully")
    return model

But I still get the following error when trying to use the model.eval function:

  File "C:\Users\[decamun]/.cache\torch\hub\intel-isl_MiDaS_master\midas\backbones\beit.py", line 47, in _get_rel_pos_bias
    new_sub_table = F.interpolate(old_sub_table, size=(new_height, new_width), mode="bilinear")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\[decamun]\AppData\Local\anaconda3\envs\n15_b\Lib\site-packages\torch\nn\functional.py", line 3924, in interpolate
    raise TypeError(
TypeError: expected size to be one of int or Tuple[int] or Tuple[int, int] or Tuple[int, int, int], but got size with types [<class 'numpy.intc'>, <class 'numpy.intc'>]

#62 fixes this for models constructed locally, but even locally instantiating zoedepthNK in 'infer' downloads the statedict from this repo.

From config_zoedepth_nk.json

"infer": {
        "train_midas": false,
        "pretrained_resource": "url::https://github.com/isl-org/ZoeDepth/releases/download/v1.0/ZoeD_M12_NK.pt",
        "use_pretrained_midas": false,
        "force_keep_ar": true
    },

We need an update to this repo (or a release in an independent repo) that incorporates the fixes in #62

The error indicates that size is currently of numpy.intc type but a native int type is expected. This TypeError checking was introduced in PyTorch v2.1.0 https://github.com/pytorch/pytorch/pull/99243/commits/49434a4af7de0540c52066664c289c59153acfec. Hence the last version that would work with beit.py is v2.0.1.
If you wish to stick to a later version, you'll have to manually edit the beit.py file. The native type can be extracted using .item():
new_sub_table = F.interpolate(old_sub_table, size=(new_height.item(), new_width.item()), mode="bilinear")

[tobiaswuerth]

I had to do both: adjust the beit.py as well as manually cast to int(..) in midas to get it to work

[Erendrgnl]

@AnthoneoJ thanks for help. An extra i changed in sanity.py at 37 as:
torch.hub.help("intel-isl/MiDaS", "DPT_BEiT_L_384", force_reload=False)

Otherwise, it downloads again and overwrites the changes I made in the beit.py code.