Error in the process of inference

[lavenda-zhou]

Hello, author. I have the following problems in the process of inference.
reassemble ...

python ./tools/merge_patches.py ms_rcnn_r50.val.segm.json data/car-coco/annotations/instances_val2017.json ms_rcnn_r40.val.refined.json/refined.pkl ms_rcnn_r40.val.refined.json/patches/detail_dir/val ms_rcnn_r40.val.refined.json/refined.json
loading annotations into memory...
Done (t=0.00s)
creating index...
index created!
0%| | 0/137 [00:00<?, ?it/s]
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "./tools/merge_patches.py", line 38, in run_inst
patch_mask = results[pid]
KeyError: 16
"""
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "./tools/merge_patches.py", line 110, in
start()
File "./tools/merge_patches.py", line 80, in start
for r in p.imap_unordered(run_inst, enumerate(dt)):
File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/multiprocessing/pool.py", line 748, in next
raise value
KeyError: 16
I look forward to your reply, which is very important to me. thank you!

[chenhang98]

This shows that the patch with id 16 is not included in refined.pkl. You may check if the refined.pkl is consistent with patches/detail_dir/val.

[lavenda-zhou]

Thank you for your reply. I got the refined.json. Can I visualize the result? I want to see the final merged pircture.

[chenhang98]

You can refer to #15 for visualization

[lavenda-zhou]

Thanks. I will look into it. Sorry to bother you again. I am still confused about the inference_coco.sh. The first step: python $BPR_ROOT/tools/split_patches.py . Through this step, I got the maskrcnn_r50.val.refined.json/patches. Is the result of it same with the maskrcnn_r50/patches that come from following command ：the IOU_THRESH=0.15
sh tools/prepare_dataset_coco.sh
mask_rcnn_r50.train.segm.json
mask_rcnn_r50.val.segm.json
maskrcnn_r50
70000
Why do you run split_patches.py again in the inference part? Thank you for your patience in answering my questions earlier

[chenhang98]

The difference between them is that the former only process the validation set and the latter also process the training set.
For the validation set, both do the same thing.

[lavenda-zhou]

Thank you for your nice reply. I got it. Does it mean that the IOU_THRESH need to be the same in these two spliting process? I found if I use the different IOU_THRESH. The number of patches is not the same. I got 1899 patches in the maskrcnn_r50/patches/mask_dir/val and 2488 patches in the maskrcnn_r50.val.refined.json/patches/mask_dir/val. The refined.pkl is not consistent with the maskrcnn_r50.val.refined.json/patches/detail_dir/val because the refined.pkl only uses 1899 patches. Therefore, I met the first question that I asked you. Does the different IOU_THRESH cause this problem? And I have another question. How can I get a series of AP like AP50 AP75 APS APM APL after I got the merged picture. Thank you in advance.

[chenhang98]

The IOU_THRESHs of training and validation sets don't need to be the same.
But the same validation set should the same throughout the inference.
E.g. if you generated the refined.pkl on maskrcnn_r50/patches/mask_dir/val, then you need to use the same path for merge_patches.py.

For AP evaluation, you can refer #24.

[lavenda-zhou]

Hello, author.I completed the whole process with your help. Thank you. Then I add some pictures in my dataset. Tha problems occurs in the first step. When I run the prepare_dataset_coco.sh, the problem is followed.

(open-mmlab) mint@mint-B460HD3:~/BPR-main$ IOU_THRESH=0.15 sh tools/prepare_dataset_coco.sh ms_rcnn_r50.train.segm.json ms_rcnn_r50.val.segm.json ms_rcnn_r50

• GREEN=\033[0;32m
• END=\033[0m\n
• mkdir ms_rcnn_r50
  mkdir: 无法创建目录"ms_rcnn_r50": 文件已存在
• printf \033[0;32mbuild training set ...\033[0m\n
  build training set ...
• python ././tools/filter.py ms_rcnn_r50.train.segm.json data/car-all-coco/annotations/instances_train2017.json ms_rcnn_r50/coarse_train_filtered.json
  loading annotations into memory...
  Done (t=0.04s)
  creating index...
  index created!
  0%| | 4/1558 [00:00<00:08, 185.00it/s]
  multiprocessing.pool.RemoteTraceback:
  """
  Traceback (most recent call last):
  File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/multiprocessing/pool.py", line 121, in worker
  result = (True, func(*args, **kwds))
  File "././tools/filter.py", line 31, in max_iou
  ious = [cal_iou(maskdt, _) for _ in masks]
  File "././tools/filter.py", line 31, in
  ious = [cal_iou(maskdt, _) for _ in masks]
  File "././tools/filter.py", line 14, in cal_iou
  si = np.sum(mask1 & mask2)
  ValueError: operands could not be broadcast together with shapes (942,1674) (1080,1920)
  """

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "././tools/filter.py", line 65, in
Fire(main)
File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/fire/core.py", line 141, in Fire
component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire
target=component.name)
File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/fire/core.py", line 681, in _CallAndUpdateTrace
component = fn(*varargs, **kwargs)
File "././tools/filter.py", line 59, in main
dt_filtered = filter_iou(dt, thresh)
File "././tools/filter.py", line 40, in filter_iou
for iou, bbox in p.imap(max_iou, bboxs):
File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/multiprocessing/pool.py", line 748, in next
raise value
ValueError: operands could not be broadcast together with shapes (942,1674) (1080,1920)

• python ././tools/split_patches.py ms_rcnn_r50/coarse_train_filtered.json data/car-all-coco/annotations/instances_train2017.json data/car-all-coco/train2017 ms_rcnn_r50/patches --iou-thresh 0.15 --mode train --sample-inst -1
  mkdir: 无法创建目录"ms_rcnn_r50/patches/img_dir": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/ann_dir": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/mask_dir": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/detail_dir": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/img_dir/train": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/ann_dir/train": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/mask_dir/train": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/detail_dir/train": 文件已存在
  loading annotations into memory...
  Done (t=0.04s)
  creating index...
  index created!
  Traceback (most recent call last):
  File "././tools/split_patches.py", line 291, in
  start()
  File "././tools/split_patches.py", line 239, in start
  dt = json.load(open(args.dt_json))
  FileNotFoundError: [Errno 2] No such file or directory: 'ms_rcnn_r50/coarse_train_filtered.json'
• printf \033[0;32mbuild validation set ...\033[0m\n
  build validation set ...
• python ././tools/split_patches.py ms_rcnn_r50.val.segm.json data/car-all-coco/annotations/instances_val2017.json data/car-all-coco/val2017 ms_rcnn_r50/patches --iou-thresh 0.15 --mode val
  mkdir: 无法创建目录"ms_rcnn_r50/patches/img_dir": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/ann_dir": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/mask_dir": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/detail_dir": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/img_dir/val": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/ann_dir/val": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/mask_dir/val": 文件已存在
  mkdir: 无法创建目录"ms_rcnn_r50/patches/detail_dir/val": 文件已存在
  loading annotations into memory...
  Done (t=0.01s)
  creating index...
  index created!
  0%| | 0/402 [00:00<?, ?it/s]
  multiprocessing.pool.RemoteTraceback:
  """
  Traceback (most recent call last):
  File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/multiprocessing/pool.py", line 121, in worker
  result = (True, func(*args, **kwds))
  File "././tools/split_patches.py", line 191, in run_inst
  maskdt = coco.annToMask(inst)
  File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/pycocotools/coco.py", line 442, in annToMask
  rle = self.annToRLE(ann)
  File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/pycocotools/coco.py", line 421, in annToRLE
  t = self.imgs[ann['image_id']]
  KeyError: 1
  """

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "././tools/split_patches.py", line 291, in
start()
File "././tools/split_patches.py", line 248, in start
for _ in p.imap_unordered(run_inst, enumerate(dt)):
File "/home/mint/anaconda3/envs/open-mmlab/lib/python3.7/multiprocessing/pool.py", line 748, in next
raise value
KeyError: 1

I don't know what is wrong with my dataset. I look forward to your reply, which is very important to me. Thank you in advance！

[Lopside1]

I have faced a similar issue and I think I have resolved it. I am assuming you have used Mask RCNN as your segmenter(?). When calling the detect in the inspect_coco_model.ipynb modules the notebooks call model.load_image_gt() which in turn calls utils.resize_image() before running through the detector. This reshapes the image size and does not undo it again later. So there is a chance that your output binary masks are set to the shape defined here whilst the images in your original instance set have the old size.

So when you are detecting the image you do not want to call utils.resize_image() beforehand which model.load_image_gt() does. I made a small change to this function as below by introducing a resize flag in model.load_image_gt(). The utils.resize_image() function accepts a mode="none" argument which will extract the important meta data without resizing the image. You may want to visualise your instances after performing this change to check the masks are correct still.

Note that resize_image is also called in model.detect() but this transform is reversed after the detection so you want to leave this unchanged. This is important here because Mask RCNN can only accept fixed size images across samples.

Also note if you have built the Mask RCNN with the setup.py you will need to re-run this so your script picks up the change to the model lib.

def load_image_gt(dataset, config, image_id, augment=False, augmentation=None,
                  use_mini_mask=False, resize=True):

    if resize == False:
        mode = "none"
    else:
        mode = config.IMAGE_RESIZE_MODE

    image = dataset.load_image(image_id)
    mask, class_ids = dataset.load_mask(image_id)
    original_shape = image.shape
    image, window, scale, padding, crop = utils.resize_image(
        image,
        min_dim=config.IMAGE_MIN_DIM,
        min_scale=config.IMAGE_MIN_SCALE,
        max_dim=config.IMAGE_MAX_DIM,
        mode=mode)
    mask = utils.resize_mask(mask, scale, padding, crop)