Issue: Mask shaping issue during data loading

[jonas4climate]

I am currently working on my final year project further exploring Machine Learning applied to the notation assembly process and have encountered an issue when attempting to run the train.py file from your project. The downloading of the data did not work with the script by default and I had to modify the requirements.txt file to include all dependencies needed but then did get it to run. However at 13% into the loading process I now get this issue:

> python munglinker/train.py
BaseConvnet(
  (cnn): Sequential(
    (conv0): Conv2d(3, 8, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm0): BatchNorm2d(8, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu0): ReLU(inplace=True)
    (pooling0): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (conv1): Conv2d(8, 16, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu1): ReLU(inplace=True)
    (pooling1): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (conv2): Conv2d(16, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm2): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu2): ReLU(inplace=True)
    (pooling2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (conv3): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm3): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu3): ReLU(inplace=True)
    (pooling3): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
    (conv4): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1))
    (batchnorm4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
    (relu4): ReLU(inplace=True)
    (pooling4): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (fully_connected): Linear(in_features=4096, out_features=1, bias=True)
  (output_activation): Sigmoid()
)
----------------------------------------------------------------
        Layer (type)               Output Shape         Param #
================================================================
            Conv2d-1          [-1, 8, 256, 512]             224
       BatchNorm2d-2          [-1, 8, 256, 512]              16
              ReLU-3          [-1, 8, 256, 512]               0
         MaxPool2d-4          [-1, 8, 128, 256]               0
            Conv2d-5         [-1, 16, 128, 256]           1,168
       BatchNorm2d-6         [-1, 16, 128, 256]              32
              ReLU-7         [-1, 16, 128, 256]               0
         MaxPool2d-8          [-1, 16, 64, 128]               0
            Conv2d-9          [-1, 32, 64, 128]           4,640
      BatchNorm2d-10          [-1, 32, 64, 128]              64
             ReLU-11          [-1, 32, 64, 128]               0
        MaxPool2d-12           [-1, 32, 32, 64]               0
           Conv2d-13           [-1, 32, 32, 64]           9,248
      BatchNorm2d-14           [-1, 32, 32, 64]              64
             ReLU-15           [-1, 32, 32, 64]               0
        MaxPool2d-16           [-1, 32, 16, 32]               0
           Conv2d-17           [-1, 32, 16, 32]           9,248
      BatchNorm2d-18           [-1, 32, 16, 32]              64
             ReLU-19           [-1, 32, 16, 32]               0
        MaxPool2d-20            [-1, 32, 8, 16]               0
           Linear-21                    [-1, 1]           4,097
          Sigmoid-22                    [-1, 1]               0
================================================================
Total params: 28,865
Trainable params: 28,865
Non-trainable params: 0
----------------------------------------------------------------
Input size (MB): 1.50
Forward/backward pass size (MB): 47.53
Params size (MB): 0.11
Estimated Total Size (MB): 49.14
----------------------------------------------------------------
Training Strategy E2EOMR_mob_split_muscima_bboxes_2020-11-23-05-33-08:
  Training for 100 epochs with a batch-size of 100.
  Optimizing with <class 'torch.optim.adam.Adam'>, starting at a Learning rate of 0.001
  Loss is computed by BCELoss()
  Early stopping after 10 epochs without improvement.
  Checkpointing every 1 epochs into models/default_model.tsd.ckpt.
  Saving best model into models/default_model.tsd.
  Validating by best validation loss.
  After early stopping, refining for max 10 epochs with patience of 2 epochs without improvement and a learning rate reduction factor of 0.2.

/Users/jonasschafer/CS/Uni/Year3/Final Project/MuNG-generation/munglinker/data_pool.py:346: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  split = yaml.load(hdl)
/Users/jonasschafer/CS/Uni/Year3/Final Project/MuNG-generation/munglinker/data_pool.py:352: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(hdl)
Loading training data...
Loading mung/image pairs from disk:  13%|█████▉                                       | 11/84 [00:14<01:38,  1.35s/it]
Traceback (most recent call last):
  File "munglinker/train.py", line 171, in <module>
    main(args)
  File "munglinker/train.py", line 137, in main
    load_test_data=False
  File "/Users/jonasschafer/CS/Uni/Year3/Final Project/MuNG-generation/munglinker/data_pool.py", line 495, in load_munglinker_data
    masks_to_bounding_boxes=train_on_bounding_boxes)
  File "/Users/jonasschafer/CS/Uni/Year3/Final Project/MuNG-generation/munglinker/data_pool.py", line 430, in __load_munglinker_data
    mungo.set_mask(image_mask)
  File "/Users/jonasschafer/CS/Uni/Year3/Final Project/MuNG-generation/venv/lib/python3.7/site-packages/muscima/cropobject.py", line 416, in set_mask
    ''.format(mask.shape, (b - t, r - l)))
ValueError: Mask shape (0, 28) does not correspond to integer shape (21, 28) of CropObject.

Here the requirements.txt if this has possibly any origin here:

torch # Renamed from pytorch
numpy>=1.13.3
muscima
omrdatasettools # prepare_dataset.py doesn't work using this
tqdm
tensorboardX
torchsummary
midiutil
pyyaml # Added
sklearn # Added

Do you have any idea where the issue could originate or if this is a simple fix?

Thank you so much.

[apacha]

Hi Jonas,
i had a quick look at the problem and first of all fixed the many issues with regards to old libraries and installation issues.
That part is fixed for now, so I was able to exactly reproduce your error. Will take a closer look at it in the next couple of days.

I guess that this is actually a data issue in the MUSCIMA++ dataset. If you debug the application, you should be able to track down the sample that is causing this problem. Feel free to also create a ticket in this repo: https://github.com/OMR-Research/muscima-pp if you can find the culprit.

[jonas4climate]

Hello Alexander,

Thank you so much for your quick response and for looking into this. I will try to find out which data points where causing the issue and then create a ticket.

[jonas4climate]

I did some debugging in the data_pool.py code to find the underlying problem files which are the following file pairs (see below for details). If this is not a loading issue but these are invalid bounding boxes, I can create a ticket for these in the dataset repo as you mentioned. Would fixing this take long and do you possibly have an idea how to hot-fix this on my side for now i.e. using only valid data? I noticed strange shapes of the image masks it attempted to use so I am assuming these are bounding box issues in the dataset. Further investigating I noticed that actually all issues satisfy this conditiont < 0 or b > image.shape[0] or l < 0 or r > image.shape[1] meaning that they must all be out of bounds for the 2D image array.

However there is quite a lot of them, the for loop loading the bounding boxes for each vertex in each MuNG had 2,223 exceptions and 60,353 valid executions so about 3.5% of labels were invalid? This appears to be a rather large number.

A few handpicked mask issues

Issue 2028 (invalid bounding box): Failed setting mask [1984:2048, 912:919], img size (1306, 3456), res mask shape (0, 7)
data/mungs/CVC-MUSCIMA_W-13_N-16_D-ideal.xml - data/images/CVC-MUSCIMA_W-19_N-19_D-ideal.png

Issue 2132 (invalid bounding box): Failed setting mask [1291:1314, 2799:2883], img size (1306, 3456), res mask shape (15, 84)
data/mungs/CVC-MUSCIMA_W-13_N-16_D-ideal.xml - data/images/CVC-MUSCIMA_W-19_N-19_D-ideal.png

Issue 2222 (invalid bounding box): Failed setting mask [1943:2061, 207:3337], img size (1306, 3456), res mask shape (0, 3130)
data/mungs/CVC-MUSCIMA_W-13_N-16_D-ideal.xml - data/images/CVC-MUSCIMA_W-19_N-19_D-ideal.png

Files affected by bounding box issues

{
('data/images/CVC-MUSCIMA_W-19_N-04_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-38_N-18_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-12_N-04_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-48_N-02_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-42_N-08_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-39_N-12_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-31_N-07_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-09_N-13_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-17_N-01_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-46_N-20_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-09_N-06_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-24_N-01_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-08_N-15_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-25_N-12_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-09_N-13_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-04_N-20_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-02_N-13_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-27_N-16_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-19_N-19_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-13_N-16_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-22_N-15_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-09_N-06_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-32_N-09_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-08_N-10_D-ideal.xml'),
('data/images/CVC-MUSCIMA_W-02_N-06_D-ideal.png', 'data/mungs/CVC-MUSCIMA_W-27_N-03_D-ideal.xml')
}

Debugging Code in data_pool.py

def __load_munglinker_data(mung_root: str, images_root: str,
                           include_names: List[str] = None,
                           max_items: int = None,
                           exclude_classes=None,
                           masks_to_bounding_boxes=False):
    if exclude_classes is None:
        exclude_classes = {}

    all_mung_files = glob(mung_root + "/**/*.xml", recursive=True)
    mung_files_in_this_split = [f for f in all_mung_files if os.path.splitext(os.path.basename(f))[0] in include_names]

    all_image_files = glob(images_root + "/**/*.png", recursive=True)
    image_files_in_this_split = [f for f in all_image_files if
                                 os.path.splitext(os.path.basename(f))[0] in include_names]

    mungs = []
    images = []
    
    n_faulty = 0
    n_valid = 0
    faulty_set = set()
    
    for mung_file, image_file in zip(mung_files_in_this_split, image_files_in_this_split):
        mung = __load_mung(mung_file, exclude_classes)
        mungs.append(mung)

        image = __load_image(image_file)
        images.append(image)

        # This is for training on bounding boxes,
        # which needs to be done in order to then process
        # R-CNN detection outputs with Munglinker trained on ground truth
        if masks_to_bounding_boxes:
            for mungo in mung.vertices:
                t, l, b, r = mungo.bounding_box
                try:
                    image_mask = image[t:b, l:r]
                    mungo.set_mask(image_mask)
                    n_valid += 1
                except ValueError:
                    faulty_set.add((image_file, mung_file))
                    if (t < 0 or b > image.shape[0] or l < 0 or r > image.shape[1]):
                        print(f'Issue {n_faulty} (invalid bounding box): Failed setting mask [{t}:{b}, {l}:{r}], img size {image.shape}, res mask shape {image_mask.shape}\n{mung_file} - {image_file}\n\n')
                    else:
                        print(f'Issue {n_faulty} (unknown)')
                    n_faulty += 1
                
        if max_items is not None:
            if len(mungs) >= max_items:
                break
    print(f'\nProblems with annotations for file pairs: {faulty_set}\n')
    print(f'Unproblematic iterations: {n_valid}')
    print(f'Problematic iterations: {n_faulty}\n\n')

    return mungs, images