Instantiating a Loader halts: Graph.group_operations gets stuck in an infinite loop

[juhosattila]

Hi there. In FFCV, version 1.0.2, the function Graph.group_operations gets stuck in an infinite loop (which happens when we initialize a Loader).

The particular code in question is the following:

while current_front:
    current_stage = list()
    jitted_stage = len(stages) % 2 == 0

    while current_front:
        node = current_front.pop()
        if node.is_jitted == jitted_stage or node.is_jitted is None:
            current_stage.append(self.node_to_id[node])
            current_front.update(set(self.adjacency_list[node]))

        else:
            next_front.add(node)

    stages.append(current_stage)
    current_front = next_front

To me it seems that at the end of the execution of the entire body of the outer while-loop, the variables current_front and next_front are set to refer to same underlying set. As a consequence, on the second run of the outer loop, the inner loop gets into an infinite loop if:

1. current_front has exactly one element
2. this element, extracted into node, does not satisfy the if condition and is put back into next_front, which refers to the same set as current_front.

From there the inner loop just gets stuck into infinite execution.

[juhosattila]

Upon further debugging it seems to occur if a ToDevice operations was removed from a pipeline, e.g.:

test_pipeline = [
     ffcv.fields.rgb_image.CenterCropRGBImageDecoder((224,224), ratio=224.0/256),
     ffcv.transforms.ToTensor(),
     # removed from pipeline #ffcv.transforms.ToDevice(torch.device('cuda'), non_blocking=True),
     ffcv.transforms.ToTorchImage(),
     ffcv.transforms.NormalizeImage(np.array(IMAGENET_MEAN) * 255, np.array(IMAGENET_STD) * 255, np.float16),
]