Thesis Repository

Author: Muhammad Irfan Mas'udi

Supervisor: Prof. Luciano Baresi, Co-Supervisor: Davide Hu

Master in Computer Science and Engineering @POLIMI

“DILLEMA: Metamorphic Testing for Deep Learning using Diffusion and Large Language Models”

This repository related to training and testing process of the state-of-the-art architectures. Here is DILLEMA repository

DILLEMA is a framework to generate new test cases to uncover the faulty behavior of Image-based Deep Learning application (e.g. Autonomous driving). In the result, DILLEMA detects 24.8 % pixel-based misclassification of Semantic Segmentation task on Autonomous driving on SHIFT dataset. Moreover, it also found 53.3 % misclassification images in state-of-the-art neural networks for Classification task with ImageNet1K dataset.

Folder Structure

1. SHIFT contains notebooks and executable Python for training the model DeepLabV3_ResNet50 and testing processes (DILLEMA augmentation) for Semantic Segmentation task with SHIFT dataset (synthetic dataset for autonomous driving). The model builds on PyTorch Lightning. It contains data exploration and data visualization for the result (e.g. confusion matrix).

2. Imagenet contains notebooks and executable for training and testing processes for the state-of-the-art (ResNet18, ResNet50, ResNet152) of pre-trained model for ImageNet1K which is built on top of PyTorch. This folder also contains data-exploration (Pandas), data-visualization (matplotlib, seaborn).

Result: Comparison between original and DILLEMA Augmented

1. SHIFT Dataset on Semantic Segmentation task

3. ImageNet1K Dataset for Classification task

Because the confusion matrix is vast, here we attach several spreadsheet results:

ResNet18 Original ResNet18 DILLEMA ResNet50 Original ResNet50 DILLEMA ResNet152 Original ResNet152 DILLEMA