Epistemic uncertainty with and without effects #4
Description
Hi, I ran the experiments for ood_loc_gpn_{10,16} to reproduce the results. The metrics are logged after training as follows:
| val | test | |
|---|---|---|
| accuracy | 0.923077 | 0.889306 |
| brier_score | 0.280527 | 0.269254 |
| ECE | 0.10638 | 0.0702577 |
| confidence_aleatoric_apr | 0.966722 | 0.980251 |
| confidence_epistemic_apr | 0.95776 | 0.962611 |
| confidence_structure_apr | nan | nan |
| confidence_aleatoric_auroc | 0.706276 | 0.864282 |
| confidence_epistemic_auroc | 0.635031 | 0.772384 |
| confidence_structure_auroc | nan | nan |
| CE | 0.37979 | 0.339846 |
| avg_prediction_confidence_aleatoric | 0.816697 | 0.819579 |
| avg_prediction_confidence_epistemic | 14919.8 | 14410.5 |
| avg_sample_confidence_aleatoric | 0.816697 | 0.819579 |
| avg_sample_confidence_epistemic | 16556.9 | 15927.3 |
| avg_sample_confidence_features | 15835.7 | 16779.6 |
| avg_sample_confidence_neighborhood | nan | nan |
| average_entropy | 0.513569 | 0.501767 |
| ood_detection_aleatoric_apr | 0.801985 | 0.78415 |
| ood_detection_aleatoric_auroc | 0.833289 | 0.836685 |
| ood_detection_epistemic_apr | 0.80048 | 0.788122 |
| ood_detection_epistemic_auroc | 0.852889 | 0.855747 |
| ood_detection_features_apr | 0.75896 | 0.748484 |
| ood_detection_features_auroc | 0.81794 | 0.8237 |
| ood_detection_neighborhood_apr | nan | nan |
| ood_detection_neighborhood_auroc | nan | nan |
| ood_detection_structure_apr | nan | nan |
| ood_detection_structure_auroc | nan | nan |
| ood_accuracy | 0 | 0 |
| ood_avg_prediction_confidence_aleatoric | 0.556919 | 0.558297 |
| ood_avg_prediction_confidence_epistemic | 2879.92 | 2808.5 |
| ood_avg_sample_confidence_aleatoric | 0.556919 | 0.558297 |
| ood_avg_sample_confidence_epistemic | 4068.24 | 4109.26 |
| ood_avg_sample_confidence_neighborhood | nan | nan |
| ood_avg_sample_confidence_features | 2063.81 | 1759.3 |
| ood_average_entropy | 1.03177 | 1.03767 |
| id_accuracy | 0.923077 | 0.889306 |
| id_avg_prediction_confidence_aleatoric | 0.816697 | 0.819579 |
| id_avg_prediction_confidence_epistemic | 14919.8 | 14410.5 |
| id_avg_sample_confidence_aleatoric | 0.816697 | 0.819579 |
| id_avg_sample_confidence_epistemic | 16556.9 | 15927.3 |
| id_avg_sample_confidence_features | 15835.7 | 16779.6 |
| id_average_entropy | 0.513569 | 0.501767 |
Is this experiment exclusive for the With Effect case? I would like to reproduce 3 cases: aleatoric w/Net, Epist w/Net and Epist w/o Net as table 1 in the paper. Could you provide me with some guidance? Thank you.