[ENH]: Calculate F1 following the recommendations in Forman et at.

[N-Nieto]

Which feature do you want to include?

For extreme imbalance scenarios (1 to 5% positive classes), F1 should not be calculated in each fold and then averaged. Instead, the True positives and False positives should be counted in each fold, and then a final F1 score calculated. This avoids biased results when computing F1 in each fold (which could also be undetermined, if no True classes are in the test set).
Both performances converge when the problem is balanced.

Forman et at.

How do you imagine this integrated in julearn?

Retain the True positives and False positives for each fold, and then calculate a final F1 score.

Do you have a sample code that implements this outside of julearn?

Anything else to say?

No response

[fraimondo]

But this is scoring across several models. You are not scoring one model but an ensemble of models (same learning algorithm, slightly different training sets). This constitutes test-to-test leakage: https://link.springer.com/article/10.1186/s40537-025-01193-8/figures/8

How does Format et al deal with this?

[fraimondo]

Discussion for @kaurao too