Handling too few classes in landmarker cross validation

[bjschoenfeld]

Our landmarkers perform cross validation with 2 folds. Some datasets may have only 1 instance of a particular target class. In this case, the validation in sklearn's cross validation throws an error, requiring at least n_folds (2 in our case) instances of each class. This is not pretty to have such an error thrown. How should we handle this?

[emrysshevek]

Running on LL0_488_colleges_aaup dataset

Traceback (most recent call last):
  File "venv/lib/python3.6/site-packages/metalearn/metafeatures/metafeatures.py", line 113, in compute
    n_folds, verbose
  File "venv/lib/python3.6/site-packages/metalearn/metafeatures/metafeatures.py", line 234, in _validate_compute_arguments
    n_folds, verbose
  File "venv/lib/python3.6/site-packages/metalearn/metafeatures/metafeatures.py", line 348, in _validate_n_folds
    f"{group.shape[0]}."
ValueError: The minimum number of instances in each class of Y is n_folds=2. Class VIIB has 1.

[bjschoenfeld]

Can we compare with OpenML on this?

[emrysshevek]

Similar to this, datasets with fewer than 4 instances per class fail. Should we handle something like this?

import pandas as pd
import numpy as np
from metalearn import Metafeatures
x = pd.DataFrame(np.random.rand(8,2))
y = pd.Series(['a','a','a','b','b','b'])
Metafeatures().compute(x,y)

Traceback (most recent call last):
File "", line 1, in
File "metalearn/metafeatures/metafeatures.py", line 158, in compute
value, compute_time = self._get_resource(metafeature_id)
File "metalearn/metafeatures/metafeatures.py", line 390, in _get_resource
computed_resources = f(**args)
File "metalearn/metafeatures/landmarking_metafeatures.py", line 72, in get_lda
return run_pipeline(X, Y, pipeline, n_folds, cv_seed)
File "metalearn/metafeatures/landmarking_metafeatures.py", line 34, in run_pipeline
'accuracy': accuracy_scorer, 'kappa': kappa_scorer
File "sklearn/model_selection/_validation.py", line 240, in cross_validate
for train, test in cv.split(X, y, groups))
File "sklearn/externals/joblib/parallel.py", line 917, in call
if self.dispatch_one_batch(iterator):
File "sklearn/externals/joblib/parallel.py", line 759, in dispatch_one_batch
self._dispatch(tasks)
File "sklearn/externals/joblib/parallel.py", line 716, in _dispatch
job = self._backend.apply_async(batch, callback=cb)
File "sklearn/externals/joblib/_parallel_backends.py", line 182, in apply_async
result = ImmediateResult(func)
File "sklearn/externals/joblib/_parallel_backends.py", line 549, in init
self.results = batch()
File "sklearn/externals/joblib/parallel.py", line 225, in call
for func, args, kwargs in self.items]
File "sklearn/externals/joblib/parallel.py", line 225, in
for func, args, kwargs in self.items]
File "sklearn/model_selection/_validation.py", line 528, in _fit_and_score
estimator.fit(X_train, y_train, **fit_params)
File "sklearn/pipeline.py", line 267, in fit
self._final_estimator.fit(Xt, y, **fit_params)
File "sklearn/discriminant_analysis.py", line 435, in fit
raise ValueError("The number of samples must be more "
ValueError: The number of samples must be more than the number of classes.

[bjschoenfeld]

datasets with fewer than 4 instances per class fail

I believe you, but why is it 4, not 2? We only do 2-fold cv.

[emrysshevek]

I think it's because with 2-fold cv the training set has half as many instances, so it needs at least 4

[bjschoenfeld]

I would think that if there were only two instances and two folds, one instance would go to each fold. The folds would take turns being the train and test sets...