Support for multi-output tree models and their ensembles

[fritshermans]

Hi Villu,

I'm a very happy user of your package. This time I wanted to convert a trained multilabel RandomForestClassifier instance to PMML. Let me show you a minimal example to reproduce the error (sklearn2pmml 0.129.2, sklearn 1.7.2):

from sklearn.datasets import make_multilabel_classification
from sklearn.ensemble import RandomForestClassifier

X, y = make_multilabel_classification()

clf = RandomForestClassifier().fit(X,y)

sklearn2pmml(clf, "clf.pmml")

I get the following error:

Exception in thread "main" java.lang.ClassCastException: class org.jpmml.converter.MultiLabel cannot be cast to class org.jpmml.converter.CategoricalLabel (org.jpmml.converter.MultiLabel and org.jpmml.converter.CategoricalLabel are in unnamed module of loader 'app')

Is there something that I do wrong?

[vruusmann]

Is there something that I do wrong?

You're doing everything correctly (from the Scikit-Learn worflow perspective).

The problem is, that JPMML does not currently support the conversion of multi-output tree models. The support is halfway complete - the converter understand that there are multiple labels involved (ie. using MultiLabel instead of a ScalarLabel).

The base roblem is that the PMML 4.4(.1) does not specify a mechanism for representing multi-target trees - the current implementation is strictly one-target only.

Adding multi-target support would be pretty straightforward (at least for classification models). Simply add a ScoreDistribution@targetField attribute, which lets you distinguish between different ScoreDistribution element subsets under one terminal Node element.

For example:

<Node>
  <!-- first target -->
  <ScoreDistribution targetField="y1" value="0" probability="0.1"/>
  <ScoreDistribution targetField="y1" value="1" probability="0.9"/>
  <!-- second target -->
  <ScoreDistribution targetField="y2" value="0" probability="0.7"/>
  <ScoreDistribution targetField="y2" value="1" probability="0.3"/>
<Node>

This kind of multi-target support is needed elsewhere as well. For example, both LightGBM and XGBoost also support such multi-target eafs.

[vruusmann]

The immediate workaround is to replace a single multi-target RandomForestClassifier step with multiple single-target RandomForestClassifier steps, possibly with the help of MultiOutputClassifier.

Sure, it bloats the size of the PMML document unnecessarily, but at least it gets the job done.

You could also try dissecting a trained multi-target RandomForestClassifier into multiple single-target estimators manually. The goal is to have scalar scores in the Scikit-Learn tree structure.

[fritshermans]

Thanks for your quick response Villu! Much appreciated.

I tried your suggestion and wrapped the classifier in a pipeline in order to deal with categorical features. I get an error in case I limit the number of categories, which is often required.

Please find an example below that raises an error:

import pandas as pd

from sklearn.datasets import make_multilabel_classification
from sklearn.ensemble import RandomForestClassifier
from sklearn.pipeline import make_pipeline
from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder
from sklearn.multioutput import MultiOutputClassifier

X, y = make_multilabel_classification()
X = pd.DataFrame(X, columns=[f"x_{x}" for x in range(len(X[0]))])

ohe_feats = ['x_0', 'x_1', 'x_2', 'x_3', 'x_4', 'x_5', 'x_6']
trafo = ColumnTransformer([('ohe', OneHotEncoder(max_categories=3), ohe_feats)], remainder='passthrough')
clf = MultiOutputClassifier(make_pipeline(trafo, RandomForestClassifier()))
clf.fit(X,y)

sklearn2pmml(clf, "clf.pmml")

I get the following error:

org.jpmml.sklearn.SkLearnException: Field 'regroup(x_0)' is already defined

Would you probably know a workaround?

[vruusmann]

org.jpmml.sklearn.SkLearnException: Field 'regroup(x_0)' is already defined

Right now you have feature transformations inside MultiOutputClassifier. Therefore, one and the same OneHotEncoder logic is trying to run multiple times - once for each label.

The fix is straightforward, move feature transformations outside of MultiOutputClassifier:

pipeline = Pipeline([
  ("transformer", trafo),
  ("classifier", MultiOutputClassifier(RandomForestClassifier()))
])
pipeline.fit(X, y)