Ensemble Selection

[otoalves]

No description provided.

[larskotthoff]

Thanks. I'm getting an exception when no configurations were evaluated:

java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    java.util.ArrayList.rangeCheck(ArrayList.java:653)
    java.util.ArrayList.get(ArrayList.java:429)
    autoweka.ConfigurationCollection.forceFirst(ConfigurationCollection.java:175)
    autoweka.ConfigurationCollection.rank(ConfigurationCollection.java:139)
    weka.classifiers.meta.AutoWEKAClassifier.buildClassifier(AutoWEKAClassifier.java:377)
    weka.gui.explorer.AutoWEKAPanel$11.run(AutoWEKAPanel.java:411)

    at java.util.ArrayList.rangeCheck(ArrayList.java:653)
    at java.util.ArrayList.get(ArrayList.java:429)
    at autoweka.ConfigurationCollection.forceFirst(ConfigurationCollection.java:175)
    at autoweka.ConfigurationCollection.rank(ConfigurationCollection.java:139)
    at weka.classifiers.meta.AutoWEKAClassifier.buildClassifier(AutoWEKAClassifier.java:377)
    at weka.gui.explorer.AutoWEKAPanel$11.run(AutoWEKAPanel.java:411)

and a null pointer exception in another case:

java.lang.NullPointerException
    at autoweka.Ensembler.parseDatasetMetadata(Ensembler.java:80)
    at autoweka.Ensembler.<init>(Ensembler.java:60)
    at weka.classifiers.meta.AutoWEKAClassifier.buildClassifier(AutoWEKAClassifier.java:382)
    at weka.classifiers.evaluation.Evaluation.evaluateModel(Evaluation.java:1527)
    at weka.classifiers.Evaluation.evaluateModel(Evaluation.java:650)
    at weka.classifiers.AbstractClassifier.runClassifier(AbstractClassifier.java:141)
    at weka.classifiers.meta.AutoWEKAClassifier.main(AutoWEKAClassifier.java:203)

Could you have a look please?

[otoalves]

Hi Lars,

Do you know which datasets/seed/constraints caused these exceptions? I might have fixed the issue but it would be safer to run autoweka with the same arguments again to confirm.

[larskotthoff]

I think I tried this with the yeast dataset from the Auto-WEKA paper. Presumably you tested with all of them to measure the performance improvement with ensembles?

[otoalves]

Hi Lars,
The issue when there are no configurations should be fixed now. I hadn't tested with all of them, I had only used a subset which mostly contained the smaller ones, which is probably why I hadn't gotten this exception recently

[larskotthoff]

Great, thanks! Do you have the performance comparison with and without ensembles as well?

[otoalves]

Not yet - I've done a few test runs, but not in a very methodical way. I'm writing a script to run on Parallel over the weekend, so that I can make some pretty graphs with more concrete measures =)
I'm sorry for the delay regarding this, things have been a permanent rush since I arrived here.

[larskotthoff]

No worries. I've run some small experiments myself and didn't see any improvement with ensembles, so I'm curious to see your experiments.

[larskotthoff]

Hi Oto, did you have a chance to look at running the experiments? I want to merge this, but would like to see some numbers on how much it improves first.

[otoalves]

Hi Lars,
I have them running right now, will be ready in a couple days =)