Fanova, Pairs of Variable Plots, AttributeError: 'CategoricalHyperparameter' object has no attribute 'lower'

[robintibor]

Hi,
If I want to create the fanova plots, the single variable plots work, but pairs fail:

>>> python3.5 /home/schirrmr/programs/ParameterImportance/scripts/evaluate.py --scenario_file scenario.txt --history 'runhistories-shared/runhistory*.json' --modus fanova
INFO:Importance:Reading Scenario file and files specified in the scenario
INFO:smac.scenario.scenario.Scenario:Reading scenario file: deep4_scenario.txt
INFO:smac.scenario.scenario.Scenario:Output to PIMP_fanova_2017_06_06_14:20:34
INFO:Importance:Reading Runhistory
INFO:Importance:Combined number of Runhistory data points: 1038
INFO:Importance:Converting Data and constructing Model
INFO:Importance:Setting up Evaluation Method
INFO:Importance:Running evaluation method fANOVA
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/allow_crash.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/data_folder.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/do_batch_norm.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/double_time_convs.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/drop_prob.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/exponential_demean_factor.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/exponential_standardize_factor.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/filter_length_2.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/filter_length_3.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/filter_length_4.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/filter_time_length.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/final_dense_length.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/first_nonlin.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/first_pool_mode.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/last_seizure_train_ms.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/later_nonlin.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/later_pool_mode.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/n_chans.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/n_epochs.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/non_seiz_to_seiz_factor.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/num_filters_2.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/num_filters_3.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/num_filters_4.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/num_filters_spat.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/num_filters_time.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/only_return_exp.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/pool_time_length.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/pool_time_stride.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/resample_mean_pred_ms.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/run_file.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/sample_by_preds.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/sampling_rate.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/save_folder.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/split_first_layer.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/threshold.png
creating PIMP_fanova_2017_06_06_14:20:34_run1/fanova/['drop_prob', 'filter_length_2'].png
Traceback (most recent call last):
  File "/home/schirrmr/programs/ParameterImportance/scripts/evaluate.py", line 52, in <module>
    importance.plot_results(name=os.path.join(save_folder, args.modus))
  File "/home/schirrmr/programs/ParameterImportance/pimp/importance/importance.py", line 254, in plot_results
    self.evaluator.plot_result(name)
  File "/home/schirrmr/programs/ParameterImportance/pimp/evaluator/fanova.py", line 68, in plot_result
    vis.create_all_plots(name)
  File "/home/schirrmr/.local/lib/python3.5/site-packages/fanova/visualizer.py", line 57, in create_all_plots
    self.plot_pairwise_marginal(combi, **kwargs)
  File "/home/schirrmr/.local/lib/python3.5/site-packages/fanova/visualizer.py", line 114, in plot_pairwise_marginal
    lower_bound = self.cs_params[p].lower
AttributeError: 'CategoricalHyperparameter' object has no attribute 'lower'

If you need more context: http://nbviewer.jupyter.org/gist/robintibor/6dad6629987cd66b060503bc43e4e2bc/Hyperparameter_June_Report.ipynb

[sfalkner]

That is definitely in the fANOVA itself. I think that is in part Christina's (@Krxsy) HiWi task for next week.

[sudo-chw]

Yes, it makes sense that there should be a "plot_pairwise_categorical_marginal" function. Right now it only works for a single categorical param.

[robintibor]

Hm, however plot is not created and variables are both not categorical, but float and integer:
drop_prob [0.0,0.9] [0.5]
filter_length_2 [3,14] [10]i

[sudo-chw]

Ok.. that's weird. Could you give me the configspace info e.g print(cs.get_hyperparameters()) , please? I would like to compare it to the pcs-file for checking if the order of the parameters is correct.

[robintibor]

Oki, but how to get the cs-object? I have no experience with the library outside of command-line-calls like
python3.5 /home/schirrmr/programs/ParameterImportance/scripts/evaluate.py --scenario_file deep4_scenario.txt --history 'runhistories-shared/runhistory*.json' --modus fanova
:)

[sudo-chw]

import ConfigSpace
cs = ConfigSpace.ConfigurationSpace()
parameters = cs.get_hyperparameters()
print(parameters)

[robintibor]

Hm ok I still don't understand, I assume you want me to create the configspace from the pcs file and then print it right? But how do I read in the pcs file to the config space? Or do you want me to do something else?

[AndreBiedenkapp]

I think she want's you to do the following:

from ConfigSpace.io import pcs
with open(file_name) as fp:
    pcs_str = fp.readlines()
    cs = pcs.read(pcs_str)
    cs.seed(42)

print(cs.get_hyperparameters())

if 'from ConfigSpace.io import pcs' does not work try it with
from ConfigSpace.io import pcs_new

[robintibor]

ok in that case

file_name = 'deep4.pcs'
from ConfigSpace.io import pcs
with open(file_name) as fp:
    pcs_str = fp.readlines()
    cs = pcs.read(pcs_str)
    cs.seed(42)

cs.get_hyperparameters()

leads to

[allow_crash, Type: Categorical, Choices: {true}, Default: true,
 data_folder, Type: Categorical, Choices: {/home/schirrmr/data/epilepsy/earlyseiz/manuel_elecs/}, Default: /home/schirrmr/data/epilepsy/earlyseiz/manuel_elecs/,
 do_batch_norm, Type: Categorical, Choices: {true, false}, Default: true,
 double_time_convs, Type: Categorical, Choices: {true, false}, Default: true,
 drop_prob, Type: UniformFloat, Range: [0.0, 0.9], Default: 0.5,
 exponential_demean_factor, Type: Categorical, Choices: {0.001}, Default: 0.001,
 exponential_standardize_factor, Type: Categorical, Choices: {null}, Default: null,
 filter_length_2, Type: UniformInteger, Range: [3, 14], Default: 10,
 filter_length_3, Type: UniformInteger, Range: [3, 14], Default: 10,
 filter_length_4, Type: UniformInteger, Range: [3, 14], Default: 10,
 filter_time_length, Type: UniformInteger, Range: [3, 25], Default: 10,
 final_dense_length, Type: UniformInteger, Range: [3, 14], Default: 10,
 first_nonlin, Type: Categorical, Choices: {elu, identity, relu, square}, Default: elu,
 first_pool_mode, Type: Categorical, Choices: {average_exc_pad, max}, Default: max,
 last_seizure_train_ms, Type: UniformFloat, Range: [10000.0, 60000.0], Default: 30000.0, on log-scale,
 later_nonlin, Type: Categorical, Choices: {elu, relu, square}, Default: elu,
 later_pool_mode, Type: Categorical, Choices: {average_exc_pad, max}, Default: max,
 n_chans, Type: Categorical, Choices: {null}, Default: null,
 n_epochs, Type: UniformInteger, Range: [15, 16], Default: 15,
 non_seiz_to_seiz_factor, Type: UniformFloat, Range: [2.0, 9.0], Default: 5.0,
 num_filters_2, Type: UniformInteger, Range: [15, 100], Default: 25,
 num_filters_3, Type: UniformInteger, Range: [15, 100], Default: 25,
 num_filters_4, Type: UniformInteger, Range: [15, 100], Default: 25,
 num_filters_spat, Type: UniformInteger, Range: [15, 100], Default: 25,
 num_filters_time, Type: UniformInteger, Range: [15, 100], Default: 25,
 only_return_exp, Type: Categorical, Choices: {false}, Default: false,
 pool_time_length, Type: UniformInteger, Range: [1, 8], Default: 3,
 pool_time_stride, Type: UniformInteger, Range: [1, 8], Default: 3,
 resample_mean_pred_ms, Type: UniformFloat, Range: [2000.0, 20000.0], Default: 2000.0, on log-scale,
 run_file, Type: Categorical, Choices: {/home/schirrmr/code/earlyseiz/configs/deep4_net.py}, Default: /home/schirrmr/code/earlyseiz/configs/deep4_net.py,
 sample_by_preds, Type: Categorical, Choices: {true, false}, Default: true,
 sampling_rate, Type: UniformFloat, Range: [32.0, 256.0], Default: 128.0,
 save_folder, Type: Categorical, Choices: {/home/schirrmr/data/epilepsy/earlyseiz/models/deep4-pscmac-dirs/}, Default: /home/schirrmr/data/epilepsy/earlyseiz/models/deep4-pscmac-dirs/,
 split_first_layer, Type: Categorical, Choices: {true, false}, Default: true,
 threshold, Type: UniformFloat, Range: [0.2, 0.9], Default: 0.59999999999999998]

[sudo-chw]

So the order is ok.
Could you send a run history?

[robintibor]

Ok I put all runhistories here: https://gist.github.com/robintibor/6a378e38cb033c58dd72da1e4c5f6e90

[sudo-chw]

Ok, thanx. It might take some time until the problem is found/fixed.

[sudo-chw]

Can you try it again with the latest changes and let me know if the error still shows up?

[robintibor]

Hm, there seems to be a new problem:

>>> python3.5 /home/schirrmr/programs/ParameterImportance/scripts/evaluate.py --scenario_file scenario.txt --history 'runhistories-shared/runhistory*.json' --modus fanova
INFO:Importance:Reading Scenario file and files specified in the scenario
INFO:smac.scenario.scenario.Scenario:Reading scenario file: scenario.txt
INFO:smac.scenario.scenario.Scenario:Output to PIMP_fanova_2017_06_10_15:08:02
INFO:Importance:Reading Runhistory
INFO:Importance:#RunHistories found: 24
INFO:Importance:Combined number of Runhistory data points: 1100
INFO:Importance:Number of Configurations: 274
INFO:Importance:Converting Data and constructing Model
INFO:Importance:Size of training X: (1100, 35)
INFO:Importance:Size of training y: (1100, 1)
INFO:Importance:Data was not imputed
INFO:Importance:Thus the size of X might be smaller than the datapoints in the RunHistory
Traceback (most recent call last):
  File "/home/schirrmr/programs/ParameterImportance/scripts/evaluate.py", line 38, in <module>
    impute_censored=args.impute)  # create importance object
  File "/home/schirrmr/programs/ParameterImportance/pimp/importance/importance.py", line 88, in __init__
    self.incumbent = incumbents[0]
IndexError: list index out of range

Anyways thanks for your help until this point I am now on vacation until 19th June :)

[AndreBiedenkapp]

That's my fault. Apparently it does not find any traj_aclib2.json files so it can also not determine an incumbent because I assumed that traj_aclib2.json files would always be located in a smac3-output... folder.

I'll quickly fix that