Test cases for automated generation of repair model

[ndangtt]

conjure parameter-generator problem.essence will generate two files:

• generator model
• repair model

Since we want to allow users to provide their own generator model (rather than the one created by conjure), we want conjure to have a functionality that does:
conjure repair-generator generator.essence
This will create a repair.essence model accordingly.

To create test cases for this functionality, we can have a set of problem.essence models, for each one, we run:
conjure parameter-generator problem.essence
and get back a pair of (generator.essence, repair.essence)

1. We can write a test script to compare repair.essence with the output of conjure repair-generator generator.essence.

2. We want to have a number of test cases, ranging from simple (with only a few parameters) to complicated

[ndangtt]

How to ensure the repair.essence model created by conjure parameter-generator is correct:

• treat both the generator.essence and repair.essence created by conjure parameter-generator as manually written.
• plug them into an irace experiment and run the tuning with them.

[ndangtt]

A test case:

• a problem model
• a pair of generator.essence and repair.essence models:
  1. created automatically by conjure parameter-generator
  2. created by ourselves

Automated checking:

• Compare two repair.essence models (only work with 1) )
• Plug problem.essence, generator.essence and repair.essence into an irace experiment and run the tuning with them

[spritezs]

proposed layout of the test script:

input <- problem in essence

1. use conjure parameter-generator to create generator and repair models

2. run conjure repair-generator ( new function ) on the generator created in step 1.

3. if they two repair models are identical, go to X

4. run irace with both repair models ( same problem and generator ) and check that the outcome is the same

5. need to still test the validity of the created repair models. we con do this by either comparing them to manually written ones or by testing it over several of our own generators

test cases for the script above would be: car (only one we have so far), etc (to be written).

[ndangtt]

Regarding 3), it just came to my mind that conjure may have a function to convert an Essence model to a json file. In that case we can compare two dictionaries instead of two text files. Let us discuss that with Oz tomorrow.

Regarding 4), it can also be used for checking the validity of a single repair model.

I think it'd be good to start with some example test cases (following steps 1 and 2 you described above), then refining the process afterwards.

[ozgurakgun]

Regarding 3), it just came to my mind that conjure may have a function to convert an Essence model to a json file. In that case we can compare two dictionaries instead of two text files. Let us discuss that with Oz tomorrow.

conjure pretty model.essence --output-format=astjson

should do the trick.

[ndangtt]

@ozgurakgun: perfect, thanks very much!

[spritezs]

Thank you! Will try that

[spritezs]

test_repair.txt

Implementation of steps 1-3. The script goes into the /scripts directory. Our repair model also goes there for now.

I had to improvise for now by using our own repair model. The code is python (github won't let me insert .py files so I changed it to .txt). Please let me know if I need to change anything. Will discuss how to implement the next steps this week. Thank you for your time!! I can show you what it does when we meet since the comments might not explain it well.
@ozgurakgun @ndangtt