Obj 4: Testing/Analysis Essence Model Compatibility

[vincepick]

To ensure that the new essence model instance generation is working correctly and begin analysis, Zayaan is going to test it and analyze the results of AutoIG using a variety of metrics.

Action Items for Zayaan, want to log all of this to have in the presentation

• Providing jsutifications for what data will be collected
• Collect the data using numerous machines in parallel
• Identify what information to extract from the data
• Identify how to present the data, potentially using Matplotlib

Initial Testing & Data Collection

• Test using various minimum solver times above 0
  • Observe how many graded instances are made for each value
• Increment the value for maximum solver time
• Experiment with different values for nEvaluations (number of runs per instance)
• Experiment with different valid instance types: sat and unsat
• Experiment with SAT solvers such as kissat
• Test generated instances by car-sequencing, using repair model

Once that data is collected, he can do some analysis on it, such as seeing how many instances were made for each config, average solving times, maximum solving times, etc.

Basic Data Analysis

• Examine number of instances made for each configuration
• Calculate average solving times for dif configs
• Calculate maximum solving times for dif configs
• Calculate the range of solving times for dif configs
• Make some plots for easier analysis, and use them during our presentation.

Basic Metrics

• How many instances are created
• What is the percentage of success for generator configurations

General Suggestions

• Run it on the lab machine
• Use Tmux