FYP-FINAL

[damienbose]

README - FYP Project

This Pull Request contains all the changes done to the base Magpie framework to integrate the RL approach and run my experiments.

Personal Contributions:

1. Contextual Operators: Changed files in xml folder to add contextual operators to Magpie (.e.g ExprStmtInsertion)
2. Operator Selectors: The implementation of the different Bandit algorithms are in magpie/base/operator_selector.py. We make use of abstract classes to log intermediary results.
3. Generate Max Coverage Suite: Added experiments/generate_coverage coverage folder with code to generate branch coverage data for MiniSAT and produce a test suite which maximise branch coverage.
4. Setup, Validate & Run experiments: Added code to automate running the experiment, validate results, and log/pickle the results for analysis.
5. Analysing the results: Code used to produce graphs and experiment metrics can be found in experiments/process_results/process_official_final.ipynb.
6. Heuristic Search Algorithms: The search algorithms were modified from the generic ones in Magpie. These can be found in magpie/algo/local_search.py under the names RandomSearch for Neighbourhood Search and FYPLocalSearch for Hill Climbing

Experiment Results

The raw experiment results can be found as a zip file in a separate repository here. (Warning: This is not anonymous)

You will also need to install git lfs to download the required data.

Replicating The Experiments

Make sure you have all the python dependencies listed.

Run experiments:

bash experiments/run.sh &

Note that running took about 50 hours 40 mins on our hardware specs. All the logs files should go in the experiments/results folder.

Produce graphs and metrics:

Change the RESULT_DIR variable in the experiments/process_results/process_official_final.ipynb notebook to point to the unzipped version of your results folder. Then run all the cells and the metrics and graphs included in the report should be computed automatically.

Software Versions Used

• g++ 11.4.0
• perf 6.5.13
• Python 3.10.12

Python Requirements:

• numpy
• gcovr
• tqdm
• notebook
• pandas
• scipy
• matplotlib