-
Notifications
You must be signed in to change notification settings - Fork 14
Description
I would like to firstly express my appreciation for this remarkable work, however, I encountered some problems when trying to reproduce the experiment. The details are as follows:
When I ran the code at the first time, I found that the example file in prompt_tuning.sh was missing.
Then, I followed the Usage section in readme.md and substituted the log files with "${dataset}_2k.log_structured.csv" files. However, the result files contain 2000 lines of parsed logs, meaning that the model has parsed all the log messages in the "2k.log_structured.csv" files for benchmarking.
Since the 32-shot examples for model training are also selected from the 2k log files of each dataset (as shown in the few_shot_sampling.py):
I consider that using the "${dataset}_2k.log_structured.csv" data files could probably lead to the data leakage. Therefore, could you upload the example files for log parsing in your previous experiments so that I can better reproduce the results? Thanks for your time!