Usage of prepare_code_trace.py?

[DanielKotroco]

Hello,

In the Trex paper, it's described in Section V that the code base implements microtracing through emulation, which appears to be done in 'micro_trace/prepare_code_trace.py'. However, the TREX github README doesn't mention calling prepare_code_trace.py in the data processing pipeline. Would it be possible to get clarification as to how the traces from emulation get created in either preprocessing or finetuning?

[peikexin9]

Hi @DanielKotroco Thanks for your interests. The traces are only used in pretraining and not finetuning. As I have released the pretrained model (trained from micro-traces), so you don't need to trace the code anymore for finetuning. Instead, in finetuning's dataset, the trace (byte1-4) are just dummy values, with some values directly coming from the assembly code (not micro-traced). For example, if you have a static code like mov eax, 0xdeadbeef. then your finetuning's train.static would be like mov eax , hexvar, and your train.byte1 would be like ## ## ## de, train.byte2 would be like ## ## ## ad, train.byte3 would be like ## ## ## be, and train.byte4 would be like ## ## ## ef, (e.g., the value for eax is dummy ##, instead of a concrete micro-trace values).

[DanielKotroco]

Thanks for the quick response! That all makes sense. I would like to pretrain the model with micro-traces from a custom dataset. In particular, I am trying to pretrain on additional instruction-set architectures beyond what Trex currently supports. As such, I cannot make use of the existing pretrained model. Are there any scripts which use the output of 'micro_trace/prepare_code_trace.py' that you can share?

[amitz-argus]

Hello,

I also want to pretrain the model on other architectures than the supplied pretrained model was trained on, and I encountered the same questions as Daniel did.

It seems to me that "prepare_code_trace.py" outputs a JSON for every (arch, bit, opt) tuple to the "data-raw/functraces" directory, while the preprocess scripts of the pretraining stage ("preprocess_pretrain.py") expects to receive a file for every field in "data-src" directory, so I guess there is another stage in between that performs this transformation?

I would appreciate if you could provide further details/explanation on those parts of the preprocessing pipeline, so I can pretrain the model on my own data, and it would be really awesome if you could share anything that use the output of "prepcare_code_trace.py" as Daniel asked :)

Thanks in advance,
amitz-argus

[peikexin9]

Hi @DanielKotroco and @amitz-argus, my apology for the delayed reply. You might want to check command/finetune/prepare_finetune.py just very briefly -- no need to look into details, as basically you can just write a very simple few lines of python script, taking the results from prepare_code_trace.py, and output 8 files with 8 extensions ['static', 'inst_pos_emb', 'op_pos_emb', 'byte1', 'byte2', 'byte3', 'byte4', 'arch_emb']. Each line in each file should be aligned (having same exact number of tokens delimited by the space) and they together correspond to a single sample.