I cannot reproduce Gemma2B checkpoint's result

[otmhi]

Hello there,

First of all, great work. I'm trying to reproduce your results table with the Gemma 2B adapter you provided on hugging face. As there is no standalone evaluation script in your code, I created one by copying exactly what is done in your training code and obtained slightly off metrics.

I'm using the hyper-parameters mentionedx in the ReadMe file for the evaluation:

do_sample = False
max_new_tokens = 640
dtype = "bfloat16"
similarity_type = "dot",
similarity_normalization = True

and I obtained on CDs and Vinyl the following metrics:

'ndcg@5': 0.03791472700262674,
'ndcg@10': 0.042755681174091224,
'ndcg@20': 0.05008330932310675,

'hit_rate@5': 0.05592841163310962,
'hit_rate@10': 0.07084265473527218,
'hit_rate@20': 0.09992542878448919

Which are lower compared to the reported metrics in the paper. Am I doing something wrong?

Can you please provide the evaluation script you are using ?