SASRec perform better in the ml-1m provided in this project

[kkkkobe7]

I want to know if the ml-1m dataset in this project and the ml-1m dataset in SASRec.pytorch are from the same source?

I have tested both projects separately, it seems right. But when I directly removed the 'rate' and 'timestamp' from each row of the ml-1m.txt in this project and tested SASRec, the results were as follows

NCDG@10: 0.8290 HR@10: 0.9831

The results are far from what was reported in the paper.

Both codes used the hyperparameters from the paper, specifically: epoch 200, batch 128, regularization 0.0005, maxlen 50, max_time intervals 2048

[pmixer]

Good intuition and great observation!

I was reproducing the research work actually, based on which I could only guarantee the dataset is the same as used in the original tf implementation https://github.com/JiachengLi1995/TiSASRec/ for consistency(you can vimdiff two files to double-check).

Concerning "the ml-1m dataset in this project and the ml-1m dataset in SASRec.pytorch are from the same source?" - I'd say not, the dataset used in TiSASRec is consistent with the original ratings in ml-1m https://github.com/RuidongZ/Recurrent_Recommender_Networks/blob/master/data/ml-1m/ratings.dat ((this is a copy on github, I'd suggest download the original file from movielens dataset website if you still have concerns)) as you can refer to num of rows and item ids.

SASRec seems filtered out few too-short sequences, and re-ordered item ids, there might still be some issues in the procedure, for which you can try your own data processing scripts to re-process original ml-1m data for exploration and comparison.

In the end, there seems no standard in data preprocessing and evaluation in the field yet, and there are inherent randomness in model training(negative sampling) and inference(still random easy-negative-examples sampling, if without ANNS), so the best way should be going deeper into raw data details as you did, that's also how people do in industry.

[kkkkobe7]

Good intuition and great observation!

I was reproducing the research work actually, based on which I could only guarantee the dataset is the same as used in the original tf implementation https://github.com/JiachengLi1995/TiSASRec/ for consistency(you can vimdiff two files to double-check).

Concerning "the ml-1m dataset in this project and the ml-1m dataset in SASRec.pytorch are from the same source?" - I'd say not, the dataset used in TiSASRec is consistent with the original ratings in ml-1m https://github.com/RuidongZ/Recurrent_Recommender_Networks/blob/master/data/ml-1m/ratings.dat ((this is a copy on github, I'd suggest download the original file from movielens dataset website if you still have concerns)) as you can refer to num of rows and item ids.

SASRec seems filtered out few too-short sequences, and re-ordered item ids, there might still be some issues in the procedure, for which you can try your own data processing scripts to re-process original ml-1m data for exploration and comparison.

In the end, there seems no standard in data preprocessing and evaluation in the field yet, and there are inherent randomness in model training(negative sampling) and inference(still random easy-negative-examples sampling, if without ANNS), so the best way should be going deeper into raw data details as you did, that's also how people do in industry.

Thank you very much for your prompt response. I will conduct more experiments to delve deeper into the detailed issues.