[Task] Profiling the training&evaluation pipeline for retrieval and ranking models

[gabrielspmoreira]

🚀 Feature request

Profiling the training&evaluation pipeline for retrieval and ranking models and finding bottlenecks that should be solved for better performance and GPU utilization.

For performing this task, the research scripts for training and evaluating retrieval and ranking models will be used on the TenRec dataset.
The instructions for setting up the experiments (container, dataset download, and how to run the scripts) is available in this document.

[viswa-nvidia]

@gabrielspmoreira , is this a bug or a feature request. The description says steps to reproduce bug. Please clarify.

[viswa-nvidia]

@gabrielspmoreira to co-ordinate with Valerie Sarge for definition

[gabrielspmoreira]

We are observing a slower runtime in the script with V2 API of retrieval models compared to V1. @sararb is currently finishing investigating the V2 implementation in PR #790 and is trying to isolate which block(s) is/are introducing the slowness.
The profiling task could start after we make sure the V2 implementation is at least as fast as V1.