Finetuning the SAM2 features model

[anwai98]

Hi @bentaculum, @C-Achard, @maweigert,

Is it possible to train a model with SAM2 features?

For context: I was trying to train a trackastra model with SAM2 features, taking inspiration from your train_config.yaml file for it. I realized that the backbone needs a couple of changes to make it work for the new training design.

My question here would be:

1. Are you planning to incorporate trackastra finetuning with SAM2 features?

If not, I have a super minimalistic pipeline change with which I could make the training work! (thanks to @C-Achard's repo which I took hints from ;))

[nim00007] u12090@glogin10 trackastra $ git diff --stat
 scripts/train.py          | 76 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---
 trackastra/data/data.py   | 72 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++----------------
 trackastra/data/wrfeat.py |  3 ++-
 3 files changed, 131 insertions(+), 20 deletions(-)
[nim00007] u12090@glogin10 trackastra $ 

cc: @constantinpape

[maweigert]

hi @anwai98 ,

We haven't yet have the time to include the finetuning, so this sounds very promising! I think @C-Achard has a better overview how things could be integrated.

[anwai98]

Hi @maweigert,

Thanks for your response. Would it be okay if I create a PR with my changes and ask you or @C-Achard to give it a careful review?

[C-Achard]

Hello @anwai98, happy to have a quick look at the PR if it is opened !

[anwai98]

Hi @C-Achard, that sounds great to me. I just clocked in, I'll check some emails and spawn a PR right away ;)