Questions on Token Manipulation Network

[vinayak-sharan]

Hello @hologerry,

Great Work!

I have some questions on Token manipulation Network, it would be great if you could answer them :) As, the details are bit sparse in the paper.

For instance what is the source token and target token for training the Adapter? If headpose is taken as a condition, then ideally the source token and target token should have same expressions but different headpose. How did you get the dataset for this? Paper mentions FFHQ in the wild but it does not have two sets of images right?

[johndpope]

Hi @vinayak-sharan you maybe interested i started this here -
https://github.com/johndpope/IMF/tree/feat/token_manipulation

once the model is trained - stage 1-
then you are manipulating the token with conditions and you can prepare dataset with that data -
just go through a video - sample the eye gazz and pack into your get item

then with the loss function just compare the predicted + eye gazz to the generated image

that layer is going to sit on top of the trained one....
i created this ticket here as there's many different ways we could manipulate it
johndpope/IMF#32

(i do want to circle back to this - but kinda lost interest - im pulling apart vasa stage 2 for like the 6th or 7th time. cant get the expression to recreate https://wandb.ai/snoozie/vasa/runs/e7hq5kl3?nw=nwusersnoozie)

[vinayak-sharan]

Hey @johndpope, Thank you for your inputs but did you get success in manipulating the tokens?

Theoretically yes, it should work but it does not because the expressions are not same right?

[johndpope]

https://github.com/johndpope/IMF/blob/feat/token_manipulation/train_token.py#L177

honestly it should be straight forward

you can see the token / image recreation working here.
https://wandb.ai/snoozie/IMF/runs/xscj3hjo?nw=nwusersnoozie

StyleFeatureEditor was really ambitous integration - you dont need it
https://github.com/AIRI-Institute/StyleFeatureEditor

this is pseudo code. / already implemented code -
i couldn't decide how what to edit - the pose / or eye gaze / emotion etc

you need this data prepared in dataset to force the loss function to find the convergence....

  token_manipulation_network = TokenManipulationNetwork(
       token_dim=512,  # Assuming w_recon has 512 dimensions
       condition_dim=config.condition_dim,
       hidden_dim=config.hidden_dim
   )
           # # Get the original token
           # with torch.no_grad():
           #     original_token = imf_model.latent_token_encoder(x_current)
           
           # # Edit the token
           # edited_token = token_manipulation_network(original_token, editing_condition)
           
           # # Generate the edited image
           # imf_model.set_token_manipulation_network(token_manipulation_network)
           # edited_image = imf_model(x_current, x_reference, editing_condition)
           
           # # Compute loss  
           # loss = compute_editing_loss(edited_image, x_current, editing_condition)
           
           # # Backpropagate and update
           # accelerator.backward(loss)
           # optimizer.step()
           # optimizer.zero_grad()