Huging Face transformer

[jaideep11061982]

How can we use it for NLP transformers

[chenjoya]

Hi, thank you so much for your concern!

DropIT can be applied to NLP transformers, but we do not have experiments for them (as I am a computer vision researcher).

DropIT can be used for fc layers in transformer network (you know, there is so many fc layers in transformer).

I would like to tell you how to do:

(1) Build your NLP model, for example, nlp_model = ....

(2) Use the to_dropit function https://github.com/ChenJoya/dropit/blob/89f42047775f00a84589b48be14a7509370b5a0b/layers/dropit.py#L193 for this network. You may follow https://github.com/ChenJoya/dropit/blob/89f42047775f00a84589b48be14a7509370b5a0b/models.py#L25:

D = 2**14 # this is the best parallel top-k value in my A5000 GPUs, I think you can use it
gamma = 0.9 # you want to drop 90% intermediate tensors
strategy = "fastmink"
dropiter = DropITer(strategy, gamma, D)
to_dropit(nlp_model, dropiter)

(3) Then your model is transferred to its DropIT version. You can use it for training, testing.

If you have any problems, feel free to contact me ;)