Why is the model called twice here to generate ref_outputs, which are then used separately for the cross-entropy loss and the distillation loss?

[whitesweater]

In model.py, the following code appears:

with torch.no_grad():
ref_outputs = self.codi(input_ids=ref_input_ids, output_hidden_states=True, attention_mask=ref_attention_mask)
ref_outputs_with_grad = self.codi(input_ids=ref_input_ids, output_hidden_states=True, attention_mask=ref_attention_mask)

Since these two forward passes are executed independently, their outputs may exhibit slight numerical differences.

Is this design choice primarily intended to reduce GPU memory usage?

Would it be feasible to use .detach() instead to achieve the same effect while avoiding redundant computation?

[zhenyi4]

Yes, I think using .detach() is a better choice. I recall that in early experiments I compared the two implementations and observed a difference, though this might have been due to an older PyTorch version. Theoretically, .detach() should behave as expected, and I intended to do this operation.