Ground Truth of Entropy Split

[andrewivan123]

I saw that the ground truth is the entropy of the model prediction across the vocabulary. However, the model prediction confidence distribution will change as the training progresses so it will be like chasing a moving target. What is your opinion on this?

[PiotrNawrot]

I agree with your concern, but on the other hand, imho, following fixed distribution of some other fixed smaller LM like it's done in Byte Latent Transformer is not perfect either. In our case you get self-supervision from the same large model - it's true that it's moving - but at the same time it might be more accurate and better adjusted to the larger and more capable LM, as opposed to the statistics from smaller fixed model.

[andrewivan123]

I see. I agree with your statement. The next question would be, for the entropy split, why do a separate module is needed to predict the boundaries when we can just predict the boundaries using the entropy from the large LM itself?