As far as I can see there's no easy way to modify the payloads for AuthCode and AccessToken/RefreshToken.
It'd be nice if there would be an event about the user being used in the request along with the entities free to modify, the custom data could just be stored as json in the entity itself.
As of now I'm simply creating an additonal entity, linking it to the AccessToken entity (or AuthCode entity), and making my own AccessTokenRepository or AuthCodeRepository where I modify some functions. (btw shame those classes are final)