I remember the sizes.

Use a global list variable to keep track of sizes of the fonts in current guifont setting, in case user explicitly sets the guifont option without size option specified making size information lost.