This causes a lot of frustration, because even though most players have most source games, most servers do not, making the [physical] models useless. Holograms have no physics, and a client who doesn't have a specific model will see an error regardless, so there is absolutely no point in not allowing the models to be used by those who have them (if modelany is enabled, of course).
Specifically, remove this from line 204 of the hologram.lua component file.
!util.IsValidModel( Model )