It seems like the spotlight class (and possibly other classes) is lacking threejs's target property. Hence it always points towards the origin, regardless of the assigned transformation.
I'd love to help fixing this, but on first look it seemed like its quite deep within viser's machinery. If you can point me to the right places I'd be happy to put in a PR.