When in the cockipt view, with a controller axis assigned to panning the camera, it feels like the pan movement speed is greatly influenced by the current fps. If the camera is facing a direction with more complex scenery it's not unusual to see fps drop from 60 to the 30's in the sim, and this results in a sharp slow down of the rate that the camera is panning.
The movement range should be based on the time between flight loop calls.