While using the hand tool (panning mode) comments are triggered and dragged around the canvas

While panning, either using hand or middle mouse button, I'm able to drag comments that are unrolled on hover and drag around the canvas. That happens often by mistake and Ctrl+Z does not work in this case.