The gizmo for changing position is automatically placed at the pivot point of the 3d object. Maybe try to place it at the center of the bounding box?