Barely moving queries on the boundary of the frame

[kirilllzaitsev]

Hi, I'm using the online mode of CoTrackerV3 for tracking points in 2D on static scenes with a moving camera. For some videos, I experience the problem of almost static point tracks close to the image boundary:

mahjong1.mp4

For another video with a few similarly placed points, all tracks evolve meaningfully:

room0_blur.mp4

These are the visuals for the first sixty frames with point tracks, interleaved with step=6 for visualization purposes:

Here are the points whose position at frames 0 and 20 has not changed by more than 10 pixels:

kpts=pred_tracks[:,:,np.linalg.norm(pred_tracks.cpu()[0, 0] - pred_tracks.cpu()[0, 20], axis=1)<10]

At the same time, when I use grid-based point queries, the tracks look plausible:

mahjong2.crop.mp4

I wonder what can cause such behavior for keypoints obtained from a detector? I tried changing the resolution of the images, but it had no effect.

[kirilllzaitsev]

Saving the same frames to video and using imageio.v3.imiter surprisingly fixes the problem.

Result with original frames:

tt.mp4

Result with frames saved to a video and read via imageio.v3.imiter:

t.mp4

Although the images appear the same (top=original frames saved as an MP4 video, read with imageio.v3.imiter, and saved with cv2.imwrite, bottom=original frames):

, the pixel differences are tangible:

idx=0
plot_cbar(frames1[idx]-frames2[idx])

I wonder if the problem with barely moving boundary points can be due to the mismatch between the image distribution used for training the model: frames were read from videos using imageio, while I'm loading frames with cv2.cvtColor(cv2.imread(path), cv2.COLOR_BGR2RGB)