Adding fisheye calibration

[aruna-ram]

Hi!

Would you be interested in adding calibration for fisheye lenses?
I was getting a strange undistortion via caliscope but I think it's because I'm filming behaviour wide field but at smaller distances to the target - which gives bad barrel distortion.

I used OpenCV's fisheye module to do intrinsic calibration and got a better undistorted image.

It would be great to continue using caliscope for calibration and triangulation but with a fisheye lens!

[mprib]

Hello Aruna!

Sounds like a pretty sensible thing to add in. I'm a little slammed this week but may have time to work on this next week. Implementing this will require changes across multiple stages of processing so it's definitely something I want to be careful about (there is always a risk of unintended consequences). Any implementation will likely just be configured optionally through the config.toml to start. If you have any sample data, that would definitely help grease the wheels of development.

Just making some notes here for my own reference while it's top of my mind:

1. include toggle for fisheye parameters in intrinsic calibration:

   caliscope/caliscope/calibration/intrinsic_calibrator.py

   Line 217 in c896d05

   self.error, self.mtx, self.dist, self.rvecs, self.tvecs = cv2.calibrateCamera(

2. Include toggle for fisheye point project in bundle adjustment:

   caliscope/caliscope/calibration/capture_volume/capture_volume.py

   Line 197 in c896d05

   cam_proj_points, _jac = cv2.projectPoints(object_points.astype(np.float64), rvec, tvec, cam_matrix, distortions)

3. Make undistort_points more generic: for triangulation
4. CameraData object would now have a fisheye parameter
5. Configurator would need to read and write fisheye parameter
6. Stereocalibration as part of initializing the array prior to bundle adjustment would need to get updated

Is it the case that using fisheye cameras are an all or nothing thing? Like all cameras will be fisheye?

I ask because the stereocalibration in opencv (which is a critical pre-processing step for the bundle adjustment initialization) has different versions for fisheye vs non-fisheye and does not appear to allow mixing by default.

And just so I understand the use-case better, it seems like this is a response to the small capture volume. Is that true? It seems like an interesting difference with human subjects research, and I definitely would like to make this more useful for people broadly doing motion capture, regardless of scale.

[aruna-ram]

Thanks for the response!

I have calibration videos, so I can send those over to you - is there a preferred way?

The lens that I am using are not technically fisheye but in order to capture a large field of view with the cameras a short distance from interest areas, I change the lens zoom and this seems to give some barrel distortion.
From this experience and the undistortion via the opencv fisheye module, I think there might be a "spectrum" (not fisheye or not) but the user might need to eyeball whether they want to use a pinhole model or projection model.

let me know if you need more information!

[aruna-ram]

To keep using the standard pinhole stereo calibrate function, the intrinsics could be found via the fisheye module and then the extrinsic videos could be undistorted and then the standard stereo calibrate function could be used?

I think someone might have tried out this logic: https://stackoverflow.com/questions/74751359/opencv-stereo-camera-calibration-support-mixed-camera-models-normal-fisheye

[mprib]

Thank you. For submitting videos, would you be comfortable just dropping them into the conversation here? The discussion thread should be able to received both .mp4 and .zip files.

Thank you for the detail about the lenses. I'm having recollections now that once a camera's FOV gets above 100 degrees, the Fisheye approaches are preferred.

Your thoughts about the stereocalibration workaround are good. I want to reflect a bit more on how to implement something like that in a way that will consolidate the undistort calls. It's one of the handful of things that gets re-implemented a few times in different contexts in slightly different ways for various reasons. It has always bothered me and this is good opportunity to clean things up. I want to tread carefully so this may take some time to reflect on.

[mprib]

hi @aruna-ram,

just wanted to follow up. Not sure if this is still on your radar. If so, but you don't feel comfortable providing video files on the open discussion, you can email me at prible - at - utexas dot edu and we can sort out another mechanism (and I'll be sure to keep your videos out of the repo).

Mac

[aruna-ram]

Hi @mprib, I'm so sorry for the late response - it's been a busy time! Thanks a lot for following up.
The videos are too large to share over here. I'll send you them via email with the specifics of the charuco board (I used a workshop made one, but it's a legacy pattern).