Enhancement - Distortion models and fitting stability

[Microno95]

Currently the calibration scheme underlying Calcam uses the OpenCV distortion model which works for many cameras, but with the MAST/MAST-U cameras, which use a wide-angle lens, calibrating higher order radial terms (k2 and k3) is quite difficult. This is partly due to the lack of image features visible on the edges of the image for pairing with the CAD geometry, and likely is not helped by small discrepancies between the CAD model and the real machine. This could also be partly that the distortion model used by OpenCV is unsuitable for these cameras outside a small radial region around the optical center of the imaging plane.

For example, with MAST-U, I have gone up to 160 point-pairs along with a constraining single frame calibration as a starting point and the same set of 160 points achieves 2.8px residual with k1-only, but a 32px residual with k2 and k3 enabled.

My suggestion would be to include tabulated calibration as a feature so that the underlying distortion map is computed directly instead of fitting the OpenCV analytical model which may help expand the domain in which the undistorted image is valid.

[ssilburn]

Hi Ekin - thanks! Yeah improving the distortion model capabilities is definitely something that would be good, some of the JET systems too have distortion that isn't super well captured by the openCV model Calcam is using now. It's interesting that the residual goes up with the higher order terms enabled, I don't remember seeing that happen before! (But also don't think I tried with so many points).

A first step probably will be to investigate the other distortion model parameters already in OpenCV, but not currently accessible through Calcam, and see if they help - having to come up with new ways to quantify & fit the distortion seems like it'd be a lot of pain. With something more arbitrary like a tabulated approach I think we'd still have to figure out a sensible way to represent the distortion in terms of smooth functions to be able to interpolate / extrapolate in a sensible way to arbitrary points in the image? And this would be a really big change to how all the numerical stuff works in Calcam which would be a big amount of work to implement & test, unless we can find someone with a lot of time to spend on this I don't think such a fundamental change is realistic at least in the near-term. But, enabling for example the quotient of polynomials type distortion model already in OpenCV (but Calcam doesn't use it), would be much easier if it helps.

Also do you get similar behaviour with both the fisheye and rectilinear distortion models? (If the fisheye fitting even works...it seems to be very twitchy and have problems that would also be good to fix!).

It'd be helpful for testing solutions for this also if you could please send the .ccc file for the example you talked about (on here or by e-mail is fine), so I can test out some potential solutions

[Microno95]

Right, that makes sense. I have never found the higher order terms to converge, so perhaps it is related to how I'm pairing points in the image and model, I've sent an email with the calibrations and CAD model I've used.

As for representing the distortion, I was thinking that perhaps using an elliptical coordinate system (ie. parameterise the distortion in terms of rotated ellipses centered at the optical center) with the distortion map stored as delta values from the center (e.g. to get the actual distortion, you take the cumulative sum from $0$ to $R$ along $\theta$) with some constraints on positivity of the deltas. Guarantees radial monotonicity and can account for rotationally varying distortion (through rotation of ellipses).

But I digress, I didn't realise that there were additional distortion coefficients/models that could be enabled within OpenCV that may be useful here so it makes much more sense to start with those. I will see if I can find time after APS DPP to look at that myself.

I have maybe had the fisheye model converge once, but in general, no matter the number of point-pairs and coefficients enabled/disabled, it has not converged.

Let me know if you need anything else!

[ssilburn]

Thanks for sending over the example files! I'll put updates here if I have any breakthroughs testing with them! In the meantime yeah having functions of ellipses sounds like a reasonable way to represent it, if we don't find anything in OpenCV already sufficient

[ssilburn]

OK I had a look, and the issue with the high fit errors when enabling more distortion terms appears to be just the stability of the OpenCV fitter, and it keeps falling in to a local minimum with a bad fit because of where the initial guess for the fitting happens to be.

To work around this, I added the initial guesses for the focal length and principal point to the fit as GUI inputs with the fit options, instead of being decided by Calcam and hidden from the user. By using an initial guess for the focal length already much closer to the correct answer (from the low order fit), it's possible to get your calibrations to fit much better with all of the parameters enabled.

This change is now done on the master branch (currently 2.16.dev2), so will be on the next release. Also in this dev version, if you open the file with the existing calibration, it will use the current focal length of the fit in the file as the initial guess when doing more fits, so it should be seamless and easy.

By using this and also tweaking and removing some of the points (and actually turning off the included previous calibration), I got your fit down to <2px RMS error with all the terms enabled. I'll e-mail the modified calibration file tomorrow.

[Microno95]

That makes a lot of sense, and that's amazing! <2px RMS is excellent for that many points, I'll be on the look-out for your email. Thanks so much!

[ssilburn]

This is now release in v2.16. I might leave this issue open, though, because there are still a bunch of model features in the OpenCV code that I'm not using in Calcam and might be helpful for some calibrations!