Scipy curve_fit fails after running for a long time

[jadball]

Hello,

I'm trying to follow this process with our Pilatus 2M CdTe detector, but the curve fit doesn't seem to converge:

python3.7/site-packages/scipy/optimize/minpack.py in curve_fit(f, xdata, ydata, p0, sigma, absolute_sigma, check_finite, bounds, method, jac, **kwargs)
    787         cost = np.sum(infodict['fvec'] ** 2)
    788         if ier not in [1, 2, 3, 4]:
--> 789             raise RuntimeError("Optimal parameters not found: " + errmsg)
    790     else:
    791         # Rename maxfev (leastsq) to max_nfev (least_squares), if specified.

RuntimeError: Optimal parameters not found: Number of calls to function has reached maxfev = 21200.

I wanted to make sure I wasn't making any obvious mistakes? The only significant difference I can see is that we have many more rings at this distance and wavelength - the rings get pretty dim so are harder to pick up at the edges. Can provide raw scan data if needed. I've pushed to my fork - https://github.com/jadball/EigerPowderSpatial/ so you have the flts and parameters (I got the rough beam centres from the detector translation motors and then fitted the distances, tilts and beam centre with ImageD11).

The hkl2 histogram I get looks very different too:

Thanks,

James

[jadball]

I have found the cause of one of the problems: the pixel size in the transform.compute_xyz_lab function call in calchkl2 is in microns, not millimetres. This has let the fit run: unfortunately, it doesn't seem to be at all convergent:

Any ideas?

Many thanks

[jonwright]

Hi James, I think there is an error in the sigma being sent to curve fit. The idea was to down weight the weaker peaks, so the sigma should get bigger as the intensity gets smaller. Something like this:
sigma = np.sqrt(avgi/avgi.max()) # relative weights
... might be better?

At 70keV you might have some incident angle effects that the beam goes further into the CdTe at lower 2theta angles. Going to a larger sample to detector distance to have fewer rings would reduce this effect. Or a lower energy. But maybe things are already improved enough compared to having no correction?

[jadball]

Hi Jon,

Thanks for your suggestion - yes we are relying on fairly dim peaks to get us decent module coverage with this dataset.
I've tried your new sigma, it looks to have improved the fit somewhat, but the ycalc histogram is still worse than before the fit:

Before:

After:

[jonwright]

I think it is improving, try over plotting on the same y axis. Also try looking at the two theta azimuth plots and see if you still have jumps at the module boundaries?

[jadball]

Hi Jon,

Here's a before vs after of a two theta vs azimuth plot.

Before spatially correcting the flt:

After spatial correction:

I also wanted to check that I'm using the spatial correction file properly (so I can use this with other data):

df = fabio.open("e2dx.edf").data
ds = fabio.open("e2dy.edf").data
t = transformer()
t.loadfiltered("eigerSpatial/109673.flt")
c = t.colfile
c.sc[:] = ds[np.round( c.s_raw[:].astype(int) ), np.round( c.f_raw[:].astype(int) ) ] + c.s_raw[:]
c.fc[:] = df[np.round( c.s_raw[:].astype(int) ), np.round( c.f_raw[:].astype(int) ) ] + c.f_raw[:]
t.loadfileparameters("eigerSpatial/109673-fit-corrected.par")
t.compute_tth_eta()
c.writefile("eigerSpatial/109673-distortion-corrected.flt")