Question regarding MSLL sensitivity to IDP scaling in PCNtoolkit

[lhzktt]

Hi

I am a new user of PCNtoolkit and currently using BLR + B-spline + WarpSinArcsinh with default parameters to fit normative models. I observed that the numerical scale of the response variables has a strong impact on MSLL, while almost no effect on other evaluation metrics.

The data I used are diffusion metrics (FA, MD, AD, RD) fitted separately. The typical ranges are:

FA: ~10⁻¹

AD: ~10⁻³

MD and RD: ~10⁻⁴

When fitting the models using the original data, the SMSE was slightly below 1, but MSLL was very large. Specifically:

For FA, MSLL was around 1–3, depending on the standardization method (scaler="minmax" or "standardize").

For AD, MD, and RD, MSLL ranged from 7–9+, also depending on the chosen scaling method.

When I linearly scaled all IDPs to roughly 10⁰ (e.g., 0.00017 → 1.7) and refitted the models:

MSLL dropped significantly to around −1 to −2 (for minmax)

RMSE increased proportionally

Other metrics remained nearly unchanged

Based on these observations, I would like to ask:

Is this scale-sensitivity of MSLL expected behavior or a potential issue?

Can the models fitted with the original and scaled data be considered the same model?

Are the results obtained from the scaled data still valid and reasonable?

Environment: Python 3.11.14, PCNtoolkit v1.1.2, WSL Ubuntu 20.04

Thank you very much for your time and assistance. I look forward to your guidance.

Best regards

Below is a comparison of the evaluation metrics between scaled and unscaled data (left: scaled, right: unscaled, minmax).

[NilsWinter]

Hey,

I stumbled upon the same issue lately when running our own analyses. We were a bit confused about the mismatch between all other metrics (which indicated the fit was okay) and MSLL, which was positive for all of our ROIs.

I've gone through the code, and from what I can tell, I think the mismatch comes from the fact that negative log-likelihoods (NLL) for the actual model are calculated based on scaled data (if you are using e.g. outscaler='standardize'), but the NLL for the baseline model is calculated on the raw Y values.

MSLL is then calculated by $MLL_{model} - MLL_{baseline}$. However, if I am not mistaken, log-likelihoods are not unit/scale independent.

The log-likelihoods are calculated based on this formula:

$$\log p = -\frac{1}{2} \left( \log(2\pi) + \log(\sigma^2) + \frac{(y - \hat{y})^2}{\sigma^2} \right)$$

The penalty term for the uncertainty of the model depends on $\log(\sigma^2)$. If the data is not scaled to unit variance (e.g., if it has very small or very large values), $\sigma^2$ will depend on that scale.

• E.g. if units are large, $\sigma^2$ is large $\rightarrow \log(\sigma^2)$ is large $\rightarrow$ likelihood drops.
• If units are small, $\sigma^2$ is small $\rightarrow \log(\sigma^2)$ is negative $\rightarrow$ likelihood rises.

This explains why you saw such a difference. Again, I believe the code calculates the baseline NLL on the raw unscaled values of Y, but the model NLL is based on the scaled values, making them incomparable.

In your experiments, when you scaled your data beforehand, the issue went away because you forced the raw data to be on the same scale (~1) as the latent model. Since other metrics (like SMSE or R2) are scale-independent, they remain unchanged (RMSE changes of course with different scaling).

I think it would make sense to calculate the baseline NLL on the scaled data as well to fix this. I am not one of the developers, though, so maybe they can jump in to confirm!

Best,
Nils

[amarquand]

Dear Nils - thank you for this. Yes, I think you might be correct. We will look into this and try to address this in the next release.

Best,
Andre.