A and D needing to be normalized

[helenegu]

Hi!
I'm reaching out as part of a "gamifying alcohol risk identification" data science team at Duke working with active inference. I'm a huge fan of pyMDP. We plan on using it to create simulations for multiple alcohol risk games, and have been trying to figure it out these past weeks.
Right now, we've been translating a MATLAB code simulating Cognitive Behavioral Therapy that uses mdp. We noticed that the a matrix and d vectors were scaled by constants to slow down learning/make it so that new observations had less of an impact.

according to the paper
"To simulate confident initial prior beliefs in danger and expected observations under approach
vs. avoidance, concentration parameters in 𝑑 for these beliefs were multiplied by 50 (e.g., [1 0] =
[50 0]; [.5 .5] = [25 25])"
and
" To prevent learning with respect to other mappings irrelevant to our simulations, all other
concentration parameters in 𝑑 or 𝒂 were multiplied by a very high value of 128"

However, when I tried to do the same using pymdp, I was told a and d need to be normalized.
Is there any way to achieve the same effect using pymdp?

[conorheins]

Hi @helenegu,
Thanks a lot for your question!

Importantly, pymdp does not require artificially up-scaling a and d to disable learning of the A and D arrays. In order to disable learning of these model parameters, all you need to do when initializing an Agent() object is to pass in the constructor variables pA or pD (or pB, in case you're also learning B) to None, which is the default value anyway. Note: pA, pB, and pD are the equivalent of a, b, and d in MATLAB versions of POMDP active inference.

However, if you want to enable learning, but just keep it with a low learning rate, then you may do the scaling exactly as you described (multiplying by 50 or 128 or whatever), but you should perform that scaling on the pA and pD arrays, not the A and D arrays. pA and pD are Dirichlet parameters, that act as priors over the (categorical) parameters of the A and D distributions.

We have a nice utility function dirichlet_like() that allows you pass in an optional scale parameter that will do exactly this, e.g.:

D = np.array([0.5, 0.5]) # initialize a flat prior over hidden states (categorical distribution, must be normalized)
pD = utils.dirichlet_like(D, scale = 50.0) # create a Dirichlet prior over the categorical parameters encoded in `D` with scale `50.0`
print(pD) # should give a vector that looks like `[25.0, 25.0]`

The critical point being that pA and pD (the equivalents of a and d) need not be normalized, since they are Dirichlet parameters -- but the categorical A and D do have to be normalized.

[helenegu]

That clarifies a lot. Using pA and pD worked, and we figured out our other problem was using "pa=A" instead of "pa=A.copy()", so changing pa actually changed our capital A matrix too (resulting in the error).

Thanks so much for the quick answer!

[conorheins]

Happy to help :) Also, regarding the line pa = A.copy: I would advise using deepcopy instead of copy when making a copy of the A array. When you use A.copy(), you will create a shallow copy of A, which will only ensure immutability of A when you change pa, if A is not an object array (which in the case of pymdp, is rare).

Alternatively, you could use the dirichlet_like() function (mentioned above) which also creates a separate array from A. So something like

my_agent = Agent(other_args ... , pa = utils.dirichlet_like(A))