Very curious about this work -- does it apply to my use case?

[jbmaxwell]

Hello! I stumbled on your work while searching for some kind of cognitively-inspired associative memory. I was interested in a hierarchical approach, because I have a situation where I want to model something like the use of associations and priming in decision making. Specifically, I was interested in something that could bring up memories on different time scales, to influence decisions on a local time scale. Given the fact that you have both an "upward" and "downward" path, it seems like this should be possible with your approach—i.e., going "up" from the local context to a higher-level of abstraction, on a higher time scale, then coming "down" again in a new but related lower-level context. (Strange description, perhaps, but maybe you follow what I'm getting at?...)

This seems like something HAMUX could do, but I'm I correct in thinking that? Although I'm generally interested in what you're doing, I don't want to spend too much time learning something entirely new if it really isn't appropriate for what I have in mind.

Thanks for your time!

[bhoov]

Hey thanks for your interest. I am honestly a bit unclear about the particular use case, but perhaps a brief explanation of what HAMUX is and how it can be used will be helpful:

HAMUX is a multimodal associative memory framework -- that is, let us assume some domain on which you have co-occurring data of different modalities (e.g., image, label, caption). We can express each modality as a tensor (image, one-hot vector, meaningful vector embeddings). Then, we can design an architecture in HAMUX that "smartly" connects each of these modalities and train the system on the data, even embedding hierarchical latents if you desire. The result is a model that can generate the most applicable image from a (label, caption) pair, or can compute a label or caption from a corrupted image, etc.

Everything in this repository is a research project -- it is a fundamentally different approach to what many in AI are studying nowadays. It is unclear how to train these models particularly well, yet we have high hopes for this class of architectures. That being said, if you are looking for a system where you don't have to spend much time learning or working out the kinks, this software library may not be the right tool for the job.

[jbmaxwell]

Thanks for the explanation. The project I'm working on is a bit "stealth", so unfortunately I'm being deliberately vague...

"...even embedding hierarchical latents, if you desire"... this sounds very much like what I'm after. I basically want a dynamic, associative memory that can plan ahead to reach a goal. The hierarchical part is necessary for the planning. I guess it's kinda basic cognitive architecture stuff, but as you point out, this isn't a central focus in AI right now. Transformers are great, but they have huge problems for the kind of thing I'm interested in (i.e., their fundamentally causal prediction, large, fixed, in-memory contexts, etc.).

I'll think about it a bit more and maybe try to come up with some "toy problems" to test.

[jbmaxwell]

Sorry... more specifically addressing your questions: A "goal" could be a low-level latent and a "plan" could be a higher-level hierarchical latent.

The thing I can maybe see more obviously "missing" is a way of transitioning between latents at the "bottom" level. I guess what I'm thinking is that jumping "up" hierarchically will put you in a state with multiple, related possibilities "below", so it's now a matter of choosing the best one, which I think could be provided by the training.

[bhoov]

Trying to translate what you have said into language I understand. If a "goal" is a controllable/visible variable, then a "plan" is a sequence of variables (visible or latent) that could achieve said goal. In the language of HAMUX we could say that "a goal variable is associated with some sequence of state variables, which we can collectively call a plan variable, which itself may be associated with some observable variables that we detect from the environment".

It sounds like you want to have a plan variable that can evolve in time. That is, we clamp our system state to some desired goal (don't evolve the goal variable over time). Then, as your system dynamically gathers information from observation variables (which are themselves associated with the goal), find the plan that is most associated with observed variables and goal.

If my understanding is correct, I would say that the HAMUX framework can certainly handle it, and indeed typing out that explanation helped me understand the flexibility and application of an associative memory-based approach to your problem. However, what you are seeking is quite the engineering challenge as we don't have much characterization of how to train or use this associative memory framework. How would you encode your "plan" and "goal" variables? Is the plan always a fixed length sequence? How do you plan to use the dynamic information that is encoded by the plan?

[jbmaxwell]

Well, what I'm actually most interested in would be a flexible goal, in the sense that it's not explicitly known beforehand. If you're a writer and think of the general idea of "a plot twist"—you may not know exactly what it should be, but you know it should happen at some point and you have a bunch of "template" ideas from literature. And this knowledge will influence the evolution of your writing. But I think I do follow your framework-specific description, so I'll think that over more to try to wrap my head around it.

I imagine something very dynamic, such that there are many ways to achieve the goal, and hence many plans, and the goal is fairly abstract—like the "plot twist" above. It may only become "specific" as you get closer to its actual realization. Liane Gabora uses the term "half-baked idea" to refer to this kind of "goal" (in creativity theory). But I have a feeling its position at a high level of hierarchy would ensure this abstraction (generality) anyway... It's kind of a continuous guide over a sequence of states, rather than one final state.

I think the lowest level representation would actually be a sequence, probably fixed-length (not ideal, but easy), represented as a latent. To me, the "goal" would probably be at a higher level of the hierarchy—not the bottom (though probably not the top either). So the last state would be an expression of the goal, but not the goal itself. The "plan" is also highly dynamic, but yes, it is generally a sequence of these low-level latents (sequences). I'm honestly not so worried about "following a plan", but rather that the sequence of states generated are all somehow "supervised" by the higher-level goal. Turning to cognition again, the two different "goals" of getting a cavity filled and having lunch are both sequences of actions guided by the high-level objective. The sequences can have variations, but there is a general template. And again, this is all somewhat "automatic", to me, if the hierarchy is working properly...

Is it possible for a second layer of HAMUX to represent something like a relationship between two latents? So that presenting one latent would associatively recall the second?

[jbmaxwell]

Is it possible to pm you on this, or email?

[bhoov]

Of course, feel free. I certainly doubt I will have all the answers to your questions, but I can certainly elucidate what HAMUX can and can't do. in general I am growing in my confidence that it is a very general Associative Memory framework that can be applied to many situations