Some differences in mxOPAQUE CLASS and Object metadata, particularly for Strings

[foreverallama]

Hi!
I was looking at how objects are stored in MAT files, and came across this repository and your explanation in MatFileHandler/objects.md. When trying to replicate, I observed some small differences in the object metadata.

Some general differences I observed:

• Instead of 6 offsets, there are 8 offsets in the list.
• The fieldContentsID in Region 4 is numbered starting from 0.
• Region 3 has a slightly different structure as follows: (classID, 0, 0, X, Y, objectID). I'll get into what X and Y are doing below

Some differences for string class:

• Region 3 had a slightly different structure. As expected, it was a block of six 32-bit integers, one block for each object with data as (classID, 0, 0, X, 0, objectID). X here I will call as stringObjectID, as it started with 1 and incremented for each string object in the file. On the contrary, for user defined classes X is set to 0. Instead, the fifth field (Y as mentioned earlier) is set starting from 1 and incrementing for every user-defined class in the file. I also tried with datetime which was using the Y field. My guess is these two fields are some type of internal identifiers for certain categories of objects.
• Region 2 was always empty for user defined classes. However, for string, Region 2 was present, and structured exactly the same as Region 4 would be, i.e., three 32-bit integers with the format (fieldID, 1, fieldContentsID). Only one field for each string object is present. No String related data was present in Region 4. In some examples, fieldID was set to 5, and in others it was set to 1. Need to take up a few more examples for this.
• The fieldContents cell for strings was interesting. The array flag for this was set as mxUINT64_CLASS with dimensions [1, (5+k)], where k depends on the length of the string. The first four 64-bit integers was set as [1,2,1,1], and the fifth integer specified the number of characters in the string. The next k columns contained the actual string contents, which is null terminated and padded to 8-byte blocks. However, the content was stored as UTF-16 characters within these 64-bit columns. Hence, each column essentially stores 4 characters.

I was looking for some help to decode what's happening with strings here, or if there's something else I could be missing. I'm looking to incorporate more objects/examples to help break this down.

The input data I used:

astring1 = "Hello!"
dt = datetime('today')
obj3 = myclass(30)
obj6 = myclass(myclass(5))
stringVar2 = "Goodbye!"

% myclass is defined with 3 properties - message, num, aeroplanes
% myclass.message is set to the input argument

[mahalex]

@foreverallama Hi!
Thanks for the information, this looks very interesting! I noticed some additional things while doing the latest beta with enumerations support, but I didn't update the description in objects.md yet. I need some time to think about all this, and to do more investigations.
Thanks again!

[foreverallama]

Sure! I'm also looking into decoding what these flags mean in more detail using other objects, and maybe expand upon the existing documentation as well. I'd be happy to help out along these lines if needed 😄

[mahalex]

@foreverallama Hi again! I mostly confirmed your findings, and the only thing missing now's this mysterious fieldID value of 5 for string data. Do you have any examples with that? For me, the fieldID value for strings is always 1.

[mahalex]

Oh, and I moved the MatFileHandler repository to another hosting, so you can follow it here.

[foreverallama]

Hey! Thanks, I'll check it out. Meanwhile I got a whole lot of new information, which I've documented over here. Do check it out. Some basic differences I think are as follows:

1. The very first digit in cell{1} of FileWrapper__ is I think a version number. You mentioned you had 6 offsets, but from the MATLAB version I'm using (2023b) I get 8 offsets. This led me to believe its a version indicator. For me, this number is set to 4.

2. I was slightly wrong about the field contents - the last integer is a dependency ID, and not the object ID. i.e., (class_id, 0, 0, X, Y, dep_id). I've further noticed that X is set for certain datatypes like string and timetable which use the any field. This I guess relates to some common superclass. The other two zeros I don't know yet, but I think there's a possible third type of object here (I've discussed this in more detail in the docs).

3. I made a mistake with the string there, the first value is the field_idx. I mistakenly referred to a different file when checking the data. Additionally, I tried to "hack" strings similar to how you did for object references. This didn't work out however, MATLAB decodes it only for string class.

Got way more information playing around with Java/COM objects, as well as stuff like dynamicprops and eventListeners. Would love to fill in some of the remaining details though!