Link mapper & cache system (and sourcing?) #3
Description
The goal of this system is to identify how data is "flowing" in the system and particularly where it's duplicated. In essence this duplication is mediated by explicit references in the description, but a struct doesn't know that something else is using its data without a link mapper. Being able to look at this flow allows for things like: deciding which data should be exported to reduce duplication the most, updating displays when related data changes, and managing cache invalidation.
The links that are established should be able to describe how the data is mirrored between the points of duplication in these broad ways:
- reversible: lossless, one-to-one translation
- recoverable: 100% quality but not bit-for-bit the same, e.g. ordering being lost
- lossy: lost quality, but can be reversed, such as ogg -> mp3
- irreversible: the translation is one-way, and cannot be recovered into a useful state when reversed
The links should be properly ordered, for instance, if something is coming from a ROM into one key, abc, then that key is referenced by another key, def, the data is flowing like: ROM -> abc -> def when unpacking. The packing case depends on which of the two keys are defined, but the ROM is the final destination of course.
At some point, we should be able to generate an interactive map that explains the flow of the data. That may be in another ticket, though.
Sourcing is a means of describing where data came from in a more concrete manner and may be necessary for the implementation of the link mapper. Such as for data unpacked from a ROM file, it would specify the filename, position in the file, and size of the data it extracted. For a value statically declared in a description file, it would specify that filename, and the line and col number for where it can be found (col is more complex than just a number, as it should handle tabs more robustly than other systems). For this ticket, just implement whatever's necessary to make the link mapper work. If no source information is necessary, that's fine.