VRT component files - are they parents?

[neilflood]

@gillins has rightly raised a question mark around the notion of whether one should add history to a VRT file, and whether the component files of the VRT are themselves parents of that VRT, for the purposes of processinghistory.

It seems fairly plain that doing so naively has the potential to create ambiguities. The VRT contains no data itself - the component files are used directly when reading the VRT. This means that a VRT could be created from a given version of a component file, and a full history included, but then if the component file were updated, the new data would pass straight through when the VRT was read, but the embedded history would not reflect this.

So, some other approach is required, which would probably treat VRTs as a special case within processinghistory. It is questionable whether the notion of a component file should be handled the same way as a parent file - the relationship is, in some ways, fundamentally different.

One possible scheme which comes to mind is to store in the VRT only the metadata for the VRT itself, no parents allowed, but instead a list of component files. This would be just the filenames, no timestamps. When the history for the file is read, the component files could be read for their history, on-the-fly, and this would then be available as ancestor metadata for any child of the VRT, while still being distinguished as components rather than parents of the VRT itself. This kind of matches what happens when reading a VRT for the data - it is read on-the-fly directly from the component files.

Not sure if the list of component files needs to be stored explicitly, as it is available via Dataset.GetFileList(). Similarly, should the metadata have an explicit flag to say that it is a VRT, or should one rely on the fact that the VRT driver is in use?

I am still exploring this possible solution for any traps, so it does not yet exist except in my head. Any thoughts are welcome.

[gillins]

Yes, I think a special case is warranted. In my mind there would be a space for the VRT but there wouldn't be anything in there to speak of - just a link to the inputs of the VRT.
I'm not sure what history data you would store in the VRT - perhaps how it was created? Perhaps we could have a helper function for creating VRTs that takes the same parameters as gdal.BuildVRT but also burns these options into the VRT's metadata, but as you say didn't do anything with the input files at this step.

[neilflood]

I have been pondering this, and finding all sorts of potential traps. I get scared with what might happen when a VRT is the parent of another file, and wondering how the component files are represented in the history in the child of the VRT. So far I have only managed to confuse myself, but there are definitely question marks over all this.

I did remember a problem we used to run into (back in the RSC version) when adding history to a big mosaic. If we listed all the input files as parents of the mosaic, the resulting metadata would easily overflow the space available in GTiff format. So, we had to just add an explicit entry to the userDict, which gave a list of the input filenames, and leave it at that. Thus any child of the mosaic would not have the grandparent relationships added, and the mosaic formed a kind of barrier in the ancestry chain. It was not entirely satisfactory, but also never seemed to cause any major problems.

This matter with VRTs has some things in common with that situation, although different in other ways. So, I wonder whether a VRT may also represent a break in the ancestry chain, so that child files of a VRT are unable to store the ancestry before the VRT.

Another option is that a VRT is never represented in any ancestry at all, and the child of a VRT just uses all the components as parents, skipping the VRT altogether. That doesn't seem quite right, though.

It could get even worse if there was a VRT with other VRTs as parents, and the real files would then be great-grandparents of any child of that VRT.

Sigh...... as I say, so far I am just confused.

[gillins]

Ah yes, well the particular case I was thinking of there would be ~1000 input images to the VRT. As you say this starts to get very large...
But also - very useful. Would allow us to track whether the correct version of an input file is present etc. Let's ponder this a bit more.

[neilflood]

My newest thought is that we write history into VRT files (i.e. when writing with the VRT driver), but it is treated in a special way. The "CURRENTFILE" metadata is written as normal, but metadata for all ancestor files is written as a special value "DEFERRED". The set of ancestor files is restricted to the component files of the VRT, and any explicit parents list is ignored (or maybe raises an exception).

When writing to a non-VRT file, any "DEFERRED" metadata is read directly from that file just before it is written. Before writing, it remains as "DEFERRED", and that is all one sees when viewing the history for the VRT. This would include grandparents and further ancestors as well, thus taking care of VRTs which are parents of a VRT, which is then a parent of a non-VRT.

This behaviour aligns more closely with the way a VRT works for the data itself, and I think should satisfy all our concerns above.

I think it is definitely of value to write history into a VRT. It stores all the usual automatic fields, so we record the command used to build it, and the date, GDAL version, and the user who created it, all of which are as useful in a VRT as in any other file. Also, when the VRT is an ancestor of a non-VRT file, this information is copied into the non-VRT file, where it could be the only record available down the track.

I will do some testing and see if this makes sense in reality, not just in my head.

[gillins]

Thanks @neilflood sounds good. But - are you saying that the VRT won't break the ancestry chain with this approach? I guess there could still be an option to break the chain if the user decided things were getting too large....

[neilflood]

Yes, that's right, it will attempt to keep the chain unbroken.

In the old version, as far as I remember it, there was nothing to deal with this automatically. For large mosaics, I just would deliberately not include the pieces of the mosaic as parents, but rather as an explicit list in the userDict.

Not sure what the right option is here, though. What do you think?

Still playing with it, finding gaps in the logic. Currently stuck on what to do with the deferred ancestor relationships at the grandparent level and beyond.

It's complicated......

[gillins]

I think the answer will depend on how many inputs to the VRT there are. I think by default do what you are suggesting, but have an option to disable this if the user decides it is getting too big...
I suspect you'll have to go all the way up the chain looking for DEFERRED (as I understand it anyway).

[neilflood]

re: up the chain. Yes, that's right, which is OK for the metadata at each level, but gets more complicated for the relationships. I think it will be OK, just working through the details.

re: too big. OK, I will keep looking for a suitable approach.

[neilflood]

re: too big. With the "deferred" approach, it means things won't actually get really big in the VRT itself (because most of it is deferred), just in the downstream non-VRT products.

[neilflood]

Fixed by #12