Use dusql to create a table of file info

[aidanheerdegen]

When nccompress times out and is re-run on the same directory it ends up trying to open every file to check if it is netCDF. This is slow and inefficient if it has already done that before.

Could use dusql to create a file info database, and create a table that stores the format information, and if it has already been compressed, so the work is not duplicated when it it is run again.

[ccarouge]

Can it handle independent changes made to the directory well? I mean would nccompress need to run a dusql update each time it runs to make sure the database is up to date? How long would that take? How would the update keep the file info table in sync? Maybe those are trivial questions when dealing with databases but I don't know much about databases.

[aidanheerdegen]

Those are good questions and I don't know all the answers myself. At the very least dusql can be run and the database updated but we'll still know if file modification times have changed since their status was determined last time it was run (if that information is saved in this other proposed table, which clearly, it should be).

[ScottWales]

We could add a minimum age to dusql.scan(), so that it only runs a full scan if the last scan was > 1 hour ago or whatever.

The scan takes however long it takes to os.walk() the directory tree, plus however long it takes to ID the file type for newly added files (file ID isn't currently implemented, but should be doable from looking at the first few bytes of the file)

[ScottWales]

See coecms/dusql#29