This what I can come up with at the current time. Please comment below if you'd like to add to this list.
- Generate a higher level data file
- Generate a hash of the file and add it to a hash list
- Put the time series data inside a database (such as amazon timestream) for the grafana dashboard
- Update a metadata database