Replace .f37 / .EEPROM with a single file format

[MaxBaex]

The following idea was proposed by @Simsys and discussed with @realtimepeople , @MaxBaex:

The idea here is that the config data (EEPROM.ini) and the flight recording *.f37 are simply written to the file one after the other. A long random number (64 or 128 bits) is used to switch between the two formats. The file format then looks like this

a. Magic Nr
b. Identifier of the file format
c. Magic no. for EEPROM flight data
d. EEPROM data in today's ini formats
e. Magic Nr for f37 flight records
f. f37 data
g. Further EEPROM and f37 flight records as required
h...

Advantages

• simple
• very close to today's formats
• easy to dismantle

Disadvantages

• each editing of ini values triggers a complete saving of the ini data
• minimal risk of random numbers occurring in the binary data and causing problems with interpretation.
• real-time risk: Writing the configuration data is relatively complex and could violate timing limits.

As it is not known how many f37 data records will be generated, the length of such a range is not fixed. When sorting apart, you search for the magic numbers and only these are used for decomposition. You can split something like this very easily with Python using the split() method.

The new format shall be identified by the file extension .lfd "larus flight data"

https://github.com/larus-breeze/sw_tools needs to be updated accordingly so that the tools there understand the new format. Implementation shall make the data handling more robust as all data is incorporated in a single file.
This will later also help if we provide analysis tools to end users to analyze flights.

Are there more pro and con arguments to be considered before an implementation?

[Simsys]

@MaxBaex Thank you very much for the initiative.

After some discussions with @realtimepeople , I have come to the conclusion that this proposal poses a risk in terms of implementation. @realtimepeople noted that recording the data is time-critical, as there is a lot of data to process. Longer “thinking times” are particularly problematic.

And that is exactly what happens with the above proposal. If the configuration data has to be written as a whole, it must be read, possibly converted, and written. That takes time. I will add this disadvantage to the above list.

[Simsys]

There was also suggested implementing a sequential key value store as an alternative.

The format is defined as follows:

• Storage is in 4-byte units. Smaller data packets are not permitted.
• Each data record consists of a key u32 and a value that is nx4 bytes long, where n is at least 1.
• The key uniquely defines the length of the value.
• New data points receive new keys.
• The header format still needs to be defined. We could keep it compatible with the firmware images.
• A file then contains the following data:
  1. Header (mandatory)
  2. Config Data (0x0200..., mandatory)
  3. Magn Calib Data (0x0300, mandatory)
  4. In any order until the end: .f37 data, config data, magn calib data, other events

The keys could be defined as follows, for example:

• 0x0100 .f37 Data row
• ...
• 0x0200 SensTiltRoll
• 0x0201 SensTiltPitch
• 0x0202 SensTiltYaw
• 0x0203 PitotOffset
• 0x0204 PitotSpan
• 0x0205 QnhDelta
• 0x0206 MagAutoCalib
• 0x0207 VarioTc
• 0x0208 VarioIntTc
• 0x0209 WindTc
• 0x020a MeanWindTc
• 0x020b GnssConfig
• 0x020c AntBaselen
• 0x020d AntSlaveDown
• 0x020e AntSlaveRight,
  -...
• 0x0300 magn calibration data
• ...

The list can be expanded as desired without losing backward compatibility. Events such as executing commands and similar actions can also be added.

Advantages:

• From the controller's point of view, this is a very simple and efficient procedure
• The format can be expanded as desired
• A complete log of all events that occur can be created
• No risk of the protocol being misinterpreted

Disadvantages:

• The evaluation on the host is not quite as trivial, but it is feasible
• The overhead caused by adding keys is approximately 3% and should therefore be easily acceptable.

The data stream should be converted into familiar formats such as .f37, .f119, and eeprom.bin using a tool. This allows the data to be used with existing analysis tools.

Additional specific analyses are also possible if required.

[realtimepeople]

I do not see the need for a different file format for the sensor.
We did never have any trouble with the current format of a file containing the metadata (*.EEPROM) and another one with the binary flight data.

For the analysis and scientific use of the data it would be desirable to have a sorted database of log files available online.
There shall be an index to different scenarios like take-off-modes, flight modes etc.
The flight data can be compressed to approximately 40% of the original binary size.
We also have long files with many flights in one recording. We shall post-process those to remove long ground recordings
and to provide one file per flight with some epilog and some prolog to get the zero readings of the sensor channels.
Here I would suggest to pick a compression algorithm like ZIP, BZIP or the like to compress the data and to combine metadata and the sensor channels into one compressed file format.

Finally, this topic belongs more to the data analysis repository.

[Simsys]

With the proposed approach, we will be able to obtain more information from the aircraft for the laboratory. For example, special occurrences, such as unexpected algorithm results that require intervention, can be stored as events in the data stream.

We would then have a precise record of everything that happens in the aircraft. We do not have that today. For example, the initial values for magnetic calibration are currently missing.

Subsequently, the records should be converted into established formats such as .f119 and EEPROM.bin so that there is no break in the formats.

Our problems in interpreting incidents with the “essential” systems alone prove that we have gaps in our records.

I am convinced that this will make us smarter