Proposal: Unified firmware analysis and filesystem reconstruction

[lacraig2]

Our current system works in the following way:

flowchart LR
    A[firmware input] -->B(fw2tar)--> C(rootfs.tar.gz)--> D(penguin)
    A -->F(User determined process) --> C

Loading

A problem we often see is that fw2tar does not correctly determine what should be in the filesystem. This is particularly relevant when considering the case of multiple filesystems.

This proposal advances and relies upon the theory that static information is insufficient in some cases to fully rectify a filesystem.

The problem of rectifying a filesystem from an input firmware is a rehosting decision that should be expressed and resolvable by the user within penguin.

Making this change would mean folding in significant portions of fw2tar inside of penguin while maintaining existing code paths and functionality.

For example:

flowchart LR
    A[firmware input] --> B(penguin init --fw) --> E[config + multiple fs] --> G[penguin run]
    C[fs.tar.gz] --> D(penguin init fs.tar.gz) --> F[config + single fs] --> G

Loading

In addition, we would need some specification within config.yaml that would describe the filesystem:

fs:
      / : fs.tar.gz
      /mnt/a: fs2.tar.gz
      /mnt/b: fs3.tar.gz

There might also be a more clever path for including this in static_files.

[jamcleod]

I think I agree with most of what you're proposing but I think the way I'd change it is to make this change a bit more incrementally. Pulling in fw2tar at the same time as adding the mountpoint analysis and the in-config partition merging feels unwise.

I think at first we could add multi-partition support in a way that maintains the existing interface but includes analysis when multiple tarballs are provided:

$ penguin init rootfs.tar.gz part2.tar.gz
Searching for possible mountpoints...
Mountpoint `/mnt/cfg` found for part2.tar.gz, assign this mountpoint? (Y/n)
[...]
Generating config...
[...]

Which then generates something like:

fs:
    mounts:
        /: rootfs.tar.gz
        /mnt/cfg: part2.tar.gz

(Additional possible concerns: should this be structured in a way that is forward-compatible with things other than a tarball being used for a mount? that might be a non-issue as those would just go in separate sections?)

Then after that is merged in we merge in fw2tar, at which point we make a breaking change on the interface (as I feel giving the firmware should be the default behavior and maybe a --tar flag for providing a premade fs, alternatively detecting a tarball was generated with fw2tar).

I think that having the "repack firmware" and "initialize project" steps be separate is not particularly user-friendly if we want to integrate mountpoints into the config file. For starters, it would mean needing some editable file between repack and init to handle mountpoints for initialization, separate from the configuration option of mountpoints available after project initialization.

I think there should be roughly 3 cases of concern:

1. a firmware a user has repacked themselves using fw2tar
2. a firmware image with a single filesystem partition of interest
3. a firmware image with multiple filesystems

From a user perspective, however, I think typically we want this to look like 2 usecases: firmware image and repacked firmware. Then penguin should be prompting the user to specify mountpoints for additional detected filesystems (providing default paths for mountpoints using unifyroot from this PR). A user then has the option as to whether to keep the partition at the default mountpoint, move it somewhere else, or to ignore the filesystem. This can happen as a part of the project initialization via interactive prompts (see above) while also being possible to skip by passing a flag to confirm all defaults as well as maybe a flag for ignoring all additional partitions (both being provided for any scripted usage that would want to ignore these).

flowchart LR
  A1[firmware image] --> B[penguin init] -->|multiple filesystems| C[mountpoint prompts] --> D[config + fs] --> E[penguin run]
  A2[fs.tar.gz] --> B
  B -->|single fs| D

Loading

One idea I've started to like is have the default behavior be automatic (detect if fw2tar tarball, otherwise run through fw2tar) but have flags to force it to treat it as pre-processed (--tar) or as a firmware blob (--firmware), credit to Ali for the idea