Package multiple partitions

[AndrewFasano]

When we extract multiple filesystems, we should consider how the information we get statically could inform how filesystems should be mounted. In particular, we can look for broken symlinks and references to paths that don't exist in one partition but do exist in another, and try to find a set of simulated mountpoints for various filesystems to minimize the number of missing paths.

I started prototyping this on the unify branch the combined extraction branch.

[AndrewFasano]

As a part of this (or after we get some initial support merged in), we should be able to present users with the unification schema we've created, but allow them to specify an alternative one (e.g., if the auto-discovered one is wrong, they should be able to tweak it)

[AndrewFasano]

I wanted to use the unblob API but ended up coming up with this terrible bash script that seems to work pretty well - I suspect there are some edge cases where things will go wrong (e.g., where unblob creates scratch files with names I haven't seen before). But it seems to work for some limited testing when we run unblob and then package its output with this.

#!/bin/bash

set -eu

# USAGE: ./package.sh extraction_root output_root

if [ "$#" -ne 2 ]; then
    echo "Usage: $0 [extraction_root] [output_dir]"
    exit 1
fi

input_dir=$1
output_dir=$2

mkdir -p $output_dir

# Find and delete debian packages - look for files named `debian_binary` that are in a directory named *_extract
# Also search for 'control' files that have a 'Package:' line and delete their parent directory
find "$input_dir" -type f -name 'debian-binary' | while read -r debian_binary; do
    if [[ "$debian_binary" == *_extract/debian-binary ]]; then
        rm -rf "$(dirname "$debian_binary")"
    fi
done
find "$input_dir" -type f -name 'control' | while read -r control_file; do
    if grep -q "^Package:" "$control_file"; then
        rm -rf "$(dirname "$control_file")"
    fi
done

# Delete all .uncompressed, .unknown, *.padding, and carved.elf files
find "$input_dir" -type f \( -name '*.uncompressed' -o -name '*.unknown' -o -name '*.padding' -o -name 'carved.elf' \) -delete


temp_dir=$(mktemp -d)

# Find all directories ending with 'carve' or 'extract'
find $1 -type d \( -name "*_carve" -o -name "*_extract" \) | while read -r dir; do
    # Create a name for the archive
    temp_archive="$temp_dir/$(uuidgen).tar.gz"

    # Create the archive, excluding subdirectories ending with '_carve' '_extract' or '_uncompressed'
    # Also filter out ###-####.[ext] files, which are almost always unblob artifacts (e.g., 0-100.lzma)
    tar -czf "$temp_archive" \
        --exclude='*_carve' --exclude='*_extract' --exclude '*.uncompressed' \
        --exclude='[0-9]*-[0-9]*.*' \
        -C "$dir" .

    # If the generated archive has a size of 0, delete it. Need to run tar to get list of files
    if [ $(tar -tf "$temp_archive" | wc -l) -lt 2 ]; then
        rm "$temp_archive"
        continue
    fi
    
    echo "Created archive: $temp_archive" from "$dir"
done

# Second pass: Sort archives by size and rename them
counter=0
while read -r size file; do
    mv "$file" "$output_dir/$counter.tar.gz"
    echo "Created archive: $output_dir/$counter.tar.gz (original: $file, size: $size)"
    counter=$((counter + 1))
done < <(find "$temp_dir" -type f -name "*.tar.gz" -print0 | xargs -0 du -s | sort -rn)

rm -rf "$temp_dir"
echo "Created $counter archives in $output_dir"

[AndrewFasano]

On the combined_extraction branch at 3c97d02 we have some initial support for packaging multiple partitions:

mkdir results/
docker run --rm -it -v $(pwd):/host \
    rehosting/fw2tar \
    unblob_package.sh \
    /host/fws/FW_RT_N66U_C1_300438510000.zip \
    /host/results/

This extracts and packages 4 filesystems into files 0.tar.gz, 1.tar.gz, 2.tar.gz and 3.tar.gz (0 is largest, 3 is smallest). These 4 archives are then combined into results/partitions.tar.gz. I considered leaving these in a directory (i.e., partitions/1.tar.gz, partitions/2.tar.gz, ...) but in the interest of using fewer inodes and files for large-scale experimentation left it archived for now. Might change that later.

The script identifies (and reports) the best mapping as 0:/,3:/opt meaning filesystem 0.tar.gz should go at / and filesystem 3.tar.gz should go at /opt. This mapping is then created as an archive at `results/best.tar.gz

If a user wishes to customize this layout, they can first extract the partitions and then run a command to do so, for example placing:

• 3.tar.gz at /
• 1.tar.gz at /foo
• 2.tar.gz at /zoo

And building this filesystem as `my_unified_fs.tar.gz by running

tar xvf results/partitions.tar.gz
docker run --rm -it -v $(pwd):/host \
    rehosting/fw2tar \
    unifyroot unify "/host/results/partitions" "/host/results/my_unified_fs.tar.gz" "3:/,1:/foo,2:/zoo"

I'd like to clean up this interface a little more still. I see a few next steps:

1. Improve CLI usage
2. Move unification python package info penguin
3. Add unification specification to penguin configurations

In the short/medium term, I'm not sure how to get this merged into main without breaking backwards compatibility. Perhaps we'll make the multi-partition mode only run if a given flag is enabled.

[AndrewFasano]

I've made a few branches for different attempts to get this working, I think I have something that fixes this issue on the package_partitions branch in #20. However, I don't think we should merge this to main yet (if ever) as it changes the output of our utility and breaks backwards compatibility.

The PR packages each filesystem as an independent tar archive. I plan to move the filesystem unification analysis/realization into subsequent stages of our rehosting pipeline.