improve error handling in case of image size failures

[freggeldoe]

When using extroot the resulting image might be too big to flash due to amount of packages installed.
owut usage does attempt to build an image, which ultimately fails:

$ owut upgrade
ASU-Server     https://sysupgrade.openwrt.org
Upstream       https://downloads.openwrt.org
Target         ath79/generic
Profile        tplink_archer-c7-v4
Package-arch   mips_24kc
Version-from   24.10.3 r28872-daca7c049b (kernel 6.6.104)
Version-to     24.10.4 r28959-29397011cc (kernel 6.6.110)
79 packages are out-of-date
Request hash:
  119f3adabc9a85b558aa3d28a6a72e5ca0e9f3433210dd0f467c9992093a77f2
--
Status:   queued - 0 ahead of you
Progress:   0s total =   0s in queue +   0s in build
--
Status:   init
Progress:   1s total =   0s in queue +   1s in build
--
Status:   container_setup
Progress:   3s total =   0s in queue +   3s in build
--
Status:   validate_manifest
Progress:  17s total =   0s in queue +  17s in build
--
Status:   building_image
Progress:  35s total =   0s in queue +  35s in build
--
Status:   failed
Progress:  37s total =   0s in queue +  37s in build

Build failed in  37s total =   0s in queue +  37s to build:

relevant logs from failure:

Building images...
Parallel mksquashfs: Using 16 processors
Creating 4.0 filesystem on /builder/build_dir/target-mips_24kc_musl/linux-ath79_generic/root.squashfs, block size 262144.
[=============================================================|] 1517/1517 100%

Exportable Squashfs 4.0 filesystem, xz compressed, data block size 262144
        compressed data, compressed metadata, compressed fragments,
        no xattrs, compressed ids
        duplicates are removed
Filesystem size 12333.93 Kbytes (12.04 Mbytes)
        33.20% of uncompressed filesystem size (37153.98 Kbytes)
Inode table size 13262 bytes (12.95 Kbytes)
        20.18% of uncompressed inode table size (65715 bytes)
Directory table size 19384 bytes (18.93 Kbytes)
        44.68% of uncompressed directory table size (43381 bytes)
Number of duplicate files found 220
Number of inodes 2008
Number of files 1455
Number of fragments 74
Number of symbolic links 405
Number of device nodes 1
Number of fifo nodes 0
Number of socket nodes 0
Number of directories 147
Number of hard-links 0
Number of ids (unique uids + gids) 1
Number of uids 1
        unknown (0)
Number of gids 1
        unknown (0)
Skip JSON creation for non existing file /builder/build_dir/target-mips_24kc_musl/linux-ath79_generic/tmp/openwrt-24.10.4-b3ba79e12a5a-ath79-generic-tplink_archer-c7-v4-squashfs-sysupgrade.bin
Skip JSON creation for non existing file /builder/build_dir/target-mips_24kc_musl/linux-ath79_generic/tmp/openwrt-24.10.4-b3ba79e12a5a-ath79-generic-tplink_archer-c7-v4-squashfs-factory.bin
JSON info file script could not find any JSON files for target

Calculating checksums...

ASU server stderr =
Package list missing or not up-to-date, generating it.

Building package index...
Downloading http://downloads.openwrt.org/releases/24.10.4/targets/ath79/generic/packages/Packages.gz
Updated list of available packages in /builder/build_dir/target-mips_24kc_musl/root-ath79/../../../../builder/dl/openwrt_core
Downloading http://downloads.openwrt.org/releases/24.10.4/targets/ath79/generic/packages/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/24.10.4/packages/mips_24kc/base/Packages.gz
Updated list of available packages in /builder/build_dir/target-mips_24kc_musl/root-ath79/../../../../builder/dl/openwrt_base
Downloading http://downloads.openwrt.org/releases/24.10.4/packages/mips_24kc/base/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/24.10.4/targets/ath79/generic/kmods/6.6.110-1-35ef4dd36891d37023436baa842fa311/Packages.gz
Updated list of available packages in /builder/build_dir/target-mips_24kc_musl/root-ath79/../../../../builder/dl/openwrt_kmods
Downloading http://downloads.openwrt.org/releases/24.10.4/targets/ath79/generic/kmods/6.6.110-1-35ef4dd36891d37023436baa842fa311/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/24.10.4/packages/mips_24kc/luci/Packages.gz
Updated list of available packages in /builder/build_dir/target-mips_24kc_musl/root-ath79/../../../../builder/dl/openwrt_luci
Downloading http://downloads.openwrt.org/releases/24.10.4/packages/mips_24kc/luci/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/24.10.4/packages/mips_24kc/packages/Packages.gz
Updated list of available packages in /builder/build_dir/target-mips_24kc_musl/root-ath79/../../../../builder/dl/openwrt_packages
Downloading http://downloads.openwrt.org/releases/24.10.4/packages/mips_24kc/packages/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/24.10.4/packages/mips_24kc/routing/Packages.gz
Updated list of available packages in /builder/build_dir/target-mips_24kc_musl/root-ath79/../../../../builder/dl/openwrt_routing
Downloading http://downloads.openwrt.org/releases/24.10.4/packages/mips_24kc/routing/Packages.sig
Signature check passed.
Downloading http://downloads.openwrt.org/releases/24.10.4/packages/mips_24kc/telephony/Packages.gz
Updated list of available packages in /builder/build_dir/target-mips_24kc_musl/root-ath79/../../../../builder/dl/openwrt_telephony
Downloading http://downloads.openwrt.org/releases/24.10.4/packages/mips_24kc/telephony/Packages.sig
Signature check passed.
Downloading file:packages/Packages
Updated list of available packages in /builder/build_dir/target-mips_24kc_musl/root-ath79/../../../../builder/dl/imagebuilder
Downloading file:packages/Packages.sig
Signature check passed.
Pseudo file "dev" exists in source filesystem "/builder/build_dir/target-mips_24kc_musl/root-ath79/dev".
Ignoring, exclude it (-e/-ef) to override.
24667+1 records in
24667+1 records out
12629946 bytes (13 MB, 12 MiB) copied, 0.030293 s, 417 MB/s
file-system partition too big (more than 12585906 bytes): Success
stat: cannot statx '/builder/build_dir/target-mips_24kc_musl/linux-ath79_generic/tmp/openwrt-24.10.4-b3ba79e12a5a-ath79-generic-tplink_archer-c7-v4-squashfs-sysupgrade.bin': No such file or directory
bash: line 1: [: 15466496: unary operator expected
    WARNING: Image file /builder/build_dir/target-mips_24kc_musl/linux-ath79_generic/tmp/openwrt-24.10.4-b3ba79e12a5a-ath79-generic-tplink_archer-c7-v4-squashfs-sysupgrade.bin is too big:  > 15466496
Failed to open firmware file
sha256sum: /builder/build_dir/target-mips_24kc_musl/linux-ath79_generic/tmp/openwrt-24.10.4-b3ba79e12a5a-ath79-generic-tplink_archer-c7-v4-squashfs-sysupgrade.bin: No such file or directory
cp: cannot stat '/builder/build_dir/target-mips_24kc_musl/linux-ath79_generic/tmp/openwrt-24.10.4-b3ba79e12a5a-ath79-generic-tplink_archer-c7-v4-squashfs-sysupgrade.bin': No such file or directory
24667+1 records in
24667+1 records out
12629946 bytes (13 MB, 12 MiB) copied, 0.0357817 s, 353 MB/s
file-system partition too big (more than 12582912 bytes): Success
cp: cannot stat '/builder/build_dir/target-mips_24kc_musl/linux-ath79_generic/tmp/openwrt-24.10.4-b3ba79e12a5a-ath79-generic-tplink_archer-c7-v4-squashfs-factory.bin': No such file or directory

Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/rq/worker.py", line 1659, in perform_job
    return_value = job.perform()
                   ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rq/job.py", line 1318, in perform
    self._result = self._execute()
                   ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/rq/job.py", line 1376, in _execute
    result = self.func(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/asu/build.py", line 437, in build
    result = _build(build_request, job)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/asu/build.py", line 301, in _build
    container.kill()
  File "/usr/local/lib/python3.12/site-packages/podman/domain/containers.py", line 290, in kill
    response.raise_for_status()
  File "/usr/local/lib/python3.12/site-packages/podman/api/client.py", line 82, in raise_for_status
    raise APIError(cause, response=self._response, explanation=message)
podman.errors.exceptions.APIError: 500 Server Error: Internal Server Error (no container with ID 1bbfa7a07ab65ecbece05fc23016c58a5c307c07f1a153512dc99f5e8b2cd041 found in database: no such container)

ERROR: Build failed with status 500
The above errors are often due to the upgrade server lagging behind the
build server, first suggestion is to wait a while and try again.

From this

file-system partition too big (more than 12585906 bytes): Success

I gathered the image might have been too big in size (although "Success" message seems weird).
Also

bash: line 1: [: 15466496: unary operator expected

looks like a broken test in a bash script.

After removing some large packages the next owut upgrade went flawlessly.
When extroot is used the image with all packages possibly is too large and the error handling/reporting could hint at that.
Perhaps this should be considered even when no extroot is used due to possible package size increases.

[efahl]

This actually better handled upstream in the ASU server, which has much better ability to pick through the returned text and find the interesting bits (and it's also not sensitive to code size, we could add a megabyte of junk there and no one would know).

I've got something working on my local test ASU server, I just need to do a PR and get it in production.