request: better PAX compatibility for cpio archives

[mcandre]

Good news!

I found one more discrepancy between the Payload cpio gz data for my .pkg installer, versus other, working .pkg installers.

Other installers have their Payload PAX compatible. To test this, use pax -v <Payload.

Requesting that the cpio-archive crate add support for PAX compatible cpio archives.

Hopefully a simple matter of tweaking cpio header.

I'm learning more about the history of archive formats. Seems that PAX helped to unify legacy cpio with legacy tar into a common spec.

[mcandre]

cpio varies greatly from GNU to BSD to kernel implementations to userland applications. It's not as consistent across implementations as zip, for example.

macOS is based on FreeBSD, including many command line utilities and flags.

Most cpio crates seem to be based on GNU cpio. The FreeBSD implementation of cpio is expected to resemble macOS cpio, and low level macOS installer package payloads, far more closely.

https://people.freebsd.org/~kientzle/libarchive/man/cpio.5.txt

[mcandre]

Unfortunately, even shelling out to pax -w -x cpio -f Payload . is unable to surmount Domain=BOMCopierFatalError Code=1 "cpio read error: bad file format".

The Payload file passes pax -v <Payload validation. It passes cpio -itv <Payload validation. It extracts just fine with cpio -idv <Payload. It's wrapped in a from Unix gzip stream just like other working macOS .pkg installers. The uids, gids, and timestamps match lsbom Bom. But installer rejects it nonetheless.

Not sure what's going on.

[mcandre]

Spotted a significant difference in behavior between CPIO data generated by pkgbuild vs cpio-archive.

We can probably ignore the AppleDouble dot underscore (<parent>/.../._<basename>) entries. I believe that part may fortunately be immaterial.

pkgbuild:

% mv Payload Payload.gz
% gunzip Payload.gz
% file Payload
Payload: ASCII cpio archive (pre-SVR4 or odc)
% pax -v <Payload
drwxr-xr-x  3 root     wheel            0 Jan 27 16:43 .
drwxr-xr-x  3 root     wheel            0 Jan 27 16:43 ./bin
-rwxr-xr-x  1 root     wheel           31 Jan 27 15:03 ./bin/hello
-rw-r--r--  1 root     wheel          163 Jan 27 15:03 ./bin/._hello
-rw-r--r--  3 root     wheel          163 Jan 27 16:43 ./._bin
% head -c 1024 Payload | xxd
00000000: 3037 3037 3037 3030 3030 3030 3030 3030  0707070000000000
00000010: 3030 3034 3037 3535 3030 3030 3030 3030  0004075500000000
00000020: 3030 3030 3030 3030 3033 3030 3030 3030  0000000003000000
00000030: 3135 3133 3632 3337 3631 3030 3030 3030  1513623761000000
00000040: 3230 3030 3030 3030 3030 3030 2e00 3037  200000000000..07
00000050: 3037 3037 3030 3030 3030 3030 3030 3031  0707000000000001
00000060: 3034 3037 3535 3030 3030 3030 3030 3030  0407550000000000
00000070: 3030 3030 3030 3033 3030 3030 3030 3135  0000000300000015
00000080: 3133 3632 3337 3631 3030 3030 3030 3630  1362376100000060
00000090: 3030 3030 3030 3030 3030 2e2f 6269 6e00  0000000000./bin.
000000a0: 3037 3037 3037 3030 3030 3030 3030 3030  0707070000000000
000000b0: 3032 3130 3037 3535 3030 3030 3030 3030  0210075500000000
000000c0: 3030 3030 3030 3030 3031 3030 3030 3030  0000000001000000
000000d0: 3135 3133 3632 3234 3034 3530 3030 3031  1513622404500001
000000e0: 3430 3030 3030 3030 3030 3337 2e2f 6269  400000000037./bi
000000f0: 6e2f 6865 6c6c 6f00 2321 2f62 696e 2f73  n/hello.#!/bin/s
00000100: 680a 6563 686f 2022 4865 6c6c 6f2c 2057  h.echo "Hello, W
00000110: 6f72 6c64 2122 0a30 3730 3730 3730 3030  orld!".070707000
00000120: 3030 3030 3030 3030 3331 3030 3634 3430  0000000031006440
00000130: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000
00000140: 3130 3030 3030 3031 3531 3336 3232 3430  1000000151362240
00000150: 3435 3030 3030 3136 3030 3030 3030 3030  4500001600000000
00000160: 3234 332e 2f62 696e 2f2e 5f68 656c 6c6f  243./bin/._hello
00000170: 0000 0516 0700 0200 004d 6163 204f 5320  .........Mac OS
00000180: 5820 2020 2020 2020 2000 0200 0000 0900  X        .......
00000190: 0000 3200 0000 7100 0000 0200 0000 a300  ..2...q.........
000001a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000001b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000001c0: 0000 0000 0041 5454 5200 0000 0000 0000  .....ATTR.......
000001d0: a300 0000 9800 0000 0b00 0000 0000 0000  ................
000001e0: 0000 0000 0000 0000 0100 0000 9800 0000  ................
000001f0: 0b00 0015 636f 6d2e 6170 706c 652e 7072  ....com.apple.pr
00000200: 6f76 656e 616e 6365 0001 0200 90d2 3246  ovenance......2F
00000210: 2f0a 4ef4 3037 3037 3037 3030 3030 3030  /.N.070707000000
00000220: 3030 3030 3034 3130 3036 3434 3030 3030  0000041006440000
00000230: 3030 3030 3030 3030 3030 3030 3033 3030  0000000000000300
00000240: 3030 3030 3135 3133 3632 3337 3631 3030  0000151362376100
00000250: 3030 3031 3030 3030 3030 3030 3032 3433  0001000000000243
00000260: 2e2f 2e5f 6269 6e00 0005 1607 0002 0000  ./._bin.........
00000270: 4d61 6320 4f53 2058 2020 2020 2020 2020  Mac OS X
00000280: 0002 0000 0009 0000 0032 0000 0071 0000  .........2...q..
00000290: 0002 0000 00a3 0000 0000 0000 0000 0000  ................
000002a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000002b0: 0000 0000 0000 0000 0000 0000 4154 5452  ............ATTR
000002c0: 0000 0000 0000 00a3 0000 0098 0000 000b  ................
000002d0: 0000 0000 0000 0000 0000 0000 0000 0001  ................
000002e0: 0000 0098 0000 000b 0000 1563 6f6d 2e61  ...........com.a
000002f0: 7070 6c65 2e70 726f 7665 6e61 6e63 6500  pple.provenance.
00000300: 0102 0090 d232 462f 0a4e f430 3730 3730  .....2F/.N.07070
00000310: 3730 3030 3030 3030 3030 3030 3530 3030  7000000000005000
00000320: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000
00000330: 3030 3030 3130 3030 3030 3030 3030 3030  0000100000000000
00000340: 3030 3030 3030 3030 3030 3133 3030 3030  0000000000130000
00000350: 3030 3030 3030 3054 5241 494c 4552 2121  0000000TRAILER!!
00000360: 2100 0000 0000 0000 0000 0000 0000 0000  !...............
00000370: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000380: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000390: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000003a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000003b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000003c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000003d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000003e0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000003f0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

cpio-archive:

% mv Payload Payload.gz
% gunzip Payload.gz
% file Payload
Payload: ASCII cpio archive (pre-SVR4 or odc)
% pax -v <Payload
pax: End of archive volume 1 reached
drwxr-xr-x  2 root     wheel            0 Jan 27 18:12 .
drwxr-xr-x  2 root     wheel            0 Jan 27 18:12 ./bin
-rwxr-xr-x  1 root     wheel           31 Jan 27 18:12 ./bin/hello
% head -c 1024 <Payload | xxd
00000000: 3037 3037 3037 3030 3030 3030 3030 3030  0707070000000000
00000010: 3030 3034 3037 3535 3030 3030 3030 3030  0004075500000000
00000020: 3030 3030 3030 3030 3032 3030 3030 3030  0000000002000000
00000030: 3135 3133 3632 3632 3134 3430 3030 3030  1513626214400000
00000040: 3230 3030 3030 3030 3030 3030 2e00 3037  200000000000..07
00000050: 3037 3037 3030 3030 3030 3030 3030 3031  0707000000000001
00000060: 3034 3037 3535 3030 3030 3030 3030 3030  0407550000000000
00000070: 3030 3030 3030 3032 3030 3030 3030 3135  0000000200000015
00000080: 3133 3632 3632 3134 3430 3030 3030 3630  1362621440000060
00000090: 3030 3030 3030 3030 3030 2e2f 6269 6e00  0000000000./bin.
000000a0: 3037 3037 3037 3030 3030 3030 3030 3030  0707070000000000
000000b0: 3032 3130 3037 3535 3030 3030 3030 3030  0210075500000000
000000c0: 3030 3030 3030 3030 3031 3030 3030 3030  0000000001000000
000000d0: 3135 3133 3632 3632 3134 3430 3030 3031  1513626214400001
000000e0: 3430 3030 3030 3030 3030 3337 2e2f 6269  400000000037./bi
000000f0: 6e2f 6865 6c6c 6f00 2321 2f62 696e 2f73  n/hello.#!/bin/s
00000100: 680a 6563 686f 2022 4865 6c6c 6f2c 2057  h.echo "Hello, W
00000110: 6f72 6c64 2122 0a30 3730 3730 3730 3030  orld!".070707000
00000120: 3030 3030 3030 3030 3330 3030 3030 3030  0000000030000000
00000130: 3030 3030 3030 3030 3030 3030 3030 3030  0000000000000000
00000140: 3030 3030 3030 3031 3531 3336 3236 3231  0000000151362621
00000150: 3434 3030 3030 3133 3030 3030 3030 3030  4400001300000000
00000160: 3030 3054 5241 494c 4552 2121 2100       000TRAILER!!!.

Crucially, pax: End of archive volume 1 reached seems to signal some kind of demarcation that is likely preventing macOS installer from reading the CPIO data.

As well, pkgbuild has a longer trailer sequence (140 extra nulls). Hmm.

Curious how we can patch the cpio-archive library to correct for this. At the 42nd byte, the CPIO data begins to diverge.