CI: Recipes about `ChangeLog` regularly fail in Linux+mingw builds

[jimklimov]

Over the past months there have been several efforts and PRs to improve the NUT recipes (e.g. #2366, #2360), and in particular the generation of ChangeLog file (but only if it is older than Git metadata) and subsequent parsing into ChangeLog.adoc and HTML/PDF variants. On weaker CI systems the cycle can take 10 minutes, so a number of hoops are jumped through to avoid needless rebuilds of these files (and also aiding nut-website builds which generate its deliverables).

Currently this is handled by Makefile.am (to allow make ChangeLog in the root dir) and docs/Makefile.am (to handle PDF/HTML renditions and their prerequisites up the chain), and hoping for interaction between copies of make implementations that have a "job server" support to avoid parallel attempts to build the thing. There were earlier mishaps with several copies of a parser script writing into the same file, causing invalid content and markup; now the writes should go into temporary files and then be atomically renamed, to avoid at least that failure mode.

In the recent weeks, NUT CI farm occasionally fails the Linux+mingw builds (only them, and not always) indicating that the interim ChangeLog.xml (from asciidoc) did not pass structural xmllint checks. Either something misfires about not-writing to same file from several builds (job server issue?) or we indeed have some invalid characters in section titles (was a problem earlier) or whatever; but then it would fail every time?..

...
Using still-valid ChangeLog file generated earlier from same revision of Git source metadata in '.git'
a2x: ERROR: "xmllint" --nonet --noout --valid "/dev/shm/jenkins-nutci/nut_nut_PR-2480@2/scripts/Windows/nut_build/docs/tmp/html-single.ChangeLog.html.983992/ChangeLog.xml" returned non-zero exit status 4

mv: cannot stat './tmp/html-single.ChangeLog.html.983992/ChangeLog.html': No such file or directory
make[2]: *** [Makefile:1389: ChangeLog.html] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [Makefile:979: all-recursive] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [Makefile:802: all-recursive] Error 1

[jimklimov]

Of 3 similar cross-building agents, overnight while BUILD_TYPE=cross-windows-mingw ./ci_build.sh ; do sleep 5 ; done loop reproduced this issue on one with a build in /dev/shm. So maybe filesystem type (timestamp precision, locking, caching...) plays a role?

• Files seen on a system that failed (no HTML yet, just the XML intermediate):

...
Using still-valid ChangeLog file generated earlier from same revision of Git source metadata in '.git'
a2x: ERROR: "xmllint" --nonet --noout --valid "/dev/shm/nut-jim/scripts/Windows/nut_build/docs/tmp/html-single.ChangeLog.html.287684/ChangeLog.xml" returned non-zero exit status 4

mv: cannot stat './tmp/html-single.ChangeLog.html.287684/ChangeLog.html': No such file or directory
make[2]: *** [Makefile:1389: ChangeLog.html] Error 1
...

$ git log --oneline -1
6e0e2fce6 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #2518 from jimklimov/issue-2512

$ find . -name ChangeLog'*' -ls
     9564   1632 -rw-rw-r--   1 jim      jim       1668902 Jul 12 06:50 ./scripts/Windows/nut_build/ChangeLog.adoc-parsed
     9563   1540 -rw-rw-r--   1 jim      jim       1574790 Jul 12 06:50 ./scripts/Windows/nut_build/ChangeLog.adoc
     8809   1528 -rw-rw-r--   1 jim      jim       1561295 Jul 12 06:50 ./scripts/Windows/nut_build/ChangeLog
     9418    592 -rw-rw-r--   1 jim      jim        605067 Jul 12 06:50 ./scripts/Windows/nut_build/docs/tmp/html-single.ChangeLog.html.287684/ChangeLog.xml
     8855      0 -rw-rw-r--   1 jim      jim             0 Jul 12 06:50 ./scripts/Windows/nut_build/docs/ChangeLog.txt-prepped
     8854      0 lrwxrwxrwx   1 jim      jim            70 Jul 12 06:50 ./scripts/Windows/nut_build/docs/ChangeLog.txt -> /dev/shm/nut-jim/scripts/Windows/nut_build/../../../docs/ChangeLog.txt
      546      4 -rw-rw-r--   1 jim      jim           601 Jul 11 14:13 ./docs/ChangeLog.txt

• Same system, another (non-tmpfs) filesystem. Atomically generated text files (ChangeLog, ChangeLog.adoc, ChangeLog.adoc-parsed) are identical; HTML file broken (but one did not emit an error, markup happened to be intact, just a phrase cut mid-word - and the footer follows):

$ find . -name ChangeLog'*' -ls
   164076      5 -rw-r--r--   1 jim      jim           601 Jan  3  2024 ./docs/ChangeLog.txt
   346772    821 -rw-rw-r--   1 jim      jim       1561295 Jul 12 08:21 ./scripts/Windows/nut_build/ChangeLog
   348130    773 -rw-rw-r--   1 jim      jim       1574790 Jul 12 08:21 ./scripts/Windows/nut_build/ChangeLog.adoc
   348132    833 -rw-rw-r--   1 jim      jim       1668902 Jul 12 08:21 ./scripts/Windows/nut_build/ChangeLog.adoc-parsed
   346748      1 lrwxrwxrwx   1 jim      jim            67 Jul 12 08:20 ./scripts/Windows/nut_build/docs/ChangeLog.txt -> /home/jim/nut/scripts/Windows/nut_build/../../../docs/ChangeLog.txt
   348484    261 -rw-rw-r--   1 jim      jim       1042478 Jul 12 08:21 ./scripts/Windows/nut_build/docs/ChangeLog.html
   346750      1 -rw-rw-r--   1 jim      jim             0 Jul 12 08:20 ./scripts/Windows/nut_build/docs/ChangeLog.txt-prepped

• On another system that survived the trials so far (even though in /dev/shm), generated text files are minutely (3-4 bytes) larger, and HTML significantly larger:

$ git log --oneline -1
6e0e2fce6 (HEAD -> master, origin/master, origin/HEAD) Merge pull request #2518 from jimklimov/issue-2512

$ find . -name ChangeLog'*' -ls
    89342   3832 -rw-r--r--   1 jim      jim       3922860 Jul 12 08:25 ./scripts/Windows/nut_install/share/doc/nut/html-doc/ChangeLog.html
    87793   1632 -rw-rw-r--   1 jim      jim       1668902 Jul 12 08:22 ./scripts/Windows/nut_build/ChangeLog.adoc-parsed
    87791   1540 -rw-rw-r--   1 jim      jim       1574790 Jul 12 08:22 ./scripts/Windows/nut_build/ChangeLog.adoc
    86256   1528 -rw-rw-r--   1 jim      jim       1561295 Jul 12 08:22 ./scripts/Windows/nut_build/ChangeLog
    87797   3832 -rw-rw-r--   1 jim      jim       3922860 Jul 12 08:23 ./scripts/Windows/nut_build/docs/ChangeLog.html
    86188      0 -rw-rw-r--   1 jim      jim             0 Jul 12 08:20 ./scripts/Windows/nut_build/docs/ChangeLog.txt-prepped
    86187      0 lrwxrwxrwx   1 jim      jim            70 Jul 12 08:20 ./scripts/Windows/nut_build/docs/ChangeLog.txt -> /dev/shm/nut-jim/scripts/Windows/nut_build/../../../docs/ChangeLog.txt
      176      4 -rw-rw-r--   1 jim      jim           601 Jul 11 11:15 ./docs/ChangeLog.txt

• On a separate system that survived the trials so far (in persistent homedir), generated text files are minutely (3-4 bytes) larger, and HTML significantly larger:

$ git log --oneline -1
6e0e2fce6 (HEAD) Merge pull request #2518 from jimklimov/issue-2512

$ find . -name ChangeLog'*' -ls
    61511   1632 -rw-r--r--   1 jim      jim       1668906 Jul 12 10:10 ./scripts/Windows/nut_build/ChangeLog.adoc-parsed
    61510   1540 -rw-r--r--   1 jim      jim       1574794 Jul 12 10:10 ./scripts/Windows/nut_build/ChangeLog.adoc
    61461   1528 -rw-r--r--   1 jim      jim       1561292 Jul 12 10:10 ./scripts/Windows/nut_build/ChangeLog
    61545   3832 -rw-r--r--   1 jim      jim       3922850 Jul 12 10:10 ./scripts/Windows/nut_build/docs/ChangeLog.html
    59986      0 -rw-r--r--   1 jim      jim             0 Jul 12 10:09 ./scripts/Windows/nut_build/docs/ChangeLog.txt-prepped
    59984      0 lrwxrwxrwx   1 jim      jim            70 Jul 12 10:09 ./scripts/Windows/nut_build/docs/ChangeLog.txt -> /dev/shm/nut-jim/scripts/Windows/nut_build/../../../docs/ChangeLog.txt
    47945      4 -rw-r--r--   1 jim      jim           601 Jul 12 10:03 ./docs/ChangeLog.txt

[jimklimov]

Scripted Windows builds seem to be parallelized (gotta check where MAKEFLAGS tricks lead), so MAYBE this is also a problem of several threads stepping onto each other's toes. Although it is used in other platforms and should be safe...

OOM killer does not seem to be a concern here.