Ktools build fails on OSX 14

[sambles]

Issue Description

As of 26-April-2024 Github actions switched the macos-latest to track macos-14 which is the (M1) based operating system

This version fails the OSX ktools build test
https://github.com/OasisLMF/ktools/actions/runs/8846411207/job/24292237370

ktest-error.txt

[sr-murthy2]

@sambles @benhayes21 Hi. I'm now working on some tasks for Renew Risk. I've only just run into this issue while trying to install MDK on an M2 Mac (2022). So it seems ktools cannot be built on the M-series Macs. It appears the problem is related to an issue with Apache Arrow. Here is the error I saw:

...
In file included from eltcalc.cpp:62:
      ./../include/oasisparquet.h:6:10: fatal error: 'arrow/io/file.h' file not found
          6 | #include "arrow/io/file.h"
            |          ^~~~~~~~~~~~~~~~~
      1 error generated.
      make[3]: *** [eltcalc.o] Error 1
      make[2]: *** [all-recursive] Error 1
      make[1]: *** [all-recursive] Error 1
      make: *** [all] Error 2
      Ktools build failed.
...

Can you please confirm if this is the cause? Is a solution likely in the near term? Thanks.

[sambles]

Hi @sr-murthy2 a few years back a dev who owned an M1 had a go building ktools, here are this notes.
He did succeed at the time, but things might have changed or broken since then.
https://github.com/OasisLMF/OasisLMF/blob/main/docs/installation_for_mac.md

If Apache Arrow is causing problems you should be able to disable with --disable-parquet

./configure --enable-osx --enable-o3 --disable-parquet  ..etc 

That would disable the parquet output format option, but most other things should still work.

[sr-murthy2]

@sambles Thanks, will try it. Hope you're well. :)

BTW I'm using an M2 (2022) on OS 15.02 (Sequoia).

[sr-murthy2]

@sambles Followed this procedure ktools (latest version) make check fails - output below. This is with the disabling of Parquet option. With the default you get the missing Arrow header file error.

I just think don't this will work on the M2 or any M-series Mac. I tried the water_seller procedure for installing MDK, but that wasn't 100% successful - it looks like the package was installed without some ktools binaries, e.g. eltcalc, and there was the Arrow header file error in relation to eltcalc. The water_seller is also not an option because I need to be able to install and test specific MDK versions, e.g. v1.28.2 or v2.3.5 - it seems water_seller will only install the latest MDK.

I'm going to have to switch to an alternative, e.g. an Intel Mac. This could work on a Linux VM, but I also remember having similar problems on an Ubuntu 22.04.05 ARM64 VM. It appears ktools is fundamentally incompatible with ARM arch.

Making check in ktest
./runtests.sh 
aggregate_vulnerability.bin: OK
amplifications.bin: OK
coverages.bin: OK
damage_bin_dict.bin: OK
damage_bin_dict_ctrl.bin: OK
damagebintobin_ctrl_stderr.out: OK
damagebintobin_stderr.out: OK
events.bin: OK
eveout1.bin: OK
eveout2.bin: OK
fm_ept_1.csv: OK
fm_ept_1_r.csv: OK
fm_ept_2.csv: OK
fm_ept_2_r.csv: OK
fm_full_uncertainty_aep_1.csv: OK
fm_full_uncertainty_aep_1_r.csv: OK
fm_full_uncertainty_aep_2.csv: OK
fm_full_uncertainty_aep_2_r.csv: OK
fm_full_uncertainty_oep_1.csv: OK
fm_full_uncertainty_oep_1_r.csv: OK
fm_full_uncertainty_oep_2.csv: OK
fm_full_uncertainty_oep_2_r.csv: OK
fm_policytc.bin: OK
fm_profile.bin: OK
fm_programme.bin: OK
fm_psept_1.csv: OK
fm_psept_1_r.csv: OK
fm_psept_2.csv: OK
fm_psept_2_r.csv: OK
fm_sample_mean_aep_1.csv: OK
fm_sample_mean_aep_1_r.csv: OK
fm_sample_mean_aep_2.csv: OK
fm_sample_mean_aep_2_r.csv: OK
fm_sample_mean_oep_1.csv: OK
fm_sample_mean_oep_1_r.csv: OK
fm_sample_mean_oep_2.csv: OK
fm_sample_mean_oep_2_r.csv: OK
fm_wheatsheaf_aep_1.csv: OK
fm_wheatsheaf_aep_1_r.csv: OK
fm_wheatsheaf_aep_2.csv: OK
fm_wheatsheaf_aep_2_r.csv: OK
fm_wheatsheaf_mean_aep_1.csv: OK
fm_wheatsheaf_mean_aep_1_r.csv: OK
fm_wheatsheaf_mean_aep_2.csv: OK
fm_wheatsheaf_mean_aep_2_r.csv: OK
fm_wheatsheaf_mean_oep_1.csv: OK
fm_wheatsheaf_mean_oep_1_r.csv: OK
fm_wheatsheaf_mean_oep_2.csv: OK
fm_wheatsheaf_mean_oep_2_r.csv: OK
fm_wheatsheaf_oep_1.csv: OK
fm_wheatsheaf_oep_1_r.csv: OK
fm_wheatsheaf_oep_2.csv: OK
fm_wheatsheaf_oep_2_r.csv: OK
fm_xref.bin: OK
fmaalcalc1.csv: OK
fmaalcalc2.csv: OK
fmaalcalcmeanonly1.csv: OK
fmaalcalcmeanonly2.csv: OK
fmalt1.csv: OK
fmalt2.csv: OK
fmaltmeanonly1.csv: OK
fmaltmeanonly2.csv: OK
fmcalc.bin: OK
fmcalc.csv: OK
fmcalt1.csv: OK
fmcalt2.csv: OK
fmelt1.csv: FAILED
fmelt2.csv: FAILED
fmplt1.csv: FAILED
fmplt2.csv: FAILED
fmsummarycalc1.bin: OK
fmsummarycalc1.csv: OK
fmsummarycalc2.bin: OK
fmsummarycalc2.csv: OK
fmsummaryxref.bin: OK
footprint.bin: OK
footprint.bin.z: OK
footprint.csv: OK
footprint.idx: OK
footprint.idx.z: OK
footprint_ctrl.bin: OK
footprint_ctrl.idx: OK
footprint_usize.bin.z: OK
footprint_usize.idx.z: OK
footprintin.bin: OK
footprintin.idx: OK
footprinttobin_ctrl_stderr.out: OK
footprinttobin_stderr.out: OK
getmodelout.bin: OK
getmodelout.csv: OK
gul_ept_1.csv: FAILED
gul_ept_1_r.csv: OK
gul_ept_2.csv: FAILED
gul_ept_2_r.csv: OK
gul_full_uncertainty_aep_1.csv: FAILED
gul_full_uncertainty_aep_1_r.csv: OK
gul_full_uncertainty_aep_2_r.csv: OK
gul_full_uncertainty_oep_1.csv: FAILED
gul_full_uncertainty_oep_1_r.csv: OK
gul_full_uncertainty_oep_2_r.csv: OK
gul_psept_1.csv: OK
gul_psept_1_r.csv: OK
gul_psept_2.csv: OK
gul_psept_2_r.csv: OK
gul_sample_mean_aep_1.csv: FAILED
gul_sample_mean_aep_1_r.csv: OK
gul_sample_mean_aep_2_r.csv: OK
gul_sample_mean_oep_1.csv: FAILED
gul_sample_mean_oep_1_r.csv: OK
gul_sample_mean_oep_2_r.csv: OK
gul_wheatsheaf_aep_1.csv: OK
gul_wheatsheaf_aep_1_r.csv: OK
gul_wheatsheaf_aep_2_r.csv: OK
gul_wheatsheaf_mean_aep_1.csv: FAILED
gul_wheatsheaf_mean_aep_1_r.csv: OK
gul_wheatsheaf_mean_aep_2_r.csv: OK
gul_wheatsheaf_mean_oep_1.csv: FAILED
gul_wheatsheaf_mean_oep_1_r.csv: OK
gul_wheatsheaf_mean_oep_2_r.csv: OK
gul_wheatsheaf_oep_1.csv: OK
gul_wheatsheaf_oep_1_r.csv: OK
gul_wheatsheaf_oep_2_r.csv: OK
gulaalcalc1.csv: FAILED
gulaalcalc2.csv: OK
gulaalcalcmeanonly1.csv: OK
gulaalcalcmeanonly2.csv: OK
gulalt1.csv: FAILED
gulalt2.csv: OK
gulaltmeanonly1.csv: OK
gulaltmeanonly2.csv: OK
gulcalcc.bin: FAILED
gulcalcc.csv: FAILED
gulcalci.bin: FAILED
gulcalci.csv: FAILED
gulcalt1.csv: OK
gulcalt2.csv: OK
gulelt1.csv: FAILED
gulelt2.csv: FAILED
gulmelt1.csv: FAILED
gulmelt2.csv: FAILED
gulmplt1.csv: FAILED
gulmplt2.csv: FAILED
gulplt1.csv: FAILED
gulplt2.csv: FAILED
gulqelt1.csv: FAILED
gulqelt2.csv: FAILED
gulqplt1.csv: FAILED
gulqplt2.csv: FAILED
gulselt1.csv: OK
gulselt2.csv: OK
gulsplt1.csv: OK
gulsplt2.csv: OK
gulsummarycalc1.bin: FAILED
gulsummarycalc1.csv: FAILED
gulsummarycalc2.bin: FAILED
gulsummarycalc2.csv: FAILED
gulsummaryxref.bin: OK
items.bin: OK
lossfactors.bin: OK
occurrence.bin: OK
placalci.bin: FAILED
placalci.csv: FAILED
placalci_absf.bin: FAILED
placalci_absf.csv: FAILED
placalci_relf.bin: FAILED
placalci_relf.csv: FAILED
quantile.bin: OK
random.bin: OK
returnperiods.bin: OK
validatedamagebin_ctrl_stderr.out: OK
validatedamagebin_stderr.out: OK
validatefootprint_ctrl_stderr.out: OK
validatefootprint_stderr.out: OK
validatevulnerability_ctrl_stderr.out: OK
validatevulnerability_stderr.out: OK
vulnerability.bin: OK
vulnerability.bin.z: OK
vulnerability.csv: OK
vulnerability.idx: OK
vulnerability.idx.z: OK
vulnerability_ctrl.bin: OK
vulnerabilityin.bin: OK
vulnerabilitytobin_ctrl_stderr.out: OK
vulnerabilitytobin_stderr.out: OK
weights.bin: OK
shasum: WARNING: 40 computed checksums did NOT match
Sorry some checks failed.

make[1]: *** [check] Error 1
make: *** [check-recursive] Error 1

[sr-murthy2]

@sambles I've now managed to get the installation (v2.3.5) to complete on the M2, which includes the ktools build completing but make check still report checksum errors. I've attached the full log. Does this count as a correct or passing installation? All of the ktools binaries seem to be present (in /usr/bin), and the build completed, but I'm not sure I should be seeing those checksum errors.

The procedure is used is to use a water_seller fork with changes to install the v2.3.5 tag and use Mac-specific ktools configuration (--enable-osx) also with disabling of Parquet. The branch with these changes is pip-installed (into a 3.10.10 virtual env.:

pip install git+https://github.com/sr-murthy2/water_seller@install-mdk-v2.3.5

and then ws-install-oasislmf --extra is run as normal.

ws-mdk-installation-log-v2.3.5.txt

[sr-murthy2]

I've now pushed a water_seller PR from my personal GitHub account, as this falls outside the scope of my RR work. Not a priority or urgent, but you may want to look at that. I could also fully package it with a project TOML and a modern package management tool such as PDM, and publish it to PyPI, but for the moment the PR is useful enough.

An additional suggestion is that, given the separate versioning of ktools, it would be useful to show the ktools version / tag in the MDK release notes, otherwise it is difficult to know which ktools release it is associated with.