build and publish ARM images for kubeflow pipelines

[thesuperzapper]

Description

Currently, Kubeflow Pipelines is only publishing amd64 container images, most other Kubeflow components are now publishing for both amd64 and arm64.

Here is the list of images that need to be updated:
(this was the list for 2.0.0-alpha.7, more may have been added for 2.0.0+)

• gcr.io/ml-pipeline/cache-server
• gcr.io/ml-pipeline/metadata-envoy
• gcr.io/ml-pipeline/metadata-writer
• gcr.io/ml-pipeline/api-server
• gcr.io/ml-pipeline/persistenceagent
• gcr.io/ml-pipeline/scheduledworkflow
• gcr.io/ml-pipeline/frontend
• gcr.io/ml-pipeline/viewer-crd-controller
• gcr.io/ml-pipeline/visualization-server
• gcr.io/tfx-oss-public/ml_metadata_store_server
• gcr.io/google-containers/busybox

While most of these can run under Rosetta (on Apple Silicon Macs only), they run much slower and so are really only useful for testing.

Furthermore, the gcr.io/tfx-oss-public/ml_metadata_store_server image straight up does not work (even under emulation), I have made a separate Issue to track this one, as it is not controlled by KFP and is part of google/ml-metadata:

• build and publish ml_metadata_store_server container image for ARM64 #10308

---

Love this idea? Give it a 👍.

[thesuperzapper]

@chensun @zijianjoy I think this is a very important issue, as ARM64 (especially MacBooks) are now very common.

[thesuperzapper]

I can see that there was a merged PR to make some builds succeed on ARM64 (from 2019):

• Add arm64 support for ml-pipeline #2507

But another one got closed due to inactivity:

• Fix container images building on arm64 #3839

I will tag the author of those PRs so they can comment on this @MrXinWang.

[rimolive]

@thesuperzapper Let me know how can I help with this.

[Talador12]

+1 on this issue. Each quarter, more people are switching to Apple Silicon from older Intel Macs

[thesuperzapper]

Another image is gcr.io/google-containers/busybox, which is used in place of the real image for cached pipeline steps (to run echo that says the step is cached).

[thesuperzapper]

In my testing trying to build the images for linux/arm64, the only hard blockers are actually Python packages in the following images:

• gcr.io/ml-pipeline/metadata-writer: (the problem is ML Metadata)
  • https://github.com/kubeflow/pipelines/blob/master/backend/metadata_writer/Dockerfile
  • https://github.com/kubeflow/pipelines/blob/master/backend/metadata_writer/requirements.in
• gcr.io/ml-pipeline/visualization-server: (the problem is TFX)
  • https://github.com/kubeflow/pipelines/blob/master/backend/Dockerfile.visualization
  • https://github.com/kubeflow/pipelines/blob/master/backend/src/apiserver/visualization/requirements.in

The problematic pip packages are:

• ML Metadata:
  • ml-metadata (https://github.com/google/ml-metadata)
• TFX Stuff:
  • tensorflow-model-analysis (https://github.com/tensorflow/model-analysis)
  • tensorflow-data-validation (https://github.com/tensorflow/data-validation)
  • tensorflow-serving-api (https://github.com/tensorflow/serving)
  • tensorflow-transform (https://github.com/tensorflow/transform)
  • tfx-bsl (https://github.com/tensorflow/tfx-bsl)
    • (this one is a transitive dependency of the others)

There are already upstream Issues for some of them, but they mostly relate to Apple Silicone (slightly different from Linux ARM64), but I imagine that solving one will make it much easier to solve the other:

• ml-metadata:
  • Add support for M1 macs google/ml-metadata#143
• tensorflow-model-analysis:
  • (no existing upstream issues)
• tensorflow-data-validation:
  • Missing support for M1 Mac tensorflow/data-validation#205 (the main issue about getting tfx working)
  • Wheel support for linux aarch64 tensorflow/data-validation#141
• tensorflow-serving-api:
  • Apple M1 support tensorflow/serving#1816
  • tensorflow-serving docker container doesn't work on Macs with Apple M1 chips. tensorflow/serving#1948
• tensorflow-transform:
  • Apple Silicon support for tensorflow-transform not available as tfx-bsl does not support it? tensorflow/transform#298
• tfx-bsl:
  • Any plans to support Apple silicon devices like M1 Macs? tensorflow/tfx-bsl#48

We either need to get those packages working so they can be pip installed on a Linux ARM, or remove our dependency on them.

[rimolive]

@thesuperzapper metadata-write and visualization-server are kfpv1 deprecated components, so they're not required for kfpv2.

[AndersBennedsgaard]

We run a small ARM-based cluster which we want to run Kubeflow on, so I have started to build the components for ARM. I've been successful at building the cache-server, persistence agent, scheduled workflow agent, viewer-crd-controller, and frontend. I only had to set --platform=$BUILDPLATFORM as an argument in the first Dockerfile stage and, for all the Go based components, add GOOS=$TARGETOS GOARCH=$TARGETARCH in the go build step. However, building the API server seems to need a little more work.

The main reason for this, is that https://github.com/mattn/go-sqlite3/ now needs to be compiled with a cross-compiler, so I have to run apt-get install -y gcc-aarch64-linux-gnu g++-aarch64-linux-gnu, and set CC=aarch64-linux-gnu-gcc CXX=aarch64-linux-gnu-g++ CGO_ENABLED=1 environment variables during go build, which works!

However, this seems very fragile to changes in build server, new CPU architectures, etc., so I looked into why we even include SQLite - and the answer seems to be that we only use SQLite for integration testing?
So perhaps it would make sense to exclude it in the production image?

One way to do this is to move SQLite references to a separate db_sqlite.go file and use a // +build integration tag, and change test runs to use go test --tags=integration for integration tests.
That would make it possible to build the API server without additional C/C++ cross-compilers.

In fact, I have done this on our custom build and now I can build the binary and Docker container without SQLite with the same configuration change as with the other components mentioned above.