Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/build-docker-image.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ on:
pull_request:
branches: "*"

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
Comment on lines +10 to +12
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cancel in-progress builds when a new push occurs in a PR or on a branch. We should actually use that in other repos (magistrate?) too.
Azure builds in vt have autoCancel so that's fine:

pr:
  drafts: false
  autoCancel: true


jobs:
get-matrix:
runs-on: ubuntu-latest
Expand Down
2 changes: 1 addition & 1 deletion ci/build-docker-image.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ def build(self, args: list):
print("---------------------------")

image_setup = setup.get(image.get("setup"))
dockerfile = image.get("dockerfile")
dockerfile = "base.dockerfile"
env = image_setup.get("env")

args = {
Expand Down
7 changes: 1 addition & 6 deletions ci/build-setup.py
Original file line number Diff line number Diff line change
Expand Up @@ -56,10 +56,7 @@ def build(self):
for (setup_id, setup_config) in setup.items():
# generate install instructions and install dependencies commands
instructions = []
downloads = []
for (dep_id, args) in setup_config.get("deps").items():
if dep_id != "cmd":
downloads.append(f"wget $WF_DEPS_URL/{dep_id}.sh")
instructions.extend(self.__instructions(dep_id, args))

setup_script = ""
Expand All @@ -70,12 +67,10 @@ def build(self):
) as file:
setup_script = file.read()
setup_script = setup_script.replace("%ENVIRONMENT_LABEL%", setup_config.get("label"))
setup_script = setup_script.replace("%DEPS_DOWNLOAD%", '\n '.join(downloads))
setup_script = setup_script.replace("%DEPS_INSTALL%", '\n'.join(instructions))

setup_filename = f"setup-{setup_id}.sh"
setup_filepath = os.path.join(os.path.dirname(__file__),
"shared", "scripts", setup_filename)
setup_filepath = os.path.join(os.path.dirname(__file__), setup_filename)

with open(setup_filepath, "w+", encoding="utf-8") as f:
f.write(setup_script)
Expand Down
21 changes: 0 additions & 21 deletions ci/config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -306,7 +306,6 @@ images:
-
repository: *docker-repository
tag: wf-amd64-alpine-clang-13-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: alpine:3.16
setup: amd64-alpine-clang-13-cpp
Expand All @@ -315,55 +314,48 @@ images:
-
repository: *docker-repository
tag: wf-amd64-ubuntu-20.04-gcc-9-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:20.04
setup: amd64-ubuntu-20.04-gcc-9-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-20.04-gcc-10-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:20.04
setup: amd64-ubuntu-20.04-gcc-10-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-20.04-gcc-10-openmpi-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:20.04
setup: amd64-ubuntu-20.04-gcc-10-openmpi-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-22.04-gcc-11-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:22.04
setup: amd64-ubuntu-22.04-gcc-11-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-22.04-gcc-12-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:22.04
setup: amd64-ubuntu-22.04-gcc-12-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-22.04-gcc-12-vtk-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:22.04
setup: amd64-ubuntu-22.04-gcc-12-vtk-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-22.04-gcc-12-zoltan-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:22.04
setup: amd64-ubuntu-22.04-gcc-12-zoltan-cpp
Expand All @@ -372,15 +364,13 @@ images:
-
repository: *docker-repository
tag: wf-amd64-ubuntu-20.04-gcc-9-cuda-12.2.0-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: nvidia/cuda:12.2.0-devel-ubuntu20.04
setup: amd64-ubuntu-20.04-gcc-9-cuda-12.2.0-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-20.04-gcc-9-cuda-11.2.2-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: nvidia/cuda:11.2.2-devel-ubuntu20.04
setup: amd64-ubuntu-20.04-gcc-9-cuda-11.2.2-cpp
Expand All @@ -389,71 +379,62 @@ images:
-
repository: *docker-repository
tag: wf-amd64-ubuntu-20.04-clang-9-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:20.04
setup: amd64-ubuntu-20.04-clang-9-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-20.04-clang-10-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:20.04
setup: amd64-ubuntu-20.04-clang-10-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-22.04-clang-11-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:22.04
setup: amd64-ubuntu-22.04-clang-11-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-22.04-clang-12-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:22.04
setup: amd64-ubuntu-22.04-clang-12-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-22.04-clang-13-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:22.04
setup: amd64-ubuntu-22.04-clang-13-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-22.04-clang-14-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:22.04
setup: amd64-ubuntu-22.04-clang-14-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-24.04-clang-16-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:24.04
setup: amd64-ubuntu-24.04-clang-16-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-24.04-clang-16-vtk-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:24.04
setup: amd64-ubuntu-24.04-clang-16-vtk-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-24.04-clang-16-zoltan-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: ubuntu:24.04
setup: amd64-ubuntu-24.04-clang-16-zoltan-cpp
Expand All @@ -462,15 +443,13 @@ images:
-
repository: *docker-repository
tag: wf-amd64-ubuntu-20.04-icpc-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: intel/oneapi:os-tools-ubuntu20.04
setup: amd64-ubuntu-20.04-icpc-cpp

-
repository: *docker-repository
tag: wf-amd64-ubuntu-20.04-icpx-cpp
dockerfile: base.dockerfile
arch: linux/amd64
base: intel/oneapi:os-tools-ubuntu20.04
setup: amd64-ubuntu-20.04-icpx-cpp
Expand Down
21 changes: 9 additions & 12 deletions ci/docker/base.dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# Base dockerfile to build images used in Darma testing.

ARG REPO=lifflander1/vt
ARG ARCH=amd64
ARG BASE=ubuntu:22.04

Expand Down Expand Up @@ -30,17 +31,10 @@ ARG CMPLR_ROOT \
LD_LIBRARY_PATH=${LD_LIBRARY_PATH:-""} \
LIBRARY_PATH=${LIBRARY_PATH:-""}

# Copy dependency scripts
ENV WF_TMP_DIR=/opt/workflows
ADD ci/shared/scripts/deps ${WF_TMP_DIR}/deps

# Setup environment variables
ENV DEBIAN_FRONTEND=noninteractive

ENV SCRIPTS_INSTALL_DIR=${SCRIPTS_INSTALL_DIR}

ENV WF_DOCKER=1 \
WF_SETUP_ID=${SETUP_ID}
ENV WF_SETUP_ID=${SETUP_ID}

ENV CC=$CC \
CXX=$CXX \
Expand Down Expand Up @@ -73,8 +67,11 @@ ENV CMPLR_ROOT=$CMPLR_ROOT \
LIBRARY_PATH=$LIBRARY_PATH \
LD_LIBRARY_PATH=$LD_LIBRARY_PATH

COPY ci/shared/scripts/setup-${SETUP_ID}.sh ${WF_TMP_DIR}/setup.sh
RUN chmod +x ${WF_TMP_DIR}/setup.sh && . ${WF_TMP_DIR}/setup.sh
# Prepare working directory
ENV WF_TMP_DIR=/opt/workflows

# Clean
RUN rm -rf $WF_TMP_DIR
# Run the setup scripts
RUN --mount=type=bind,rw,source=ci,target=${WF_TMP_DIR} \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mounting as rw to allow saving customized setup script. Docker discards the written data anyways.

https://docs.docker.com/reference/dockerfile/#run---mounttypebind

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: check if --mount=type=cache can be applied here.

https://docs.docker.com/reference/dockerfile/#run---mounttypecache

sh ${WF_TMP_DIR}/setup-basic.sh && \
python3 ${WF_TMP_DIR}/build-setup.py ${REPO}:wf-${SETUP_ID} && \
sh ${WF_TMP_DIR}/setup-${SETUP_ID}.sh
54 changes: 54 additions & 0 deletions ci/setup-basic.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
#!/usr/bin/env sh

# Basic universal setup script

echo "------------- Setup --------------"

OS_VERSION=
UNAME=$(uname)
if [ "$UNAME" = "Darwin" ]
then
OS_NAME=$(sw_vers -productName)
OS_VERSION=$(sw_vers -productVersion)
elif [ "$UNAME" = "Linux" ]
then
OS_NAME=$(cat /etc/os-release | grep -E "^NAME=*" | cut -d = -f 2 | tr -d '"')
OS_VERSION=$(cat /etc/os-release | grep -E "^VERSION_ID=*" | cut -d = -f 2 | tr -d '"')
fi

echo "Operating system: $OS_NAME / Version: $OS_VERSION"
echo "PATH=$PATH"
echo "CPATH=$CPATH"
echo "LIBRARY_PATH=$LIBRARY_PATH"
echo "LD_LIBRARY_PATH=$LD_LIBRARY_PATH"
echo "----------------------------------"

# Save setup environment to ~/.setuprc (used by packages.sh dep for example)
echo "-- Set up variables (WF_SETUP_ID, OS_NAME, OS_VERSION) > ~/.setuprc"
{
echo "export WF_SETUP_ID=\"$WF_SETUP_ID\""
echo "export OS_NAME=\"$OS_NAME\""
echo "export OS_VERSION=\"$OS_VERSION\""
} >> ~/.setuprc

### UPDATE PACKAGE LIST AND INSTALL CORE DEPENDENCIES
echo "-- Installing Core packages ($OS_NAME)..."
if [ "$OS_NAME" = "Ubuntu" ]
then
apt-get update -y -q
apt-get install -y -q --no-install-recommends ca-certificates wget git python3 python3-yaml
elif [ "$OS_NAME" = "Alpine Linux" ]
then
apk update
apk add --no-cache wget git bash python3 py3-yaml
elif [ "$OS_NAME" = "macOS" ]
then
brew update
else
echo "Error. Please implement the pre-setup instructions for OS=$OS_NAME"
exit 1
fi

echo "--"
echo "-- Core packages installed!"
echo "--"
Loading