From 6c39b670a386dd4839e6da8c428b6d6ca035c397 Mon Sep 17 00:00:00 2001 From: "Thomas Irgang (thir820)" Date: Thu, 24 Oct 2024 14:23:39 +0000 Subject: [PATCH 1/4] start documentation --- docs/SUMMARY.md | 10 ++++++++++ docs/layers/appdev.md | 1 + docs/layers/base.md | 1 + docs/layers/elbe.md | 1 + docs/layers/embdgen.md | 1 + docs/layers/index.md | 17 +++++++++++++++++ docs/layers/kiwi.md | 1 + docs/layers/pbuilder.md | 1 + docs/layers/rust.md | 1 + docs/layers/vscode.md | 1 + docs/overview.md | 27 +++++++++++++++++++++++++++ docs/tests.md | 16 ++++++++++++++++ 12 files changed, 78 insertions(+) create mode 100644 docs/layers/appdev.md create mode 100644 docs/layers/base.md create mode 100644 docs/layers/elbe.md create mode 100644 docs/layers/embdgen.md create mode 100644 docs/layers/index.md create mode 100644 docs/layers/kiwi.md create mode 100644 docs/layers/pbuilder.md create mode 100644 docs/layers/rust.md create mode 100644 docs/layers/vscode.md create mode 100644 docs/tests.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 506b5d2..006d7d8 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -3,3 +3,13 @@ [EB corbos Linux dev container](README.md) - [Overview](overview.md) +- [Testing the container](tests.md) +- [Layers](layers/index.md) + - [Base Layer](layers/base.md) + - [PBuilder Layer](layers/pbuilder.md) + - [Rust Layer](layers/rust.md) + - [Kiwi-ng Layer](layers/kiwi.md) + - [AppDev Layer](layers/appdev.md) + - [Embdgen Layer](layers/embdgen.md) + - [VS Code Layer](layers/vscode.md) + - [Elbe Layer](layers/elbe.md) diff --git a/docs/layers/appdev.md b/docs/layers/appdev.md new file mode 100644 index 0000000..2f5602a --- /dev/null +++ b/docs/layers/appdev.md @@ -0,0 +1 @@ +# AppDev Layer diff --git a/docs/layers/base.md b/docs/layers/base.md new file mode 100644 index 0000000..85a4056 --- /dev/null +++ b/docs/layers/base.md @@ -0,0 +1 @@ +# Base Layer diff --git a/docs/layers/elbe.md b/docs/layers/elbe.md new file mode 100644 index 0000000..5dc5937 --- /dev/null +++ b/docs/layers/elbe.md @@ -0,0 +1 @@ +# Elbe Layer diff --git a/docs/layers/embdgen.md b/docs/layers/embdgen.md new file mode 100644 index 0000000..f62d198 --- /dev/null +++ b/docs/layers/embdgen.md @@ -0,0 +1 @@ +# Embdgen Layer diff --git a/docs/layers/index.md b/docs/layers/index.md new file mode 100644 index 0000000..39e45f6 --- /dev/null +++ b/docs/layers/index.md @@ -0,0 +1,17 @@ +# Layers + +The different EB corbos Linux dev container features are grouped into separate Docker container layers. +Each layer should be independent of the other layers. +The default layers are contained in _docs/layers_. + +## Customer specific layers + +You can add your own layers by using an own container configuration file. +The default configuration is contained in _configuration/build_config.yaml_. + +To create an own layer follow these steps: + +- Prepare your layer, i.e. a Docker image in a separate folder. +- Copy the configuration from _configuration/build_config.yaml_ and add the path to your layer. +- Make sure the Python dependencies are available, e.g. by creating a venv and installing the dependencies form _requirements.txt_. +- Run the build using your configuration: `python builder/build_container.py path/to/your/configuration.yaml` \ No newline at end of file diff --git a/docs/layers/kiwi.md b/docs/layers/kiwi.md new file mode 100644 index 0000000..18f4c24 --- /dev/null +++ b/docs/layers/kiwi.md @@ -0,0 +1 @@ +# Kiwi-ng Layer diff --git a/docs/layers/pbuilder.md b/docs/layers/pbuilder.md new file mode 100644 index 0000000..cffef7d --- /dev/null +++ b/docs/layers/pbuilder.md @@ -0,0 +1 @@ +# PBuilder Layer diff --git a/docs/layers/rust.md b/docs/layers/rust.md new file mode 100644 index 0000000..da91951 --- /dev/null +++ b/docs/layers/rust.md @@ -0,0 +1 @@ +# Rust Layer diff --git a/docs/layers/vscode.md b/docs/layers/vscode.md new file mode 100644 index 0000000..b0f4fff --- /dev/null +++ b/docs/layers/vscode.md @@ -0,0 +1 @@ +# VS Code Layer diff --git a/docs/overview.md b/docs/overview.md index 476060d..5381529 100644 --- a/docs/overview.md +++ b/docs/overview.md @@ -1,2 +1,29 @@ # Overview +The EB corbos Linux dev container provides a stable and versioned built environment +for building images and applications. This container is used as base for the +EB corbos Linux SDK workspace. It can be used for local development and for automated +application and image building in a CI environment. + +To improve maintainability and variant support, the EB corbos Linux dev container +is implemented as a set of layers. Each layer is a default Docker container, and +the layer are stacked on top of each other. The overall container is defined using +the configuration file _configuration/build_config.yaml_. It defines the base container, +some metadata like the container version, and the used layers, as list of relative paths +to the folders containing the containers. + +For building the container a Python script, _builder/build_container.py_, is used with takes +care of running the Docker builds for the layers, and stacking the resulting containers. +The dependencies of this script are also contained in _requirements.txt_. +The shell script _builder/build_container_ takes care that a virtual environment with the +dependencies is available and runs the Python script to build the dev container. + +At the moment, the container is distributed as exported image, and this image is generated +using _builder/export_container_. + +There are also some helper scripts which supports and demos how to use the container stand-alone. +These scripts are contained in _usage_. +The script _usage/run_container_ runs the dev container for interactive usage. +The script _usage/run_background_ runs the container in the background. +This script is also used for the robot tests. +The script _usage/stop_container_ stops a dev container running in the background. diff --git a/docs/tests.md b/docs/tests.md new file mode 100644 index 0000000..60f0dd4 --- /dev/null +++ b/docs/tests.md @@ -0,0 +1,16 @@ +# Testing the dev container + +## Tooling tests + +The image build tests are contained in _tests_. +You can run these tests by running the _tests/run_tests_ script. +This script will ensure that a proper Python virtual environment is available +and then run all robot tests contained in _tests_. + +## Image build tests + +To ensure a good test coverage, also the build tests for the EB corbos Linux template workspace should be used. +Running these tests is supported by the shell script _tests/image_build_tests/run_tests_. +This script clones the current EB corbos Linux template workspace, +mounts is into the latest local built dev container, +and runs the image build robot tests from the EB corbos Linux template workspace in this container. From 3c8e98d2b9cf89996647bb77946f2da22a49df33 Mon Sep 17 00:00:00 2001 From: "Thomas Irgang (thir820)" Date: Thu, 24 Oct 2024 14:24:08 +0000 Subject: [PATCH 2/4] spell checker --- .vscode/settings.json | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 .vscode/settings.json diff --git a/.vscode/settings.json b/.vscode/settings.json new file mode 100644 index 0000000..7a016c1 --- /dev/null +++ b/.vscode/settings.json @@ -0,0 +1,6 @@ +{ + "cSpell.language": "en,en-US,en-GB", + "cSpell.words": [ + "Embdgen" + ] +} \ No newline at end of file From 2e5b8ffba6fca3c95f30206150cc231ade8910b3 Mon Sep 17 00:00:00 2001 From: "Thomas Irgang (thir820)" Date: Thu, 24 Oct 2024 14:24:21 +0000 Subject: [PATCH 3/4] fix pr template --- {.gtihub => .github}/pull_request_template.md | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename {.gtihub => .github}/pull_request_template.md (100%) diff --git a/.gtihub/pull_request_template.md b/.github/pull_request_template.md similarity index 100% rename from .gtihub/pull_request_template.md rename to .github/pull_request_template.md From 753fe3c7105c79a2750643ab0cb19f277e6eaba5 Mon Sep 17 00:00:00 2001 From: "Thomas Irgang (thir820)" Date: Thu, 24 Oct 2024 15:16:28 +0000 Subject: [PATCH 4/4] document and cleanup base and pbuilder layer --- docs/layers/base.md | 15 +++++++++++ docs/layers/pbuilder.md | 10 +++++++ layers/base/Dockerfile | 6 ----- layers/base/scripts/bash/ebcl_env | 5 ---- layers/base/scripts/mount/overlay_mount | 26 ------------------ layers/base/scripts/qemu/qemu_aarch64 | 33 ----------------------- layers/base/scripts/qemu/qemu_efi_aarch64 | 26 ------------------ layers/base/scripts/qemu/qemu_efi_x86_64 | 23 ---------------- layers/base/scripts/qemu/qemu_x86_64 | 33 ----------------------- layers/pbuilder/conf/pbuilder/pbuilderrc | 2 +- 10 files changed, 26 insertions(+), 153 deletions(-) delete mode 100644 layers/base/scripts/mount/overlay_mount delete mode 100755 layers/base/scripts/qemu/qemu_aarch64 delete mode 100755 layers/base/scripts/qemu/qemu_efi_aarch64 delete mode 100755 layers/base/scripts/qemu/qemu_efi_x86_64 delete mode 100755 layers/base/scripts/qemu/qemu_x86_64 diff --git a/docs/layers/base.md b/docs/layers/base.md index 85a4056..e29cb4b 100644 --- a/docs/layers/base.md +++ b/docs/layers/base.md @@ -1 +1,16 @@ # Base Layer + +The base layer does the basic configuration needed for all additional layers, and installs the common tools. These configurations are: + +- [Setup the **ebcl** user](https://github.com/Elektrobit/ebcl_dev_container/blob/9727219bfd009b717d7b29d1eb54d42586e9eebd/layers/base/Dockerfile#L3). +- [Setup the apt sources](https://github.com/Elektrobit/ebcl_dev_container/blob/9727219bfd009b717d7b29d1eb54d42586e9eebd/layers/base/Dockerfile#L8). +- [Install the common packages](https://github.com/Elektrobit/ebcl_dev_container/blob/9727219bfd009b717d7b29d1eb54d42586e9eebd/layers/base/Dockerfile#L26). +- [Make the _addon_ packages available](https://github.com/Elektrobit/ebcl_dev_container/blob/9727219bfd009b717d7b29d1eb54d42586e9eebd/layers/base/Dockerfile#L38). +- [Configure the expected folder structure](https://github.com/Elektrobit/ebcl_dev_container/blob/9727219bfd009b717d7b29d1eb54d42586e9eebd/layers/base/Dockerfile#L56). +- [Configure the shell environment and common scripts](https://github.com/Elektrobit/ebcl_dev_container/blob/9727219bfd009b717d7b29d1eb54d42586e9eebd/layers/base/Dockerfile#L70). +- [Prepare a common Python virtual environment](https://github.com/Elektrobit/ebcl_dev_container/blob/9727219bfd009b717d7b29d1eb54d42586e9eebd/layers/base/Dockerfile#L100). + +## Remarks + +- The _bashrc_ is used to source a EB corbos Linux environment setup file _/build/bin/ebcl_env_. This file is _layers/base/scripts/bash/ebcl_env_. +- The base layer provides some helper scripts to generate a user-specific GPG key. See _layers/base/scripts/gpg_. diff --git a/docs/layers/pbuilder.md b/docs/layers/pbuilder.md index cffef7d..144af52 100644 --- a/docs/layers/pbuilder.md +++ b/docs/layers/pbuilder.md @@ -1 +1,11 @@ # PBuilder Layer + +The pbuilder layer adds support for the Debian pbuilder tool. +This tool is used for application packaging. +This layer also provides the helper scripts for application packaging and generating apt repository metadata, see _layers/pbuilder/scripts/pbuilder_ and _layers/pbuilder/scripts/deb_. + +## Remarks + +- By default, pbuilder fails for Ubuntu because the repository archive.ubuntu.com lists the arm64 architecture, but doesn't provide package indices. This is OK from a specification point of view, but causes an apt exit code != 0, which causes a pbuilder failure. The workaround is _layers/pbuilder/conf/pbuilder/G99apt_arch_, which modifies the apt sources and adds and architecture filter. +- Pbuilder is configured ot use the Canonical and Elektrobit apt repositories, see _layers/pbuilder/conf/pbuilder/pbuilderrc_. +- Pbuilder always makes use of the **latest EB corbos Linux release** provided by the public repository. To change this, the file _layers/pbuilder/conf/pbuilder/pbuilderrc_ (_/home/ebcl/.pbuilderrc_) needs to be adapted. diff --git a/layers/base/Dockerfile b/layers/base/Dockerfile index 3ff3252..249e0b5 100644 --- a/layers/base/Dockerfile +++ b/layers/base/Dockerfile @@ -82,12 +82,6 @@ COPY scripts/bash/* /build/bin/ # Copy GnuPG helper scripts COPY scripts/gpg/* /build/bin/ -# Copy QEMU helper scripts -COPY scripts/qemu/* /build/bin/ - -# Copy mount helper scripts -COPY scripts/mount/* /build/bin/ - # Ensure correct attributes for scripts RUN chown -R $CONTAINER_USER:$CONTAINER_USER /build/bin && \ chmod +x /build/bin/* diff --git a/layers/base/scripts/bash/ebcl_env b/layers/base/scripts/bash/ebcl_env index 1ac3f52..ca35726 100755 --- a/layers/base/scripts/bash/ebcl_env +++ b/layers/base/scripts/bash/ebcl_env @@ -45,13 +45,8 @@ if [ -f "$ENV" ]; then source $ENV fi -if [ ! -d "/tmp/build/images" ]; then - bash -c "overlay_mount /build/images" -fi - source /build/venv/bin/activate - # Run user setup if [ -d "/workspace/tools/user_config" ]; then echo "Running user extensions..." diff --git a/layers/base/scripts/mount/overlay_mount b/layers/base/scripts/mount/overlay_mount deleted file mode 100644 index e120e3e..0000000 --- a/layers/base/scripts/mount/overlay_mount +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/bash - -FOLDER=$(realpath $1) - -echo "Mounting $FOLDER as rw using overlay." - -if [ -d "/tmp$FOLDER" ]; then - echo "/tmp$FOLDER folder already exists." - exit 1 -fi - -mkdir -p /tmp/overlay -mkdir -p /tmp$FOLDER - -MOUNT=$(sudo mount | grep "/tmp/overlay") -if [ $? -ne 0 ]; then - sudo mount -t tmpfs tmpfs /tmp/overlay -fi - -mkdir -p /tmp/overlay${FOLDER}/{work,upper} - -sudo mount -t overlay overlay \ - -o lowerdir=$FOLDER,upperdir=/tmp/overlay${FOLDER}/upper,workdir=/tmp/overlay${FOLDER}/work \ - /tmp$FOLDER - -echo $FOLDER is now RW available as /tmp$FOLDER diff --git a/layers/base/scripts/qemu/qemu_aarch64 b/layers/base/scripts/qemu/qemu_aarch64 deleted file mode 100755 index f61e4cf..0000000 --- a/layers/base/scripts/qemu/qemu_aarch64 +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/bash - -DISK=$(realpath $1) - -if [ ! -f $DISK ]; then - echo "No disk image found at ${DISK}!" - exit 1 -fi - -FOLDER=$(dirname "${DISK}") - -INITRD="${FOLDER}/initrd.img" - -if [ ! -f $INITRD ]; then - echo "No initrd found at ${INITRD}!" - exit 1 -fi - -KERNEL="${FOLDER}/vmlinuz" - -if [ ! -f $KERNEL ]; then - echo "No kernel found at ${KERNEL}!" - exit 1 -fi - -qemu-system-aarch64 \ - -machine virt -cpu cortex-a72 -machine type=virt -nographic -m 4G \ - -netdev user,id=mynet0 \ - -device virtio-net-pci,netdev=mynet0 \ - -kernel ${KERNEL} \ - -append "root=LABEL=root rw console=ttyAMA0,115200n8 systemd.machine_id=004711" \ - -initrd ${INITRD} \ - -drive format=raw,file=${DISK},if=virtio diff --git a/layers/base/scripts/qemu/qemu_efi_aarch64 b/layers/base/scripts/qemu/qemu_efi_aarch64 deleted file mode 100755 index 47828c5..0000000 --- a/layers/base/scripts/qemu/qemu_efi_aarch64 +++ /dev/null @@ -1,26 +0,0 @@ -#!/bin/bash - -# get disk image -if [ $# -eq 0 ]; then - echo "ERROR: Image file not found" - echo "Example of Image file path: result_image//.qcow2" - exit 1 -fi - -if [ -f "$1" ]; then - # file exists - use it - image_file=$(realpath $1) -else - # try relative path in build result folder - image_file=/build/result_image/$1 -fi - -qemu-system-aarch64 \ - -machine virt \ - -cpu cortex-a72 \ - -m 4096 \ - -nographic \ - -netdev user,id=mynet0,hostfwd=tcp::2222-:22,hostfwd=tcp::1234-:1234 \ - -device virtio-net-pci,netdev=mynet0 \ - -drive file=${image_file},if=virtio \ - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd diff --git a/layers/base/scripts/qemu/qemu_efi_x86_64 b/layers/base/scripts/qemu/qemu_efi_x86_64 deleted file mode 100755 index 704a57b..0000000 --- a/layers/base/scripts/qemu/qemu_efi_x86_64 +++ /dev/null @@ -1,23 +0,0 @@ -#!/bin/bash - -# get disk image -if [ $# -eq 0 ]; then - echo "ERROR: Image file not found" - echo "Example of Image file path: result_image//.qcow2" - exit 1 -fi - -if [ -f "$1" ]; then - # file exists - use it - image_file=$(realpath $1) -else - # try relative path in build result folder - image_file=/build/result_image/$1 -fi - -qemu-system-x86_64 \ - -m 4096 \ - -nographic \ - -netdev user,id=mynet0,hostfwd=tcp::2222-:22,hostfwd=tcp::1234-:1234 \ - -device virtio-net-pci,netdev=mynet0 \ - -drive file=${image_file},if=virtio diff --git a/layers/base/scripts/qemu/qemu_x86_64 b/layers/base/scripts/qemu/qemu_x86_64 deleted file mode 100755 index f222c5d..0000000 --- a/layers/base/scripts/qemu/qemu_x86_64 +++ /dev/null @@ -1,33 +0,0 @@ -#!/bin/bash - -DISK=$(realpath $1) - -if [ ! -f $DISK ]; then - echo "No disk image found at ${DISK}!" - exit 1 -fi - -FOLDER=$(dirname "${DISK}") - -INITRD="${FOLDER}/initrd.img" - -if [ ! -f $INITRD ]; then - echo "No initrd found at ${INITRD}!" - exit 1 -fi - -KERNEL="${FOLDER}/vmlinuz" - -if [ ! -f $KERNEL ]; then - echo "No kernel found at ${KERNEL}!" - exit 1 -fi - -qemu-system-x86_64 \ - -nographic -m 4G \ - -netdev user,id=mynet0 \ - -device virtio-net-pci,netdev=mynet0 \ - -kernel ${KERNEL} \ - -append "root=LABEL=root rw console=ttyS0,115200n8 systemd.machine_id=004711" \ - -initrd ${INITRD} \ - -drive format=raw,file=${DISK},if=virtio diff --git a/layers/pbuilder/conf/pbuilder/pbuilderrc b/layers/pbuilder/conf/pbuilder/pbuilderrc index a82bdd9..924e77b 100644 --- a/layers/pbuilder/conf/pbuilder/pbuilderrc +++ b/layers/pbuilder/conf/pbuilder/pbuilderrc @@ -1,7 +1,7 @@ HOOKDIR="/home/ebcl/.pbuilder/hooks/" DISTRIBUTION=jammy MIRRORSITE=http://archive.ubuntu.com/ubuntu -OTHERMIRROR="deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy main universe | deb [arch=amd64] http://archive.ubuntu.com/ubuntu jammy-security main universe | deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy-security main universe | deb http://linux.elektrobit.com/eb-corbos-linux/1.1.1 ebcl prod dev" +OTHERMIRROR="deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy main universe | deb [arch=amd64] http://archive.ubuntu.com/ubuntu jammy-security main universe | deb [arch=arm64] http://ports.ubuntu.com/ubuntu-ports jammy-security main universe | deb http://linux.elektrobit.com/eb-corbos-linux/latest ebcl prod dev" PBUILDERSATISFYDEPENDSCMD="/usr/lib/pbuilder/pbuilder-satisfydepends-apt" BUILDRESULT=/build/result_app COMPONENTS="main universe"