CUBE-92 - Add cloud compose and deployment workflow #111

WashingtonKK · 2025-12-10T14:36:07Z

What type of PR is this?

This is a feature

What does this do?

Adds cloud compose workflow that deploys cube to the cloud.
Fixes creation of cvm with cube-agent and ollama using buildroot
Adds dev env for cube

Which issue(s) does this PR fix/relate to?

Resolves Deploy cvm on tdx machine #91
Resolves Deploy dev env for cube #92
Resolves Create cloud init script for cube new architecture #83

Have you included tests for your changes?

Did you document any new/modified features?

Notes

SammyOina · 2025-12-10T15:02:25Z

.github/workflows/deploy-cloud.yaml

should depend on images publication workflow

SammyOina · 2025-12-15T08:28:19Z

hal/buildroot/qemu.sh

    -fsdev local,id=cert_fs,path=$CERTS_PATH,security_model=mapped \
    -device virtio-9p-pci,fsdev=cert_fs,mount_tag=certs_share \
-    -device vhost-vsock-pci,guest-cid=6 \
+    -device vhost-vsock-pci,guest-cid=42 \


we don't need or use vsock

SammyOina · 2025-12-15T08:28:21Z

hal/buildroot/qemu.sh

    -fsdev local,id=cert_fs,path=$CERTS_PATH,security_model=mapped \
    -device virtio-9p-pci,fsdev=cert_fs,mount_tag=certs_share \
-    -device vhost-vsock-pci,guest-cid=6 \
+    -device vhost-vsock-pci,guest-cid=42 \


we don't need or use vsock

SammyOina · 2025-12-15T08:28:55Z

hal/buildroot/linux/board/cube/linux.config

+CONFIG_VIRTIO_BLK=y
+CONFIG_VIRTIO_NET=y


use the same config as cocos

https://github.com/ultravioletrs/cocos/blob/7807e3811c2c504e7483647647910a0ae9ae24a0/hal/linux/board/cocos/linux.config#L16-L17

Those are on cocos

SammyOina · 2025-12-16T10:45:05Z

hal/buildroot/linux/configs/cube_defconfig

buildroot changes are not expected in this pr

...linux/board/cube/overlay/etc/systemd/system/multi-user.target.wants/mount-host-certs.service

hal/buildroot/linux/board/cube/overlay/usr/lib/systemd/system/mount-host-certs.service

hal/buildroot/linux/board/cube/overlay/usr/local/bin/mount-host-certs.sh

hal/buildroot/linux/board/cube/post-build.sh

hal/buildroot/linux/configs/cube_defconfig

hal/buildroot/cvm-monitor.sh

SammyOina · 2025-12-18T13:59:25Z

hal/buildroot/cvm-monitor.sh

+        print_help
+        exit 1
+        ;;
+esac


Suggested change

esac

esac

same for any other instances

SammyOina · 2025-12-18T14:00:45Z

hal/buildroot/qemu.sh

    -machine q35 \
    -enable-kvm \
-    -netdev user,id=vmnic,hostfwd=tcp::6190-:22,hostfwd=tcp::6191-:80,hostfwd=tcp::6192-:443,hostfwd=tcp::6193-:6193,dns=8.8.8.8 \
+    -netdev user,id=vmnic,hostfwd=tcp::6190-:22,hostfwd=tcp::6191-:80,hostfwd=tcp::6192-:443,hostfwd=tcp::6193-:7001,hostfwd=tcp::6194-:11434,hostfwd=tcp::6195-:8000,dns=8.8.8.8 \


since this same config is used in our cvm make sure no one can ssh to our cvm, for examples root password

SammyOina · 2025-12-22T12:07:00Z

hal/terraform/azure/main.tf

we do not need to duplicate the terraform scripts since we can reuse the ones on cocos-infra, the only thing that needs to change is cloud init script

SammyOina · 2025-12-22T12:10:13Z

docker/traefik/dynamic.toml

+  priority = 1
+  [http.routers.cube-ui.tls]
+    certResolver = "letsEncrypt"
+    [[http.routers.cube-ui.tls.domains]]


The dynamic.toml file uses CUBE_DOMAIN placeholders (lines 9, 16, 19), but no substitution mechanism exists in the codebase. The main compose.yaml uses dynamic.yaml (without placeholders), but cloud-compose.yaml mounts dynamic.toml directly. This will cause deployment failures. Either:

Add a CUBE_DOMAIN variable to .env and preprocess dynamic.toml before deployment (using envsubst in a Makefile target or entrypoint script), or
Create a properly templated dynamic.toml with actual domains for cloud deployments

same applies for other placeholders added throughout the code

SammyOina · 2025-12-22T12:12:03Z

hal/buildroot/cvm-monitor.sh

+CHECK_INTERVAL=30
+LOG_DIR="/tmp/cube-logs"
+LOG_FILE="$LOG_DIR/cube-cvm-monitor.log"
+QEMU_SCRIPT="/home/washington/cube/hal/buildroot/qemu.sh"


fix this line (we work on a public repo), also document this script

SammyOina · 2025-12-22T12:13:24Z

hal/buildroot/cvm-monitor.sh

+        pgrep -f "qemu-system-x86_64" -l | head -5 || echo "No QEMU processes found"
+
+        if [ -f "$PIDFILE" ]; then
+            local pid=$(cat "$PIDFILE" 2>/dev/null)


uses local pid=$(...) inside the case statement's "status" block, which is not within a function. This will cause a runtime error. Either move this logic to a function or use a regular variable.

smithjilks · 2025-12-22T14:34:04Z

docker/traefik/letsencrypt/.gitkeep

@@ -0,0 +1,2 @@
+# Copyright (c) Ultraviolet
+# SPDX-License-Identifier: Apache-2.0


smithjilks · 2025-12-22T14:34:13Z

docker/traefik/ssl/certs/.gitkeep

@@ -0,0 +1,2 @@
+# Copyright (c) Ultraviolet
+# SPDX-License-Identifier: Apache-2.0


smithjilks · 2025-12-22T14:47:29Z

.github/workflows/deploy-cloud.yaml

+        type: choice
+        options:
+          - dev
+          - staging


Prod and staging env are supposed to be k8s

smithjilks · 2025-12-22T14:49:31Z

.github/workflows/deploy-cloud.yaml

+jobs:
+  deploy-cloud:
+    runs-on: ubuntu-latest
+    environment: ${{ inputs.environment || 'dev' }}


Is this environment being used?

Yes, when triggering the pipeline manually from github.

SammyOina · 2025-12-22T15:58:47Z

docker/.env

 UV_CUBE_PROXY_SERVER_CERT=
 UV_CUBE_PROXY_SERVER_KEY=
-UV_CUBE_AGENT_URL=http://cube-agent:8901
+UV_CUBE_AGENT_URL=https://10.172.192.41:6193


SammyOina · 2025-12-22T16:00:49Z

.github/workflows/deploy-cloud.yaml

+  push:
+    branches:
+      - main
+      - cube-92


remove before merge

SammyOina · 2025-12-22T16:02:10Z

.github/workflows/deploy-cloud.yaml

+            fi
+
+            # Initialize acme.json if it doesn't exist
+            # if [ ! -f "traefik/ssl/certs/acme.json" ]; then


The conditional logic is commented out, so acme.json is overwritten with {} on every deployment.

SammyOina · 2025-12-22T16:03:44Z

hal/ubuntu/cube-agent-config.yml

this cloud init file should be the same one used in ubuntu dir, we don't need a tf directory as well, this can simply be a section in docs

SammyOina · 2025-12-22T16:07:13Z

hal/ubuntu/cube-agent-config.yml

+    content: |
+      UV_CUBE_AGENT_LOG_LEVEL=info
+      UV_CUBE_AGENT_HOST=0.0.0.0
+      UV_CUBE_AGENT_PORT=7001
+      UV_CUBE_AGENT_INSTANCE_ID=cube-agent-01
+      UV_CUBE_AGENT_TARGET_URL=http://localhost:11434


include all possible agent configs even if they will be left empty, same for writing tls/mtls certs into cloud image, same goes for a mechanism for custom models, allow for these to be configurable

SammyOina · 2025-12-22T16:08:22Z

hal/terraform/cloud-init/cube-agent-config.yml

+
+      # Pull default model
+      echo "Pulling tinyllama model..."
+      /usr/local/bin/ollama pull tinyllama:1.1b


same as in hal pull the base models supported in cube

SammyOina · 2025-12-22T16:08:49Z

hal/ubuntu/cube-agent-config.yml

+
+  # Install Ollama
+  - echo "Installing Ollama..."
+  - curl -fsSL https://ollama.com/install.sh | sh


pin to a specific version

conditonally support vllm since we support both ollama and vllm or create another cloud init script if needed

pin to a specific version

ollama download there is no specific version to pin to.

- Add virtio-9p runtime certificate mounting (not baked into image) - Configure QEMU with certificate sharing via virtio-9p-pci device - Update TDX guest CID from 6 to 42 to avoid conflicts - Add kernel support for VIRTIO_BLK and VIRTIO_NET - Enable systemd-networkd and systemd-resolved for automatic network config - Add buildroot overlay configuration for network and systemd services - Implement proven cocos approach with fstab-based 9p mounting - Add BR2_PACKAGE_9PFS for 9p filesystem support in userspace - Create post-build script for automatic mount point creation - Fix buildroot overlay path to include /linux/ directory This enables automatic network configuration and runtime certificate mounting via virtio-9p for secure mTLS communication without baking certificates into the VM image. Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> remove legacy config Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> update network config Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> update agent url Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test deployment Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test deployment Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test deployment Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test deployment Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test deployment Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test deployment Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test deployment Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test deployment Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test deployment Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

- Update SMQ_RELEASE_TAG from latest to v0.18.3 across all compose files - Add fallback version to docker image tags in compose files - Ensures consistent SuperMQ version deployment Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> update agent target url Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> Add cvm service Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> Add cvm service Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> squash Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

SammyOina · 2025-12-23T09:59:48Z

hal/buildroot/README.md

+cd cocos-infra/gcp
+cat > terraform.tfvars <<EOF
+cloud_init_config = "/path/to/cube/hal/ubuntu/cube-agent-config.yml"
+# ... other variables


be specific a user needs to rely on this readme without looking into the code

SammyOina · 2025-12-23T10:06:35Z

hal/ubuntu/qemu.sh

+  - chown -R ollama:ollama /home/ollama
+  - curl -fsSL https://ollama.com/install.sh | sh
+  - git clone https://github.com/ultravioletrs/cube.git /tmp/cube
+  - cd /tmp/cube && git fetch origin pull/88/head:pr-88 && git checkout pr-88


fix before merge

SammyOina · 2025-12-23T10:12:52Z

hal/buildroot/README.md

+  --allow=tcp:7001 \
+  --source-ranges=0.0.0.0/0 \
+  --target-tags=cube-ai-cvm-01 \
+  --project=valued-base-354714


this is a specific project, should not be exposed

SammyOina · 2025-12-23T10:13:28Z

hal/buildroot/README.md

+gcloud compute firewall-rules create allow-cube-agent-7001 \
+  --allow=tcp:7001 \
+  --source-ranges=0.0.0.0/0 \
+  --target-tags=cube-ai-cvm-01 \


are there instructions to set this tag on the terraform scripts?

SammyOina · 2025-12-23T10:15:40Z

hal/ubuntu/cube-agent-config.yml

+      sed -i "s|__VLLM_MODEL__|$VLLM_MODEL|g" /etc/systemd/system/vllm.service
+
+      echo "Installing vLLM..."
+      pip3 install vllm


pin to a specific vllm version, there was also a comment simillarly for ollama which was not resolved

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> update deployment workflow Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

SammyOina · 2025-12-23T13:40:43Z

docker/.env

+UV_CUBE_AGENT_CLIENT_CERT=/etc/cube/certs/client.crt
+UV_CUBE_AGENT_CLIENT_KEY=/etc/cube/certs/client.key
+UV_CUBE_AGENT_CLIENT_CA=/etc/cube/certs/ca.pem


should be templated or overridden, these files are not tracked in repo or mounted with docker

SammyOina · 2025-12-23T13:41:43Z

docker/.env

+CUBE_AI_ATTESTATION_URL=http://cube-proxy:8900
+CUBE_AUDIT_LOGS_URL=http://opensearch:9200


audit logs should be fetched from proxy same for attestation this only needs one env variable

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

SammyOina · 2025-12-23T15:18:25Z

docker/.env

+CUBE_AI_ATTESTATION_URL=http://cube-proxy:${UV_CUBE_PROXY_PORT}
+CUBE_AUDIT_LOGS_URL=http://cube-proxy:${UV_CUBE_PROXY_PORT}


should all be a single variable open an issue on ui repo

https://github.com/absmach/magistrala-ui/issues/1315

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

WashingtonKK force-pushed the cube-92 branch from dafd9d0 to 809d2a0 Compare December 10, 2025 14:44

This was linked to issues Dec 10, 2025

Create cloud init script for cube new architecture #83

Open

Deploy cvm on tdx machine #91

Open

Deploy dev env for cube #92

Open

SammyOina requested changes Dec 10, 2025

View reviewed changes

.github/workflows/deploy-cloud.yaml

Copy link

Contributor

SammyOina Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should depend on images publication workflow

SammyOina requested changes Dec 15, 2025

View reviewed changes

SammyOina reviewed Dec 15, 2025

View reviewed changes

WashingtonKK force-pushed the cube-92 branch 3 times, most recently from df3a318 to f5472d1 Compare December 16, 2025 09:08

SammyOina requested changes Dec 16, 2025

View reviewed changes

hal/buildroot/linux/configs/cube_defconfig

Copy link

Contributor

SammyOina Dec 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

buildroot changes are not expected in this pr

WashingtonKK force-pushed the cube-92 branch 4 times, most recently from cc32851 to ce96647 Compare December 18, 2025 13:59

SammyOina requested changes Dec 18, 2025

View reviewed changes

SammyOina requested changes Dec 22, 2025

View reviewed changes

SammyOina requested a review from smithjilks December 22, 2025 12:08

SammyOina requested changes Dec 22, 2025

View reviewed changes

smithjilks requested changes Dec 22, 2025

View reviewed changes

SammyOina requested changes Dec 22, 2025

View reviewed changes

WashingtonKK added 8 commits December 23, 2025 01:03

CUBE-92: Address comments

b495583

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Add MG_UI_ALLOW_UNVERIFIED_USER to ui

1af6261

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Update audit logs and attestation url

dfc976a

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

Update audit logs and attestation url

095ae70

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

use secret for admin password

9768b85

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

WashingtonKK added 9 commits December 23, 2025 01:04

revert changes in hal/buildroot

bcaca40

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

use terraform for gcp vm creation

70da67a

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

address comments

bddffa0

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

update deployment workflow

b06a28c

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

update deployment workflow

f13cb96

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

update deployment workflow

0229827

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

update deployment workflow

657035f

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

add cloud init script

8109c44

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

address comments

955a7ce

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

WashingtonKK force-pushed the cube-92 branch from 7582770 to 955a7ce Compare December 22, 2025 22:04

WashingtonKK added 2 commits December 23, 2025 08:51

refactor cloud init and docs

a29b27d

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

conditionally allow vllms

683bcd2

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

SammyOina requested changes Dec 23, 2025

View reviewed changes

Update cloud compose

b798aaa

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> test Signed-off-by: WashingtonKK <washingtonkigan@gmail.com> update deployment workflow Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

WashingtonKK force-pushed the cube-92 branch from ae52cb6 to b798aaa Compare December 23, 2025 11:31

update guardrails url

ffa3987

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

SammyOina requested changes Dec 23, 2025

View reviewed changes

update proxy url and agent certs

d7927a3

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

SammyOina reviewed Dec 23, 2025

View reviewed changes

update agent certs mount

6fff658

Signed-off-by: WashingtonKK <washingtonkigan@gmail.com>

SammyOina approved these changes Dec 23, 2025

View reviewed changes

		@@ -0,0 +1,2 @@
		# Copyright (c) Ultraviolet
		# SPDX-License-Identifier: Apache-2.0 No newline at end of file

		CUBE_AI_ATTESTATION_URL=http://cube-proxy:8900
		CUBE_AUDIT_LOGS_URL=http://opensearch:9200

		CUBE_AI_ATTESTATION_URL=http://cube-proxy:${UV_CUBE_PROXY_PORT}
		CUBE_AUDIT_LOGS_URL=http://cube-proxy:${UV_CUBE_PROXY_PORT}

		CONFIG_VIRTIO_BLK=y
		CONFIG_VIRTIO_NET=y

CUBE-92 - Add cloud compose and deployment workflow #111

Are you sure you want to change the base?

CUBE-92 - Add cloud compose and deployment workflow #111

Uh oh!

Conversation

WashingtonKK commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What type of PR is this?

What does this do?

Which issue(s) does this PR fix/relate to?

Have you included tests for your changes?

Did you document any new/modified features?

Notes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WashingtonKK Dec 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WashingtonKK Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WashingtonKK commented Dec 10, 2025 •

edited

Loading

WashingtonKK Dec 18, 2025 •

edited

Loading

WashingtonKK Dec 22, 2025 •

edited

Loading