Skip to content

feat: architect observability stack with Garage S3 backend#88

Open
ProjectInitiative wants to merge 3 commits intomainfrom
feat/observability-arch-11620278594626914662
Open

feat: architect observability stack with Garage S3 backend#88
ProjectInitiative wants to merge 3 commits intomainfrom
feat/observability-arch-11620278594626914662

Conversation

@ProjectInitiative
Copy link
Owner

Architect and deploy observability stack (Prometheus, Thanos, Loki) on Capstan cluster, backed by Garage S3 on Management Cluster, with RAM optimization for S3 write batching.


PR created automatically by Jules for task 11620278594626914662 started by @ProjectInitiative

- Deployed `kube-prometheus-stack` (Thanos) and `loki` on Capstan cluster (`cc`) in `monitoring` namespace.
- Configured Loki and Thanos to use Garage S3 on Management Cluster (`mc`) as backend.
- Enabled Ingress for Garage S3 on `mc` via Tailscale (`s3.moonwake.io`).
- Tuned Loki to utilize RAM and batch writes (1.5MB chunks, 2h max age) to reduce S3 ops.
- Configured Vault Secrets injection for S3 credentials in `monitoring` namespace.
- Disabled CRD installation in `kube-prometheus-stack` to avoid conflict with `prometheus-operator-crds`.

Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
@google-labs-jules
Copy link
Contributor

👋 Jules, reporting for duty! I'm here to lend a hand with this pull request.

When you start a review, I'll add a 👀 emoji to each comment to let you know I've read it. I'll focus on feedback directed at me and will do my best to stay out of conversations between you and other bots or reviewers to keep the noise down.

I'll push a commit with your requested changes shortly after. Please note there might be a delay between these steps, but rest assured I'm on the job!

For more direct control, you can switch me to Reactive Mode. When this mode is on, I will only act on comments where you specifically mention me with @jules. You can find this option in the Pull Request section of your global Jules UI settings. You can always switch back!

New to Jules? Learn more at jules.google/docs.


For security, I will only act on instructions from the user who triggered this task.

google-labs-jules bot and others added 2 commits February 26, 2026 16:48
- Updated `DeterminateSystems/nix-installer-action` from v10 to v16 in all workflows (`manifest-diff.yaml`, `build-push-nixos-remote-builder.yaml`, `build-push-pulumi-cmp.yaml`) to fix "Invalid URL" error.
- Fixed namespace mismatch for `observability` vaultSecrets (set to `monitoring`).
- Corrected Loki datasource URL in `apps/base/observability/values.yaml` to use port 3100.

Co-authored-by: ProjectInitiative <6314611+ProjectInitiative@users.noreply.github.com>
@github-actions
Copy link

Manifest Changes Detected

Click to expand diff
Only in manifests-pr/1-manifest: argoproj.io_v1alpha1-application-argocd-cc-observability.yaml
diff -r -u manifests-main/1-manifest/argoproj.io_v1alpha1-application-argocd-mc-garage.yaml manifests-pr/1-manifest/argoproj.io_v1alpha1-application-argocd-mc-garage.yaml
--- manifests-main/1-manifest/argoproj.io_v1alpha1-application-argocd-mc-garage.yaml	2026-02-27 03:53:29.806693751 +0000
+++ manifests-pr/1-manifest/argoproj.io_v1alpha1-application-argocd-mc-garage.yaml	2026-02-27 03:53:44.310783410 +0000
@@ -42,6 +42,20 @@
           extraVolumeMounts:
             - name: snapshots
               mountPath: /snapshots
+          ingress:
+            s3:
+              api:
+                enabled: true
+                ingressClassName: tailscale
+                hosts:
+                  - host: s3.moonwake.io
+                    paths:
+                      - path: /
+                        pathType: Prefix
+                tls:
+                  - secretName: garage-ingress-cert
+                    hosts:
+                      - s3.moonwake.io
       - name: KUSTOMIZE_PATCH
         value: |
           - op: add
@@ -76,7 +90,7 @@
               serviceAccount: operator-auth-sa
             method: kubernetes
             mount: kubernetes_cluster_mc
-            namespace: production
+            namespace: monitoring
         target:
           kind: VaultAuth
           name: placeholder-auth
@@ -104,7 +118,7 @@
               create: true
               name: garage-rpc-secret
             mount: k8s
-            namespace: production
+            namespace: monitoring
             path: garage/rpc
             type: kv-v2
             vaultAuthRef: operator-auth
diff -r -u manifests-main/1-manifest/argoproj.io_v1alpha1-application-argocd-mc-postgres-cluster.yaml manifests-pr/1-manifest/argoproj.io_v1alpha1-application-argocd-mc-postgres-cluster.yaml
--- manifests-main/1-manifest/argoproj.io_v1alpha1-application-argocd-mc-postgres-cluster.yaml	2026-02-27 03:53:29.806562578 +0000
+++ manifests-pr/1-manifest/argoproj.io_v1alpha1-application-argocd-mc-postgres-cluster.yaml	2026-02-27 03:53:44.311123230 +0000
@@ -32,7 +32,7 @@
               serviceAccount: operator-auth-sa
             method: kubernetes
             mount: kubernetes_cluster_mc
-            namespace: production
+            namespace: monitoring
         target:
           kind: VaultAuth
           name: placeholder-auth
@@ -60,7 +60,7 @@
               create: true
               name: postgres-cluster-s3
             mount: k8s
-            namespace: production
+            namespace: monitoring
             path: cnpg/s3
             type: kv-v2
             vaultAuthRef: operator-auth

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant