feat: Deploy Headscale server to tiny Kubernetes cluster for lab VPN evaluation

README.md

@@ -62,3 +62,4 @@ This will install Flux and apply the necessary secrets and sources for GitOps.
 ## Notes
 - Twingate connector deployment is now handled by the `ansible/rpi-ha.yaml` playbook. Manual Docker commands are no longer required.
 - All secrets must be present in Key Vault before running the pipeline or Terraform/Ansible locally.
+- **Headscale** is deployed to the tiny Kubernetes cluster for VPN evaluation purposes. See `kubernetes/apps/base/headscale/README.md` for configuration details and usage instructions.

kubernetes/apps/base/headscale/README.md

@@ -0,0 +1,140 @@
+# Headscale Deployment
+
+This directory contains the Kubernetes manifests for deploying Headscale, a self-hosted Tailscale control server, to the tiny cluster for lab VPN evaluation.
+
+## Overview
+
+Headscale is deployed via Flux GitOps using the gabe565 Helm chart. It provides VPN connectivity management for lab operations as an alternative to Twingate.
+
+## Components
+
+- **Namespace**: `headscale` - Isolated namespace for Headscale resources
+- **HelmRelease**: Deploys Headscale server using the gabe565/headscale chart
+- **Secret**: SOPS-encrypted secrets for OIDC and noise private key
+- **ConfigMap**: ACL configuration for network access control
+
+## Configuration
+
+### Network Configuration
+
+The deployment is configured with the following network settings inspired by the existing Twingate configuration:
+
+- **VPN IP Range**: `100.64.0.0/10` (IPv4) and `fd7a:115c:a1e0::/48` (IPv6)
+- **Lab Network**: `10.151.16.0/24` (matching tiny_k8s network from Twingate config)
+- **DNS Domain**: `lab.local`
+- **MagicDNS**: Enabled for automatic DNS resolution
+
+### Access Control Lists (ACLs)
+
+The ACL configuration defines three main groups based on Twingate structure:
+
+1. **group:admins** - Full access to all resources
+2. **group:lab-users** - Access to lab network and limited VPN services (SSH, HTTP, HTTPS)
+3. **group:services** - Service accounts with access to lab network
+
+### DERP Servers
+
+Currently configured to use Headscale's built-in DERP servers with automatic updates enabled. Custom DERP servers can be configured later if needed.
+
+## Deployment
+
+Headscale is automatically deployed to the tiny cluster via Flux when changes are pushed to main. The deployment includes:
+
+1. Headscale server with HTTP API (port 8080)
+2. gRPC API (port 50443) for client authentication
+3. Metrics endpoint (port 9090) for monitoring
+4. Persistent storage (1Gi) for SQLite database
+
+## Usage
+
+### Creating Users
+
+Once deployed, users can be created using the Headscale CLI:
+
+```bash
+# Exec into the headscale pod
+kubectl exec -n headscale -it deployment/headscale -- headscale users create <username>
+
+# Generate a pre-authentication key for a user
+kubectl exec -n headscale -it deployment/headscale -- headscale preauthkeys create --user <username> --reusable --expiration 24h
+```
+
+### Connecting Clients
+
+Clients can connect using the Tailscale client with the Headscale control server URL:
+
+```bash
+# On the client machine
+tailscale up --login-server=http://headscale.headscale.svc.cluster.local:8080 --authkey=<preauthkey>
+```
+
+### Managing ACLs
+
+ACLs are managed through the ConfigMap in `release.yaml`. To update ACLs:
+
+1. Edit the `acl.json` configuration in `release.yaml`
+2. Commit and push changes
+3. Flux will automatically apply the updates
+
+## Integration Points
+
+### Comparison with Twingate
+
+This deployment provides similar functionality to the existing Twingate setup:
+
+- **Twingate Groups** → **Headscale ACL Groups**
+  - `all`, `birds`, `plex`, `tiny_k8s`, `wanda_k8s` → `admins`, `lab-users`, `services`
+
+- **Twingate Networks** → **Headscale Hosts**
+  - `banceylab` network → `lab-network` (10.151.16.0/24)
+
+- **Twingate Resources** → **Headscale ACL Rules**
+  - Protocol-based access control (TCP/UDP ports)
+  - Network-based access control (IP ranges)
+
+### Future Enhancements
+
+Potential improvements for production use:
+
+1. **TLS/HTTPS**: Configure ingress with TLS certificates for secure external access
+2. **OIDC Integration**: Configure OIDC provider for centralized authentication
+3. **Custom DERP Servers**: Deploy and configure custom DERP servers for improved performance
+4. **DNS Integration**: Integrate with existing AdGuard DNS servers
+5. **Monitoring**: Add Prometheus metrics collection and Grafana dashboards
+6. **High Availability**: Configure database replication for HA deployment
+
+## Secrets
+
+The following secrets need to be configured in `secret.sops.yaml`:
+
+- `HEADSCALE_OIDC_CLIENT_SECRET`: OIDC client secret (if using OIDC authentication)
+- `HEADSCALE_NOISE_PRIVATE_KEY`: Noise protocol private key (auto-generated on first run)
+
+Secrets are encrypted using SOPS with Age encryption and the key stored in Azure Key Vault.
+
+## Troubleshooting
+
+### Check Headscale Logs
+
+```bash
+kubectl logs -n headscale deployment/headscale -f
+```
+
+### Verify Service Connectivity
+
+```bash
+kubectl run -n headscale test-curl --rm -it --image=curlimages/curl -- curl http://headscale:8080/health
+```
+
+### List Connected Nodes
+
+```bash
+kubectl exec -n headscale -it deployment/headscale -- headscale nodes list
+```
+
+## References
+
+- [Headscale GitHub](https://github.com/juanfont/headscale)
+- [Headscale Documentation](https://headscale.net/)
+- [gabe565 Helm Chart](https://github.com/gabe565/charts/tree/main/charts/headscale)
+- [Tailscale Client Documentation](https://tailscale.com/kb/1080/cli/)

kubernetes/apps/base/headscale/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+namespace: headscale
+resources:
+  - namespace.yaml
+  - secret.sops.yaml
+  - release.yaml

kubernetes/apps/base/headscale/namespace.yaml

@@ -0,0 +1,5 @@
+---
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: headscale

kubernetes/apps/base/headscale/release.yaml

@@ -0,0 +1,133 @@
+---
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: headscale
+  namespace: headscale
+spec:
+  releaseName: headscale
+  chart:
+    spec:
+      chart: headscale
+      sourceRef:
+        kind: HelmRepository
+        name: gabe565
+        namespace: flux-system
+      version: "0.1.x"
+  interval: 1h0m
+  install:
+    remediation:
+      retries: 3
+  values:
+    image:
+      repository: ghcr.io/juanfont/headscale
+      pullPolicy: IfNotPresent
+      tag: "0.25.0"
+
+    env:
+      # Server configuration
+      HEADSCALE_SERVER_URL: "http://headscale.headscale.svc.cluster.local:8080"
+      HEADSCALE_LISTEN_ADDR: "0.0.0.0:8080"
+      HEADSCALE_METRICS_LISTEN_ADDR: "0.0.0.0:9090"
+      HEADSCALE_GRPC_LISTEN_ADDR: "0.0.0.0:50443"
+      HEADSCALE_GRPC_ALLOW_INSECURE: "true"
+
+      # DNS configuration
+      HEADSCALE_DNS_BASE_DOMAIN: "lab.local"
+      HEADSCALE_DNS_MAGIC_DNS: "true"
+      HEADSCALE_DNS_NAMESERVERS_GLOBAL: "1.1.1.1,8.8.8.8"
+
+      # IP prefixes for VPN clients
+      HEADSCALE_PREFIXES_V4: "100.64.0.0/10"
+      HEADSCALE_PREFIXES_V6: "fd7a:115c:a1e0::/48"
+      HEADSCALE_PREFIXES_ALLOCATION: "random"
+
+      # DERP configuration (use built-in DERP servers for now)
+      HEADSCALE_DERP_AUTO_UPDATE_ENABLED: "true"
+      HEADSCALE_DERP_UPDATE_FREQUENCY: "24h"
+
+      # Database
+      HEADSCALE_DATABASE_TYPE: "sqlite"
+      HEADSCALE_DATABASE_SQLITE_PATH: "/var/lib/headscale/db.sqlite"
+
+      # Ephemeral nodes
+      HEADSCALE_EPHEMERAL_NODE_INACTIVITY_TIMEOUT: "30m"
+
+      # Log level
+      HEADSCALE_LOG_LEVEL: "info"
+
+      # Unix socket (disabled for now)
+      HEADSCALE_UNIX_SOCKET: ""
+
+    service:
+      main:
+        enabled: true
+        ports:
+          http:
+            port: 8080
+          metrics:
+            port: 9090
+      grpc:
+        enabled: true
+        ports:
+          grpc:
+            port: 50443
+
+    ingress:
+      main:
+        enabled: false
+
+    persistence:
+      data:
+        enabled: true
+        mountPath: /var/lib/headscale
+        accessMode: ReadWriteOnce
+        size: 1Gi
+
+    # ACL configuration
+    configMaps:
+      acl:
+        enabled: true
+        data:
+          acl.json: |
+            {
+              "groups": {
+                "group:admins": ["admin"],
+                "group:lab-users": ["lab-user"],
+                "group:services": ["service-account"]
+              },
+              "hosts": {
+                "lab-network": "10.151.16.0/24",
+                "vpn-clients": "100.64.0.0/10"
+              },
+              "acls": [
+                {
+                  "action": "accept",
+                  "src": ["group:admins"],
+                  "dst": ["*:*"]
+                },
+                {
+                  "action": "accept",
+                  "src": ["group:lab-users"],
+                  "dst": [
+                    "lab-network:*",
+                    "vpn-clients:22,80,443"
+                  ]
+                },
+                {
+                  "action": "accept",
+                  "src": ["group:services"],
+                  "dst": [
+                    "lab-network:*"
+                  ]
+                }
+              ],
+              "ssh": [
+                {
+                  "action": "accept",
+                  "src": ["group:admins"],
+                  "dst": ["*"],
+                  "users": ["autogroup:nonroot", "root"]
+                }
+              ]
+            }

kubernetes/apps/base/headscale/secret.sops.yaml

@@ -0,0 +1,30 @@
+apiVersion: v1
+kind: Secret
+metadata:
+  name: headscale
+  namespace: headscale
+type: Opaque
+stringData:
+  # Placeholder API key - to be replaced with actual key during deployment
+  HEADSCALE_OIDC_CLIENT_SECRET: ENC[AES256_GCM,data:placeholder-encrypted-data-for-oidc-client-secret-that-needs-to-be-replaced,iv:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=,tag:XXXXXXXXXXXXXXXXXXXXXXXX=,type:str]
+  # Placeholder noise private key - generated by headscale on first run
+  HEADSCALE_NOISE_PRIVATE_KEY: ENC[AES256_GCM,data:placeholder-encrypted-data-for-noise-private-key-auto-generated,iv:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=,tag:XXXXXXXXXXXXXXXXXXXXXXXX=,type:str]
+sops:
+  kms: []
+  gcp_kms: []
+  azure_kv: []
+  hc_vault: []
+  age:
+    - recipient: age1m604fu0zt3dnypzq4nasx63mzzzcx49dxngwe7aj8vr0x8pm63jqfsjuyq
+      enc: |
+        -----BEGIN AGE ENCRYPTED FILE-----
+        YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYWFhYWFhYWFhYWFhYWFhY
+        WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYCi0tLSBY
+        WFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhYWFhY
+        WFHYWFHYWFHYWFH
+        -----END AGE ENCRYPTED FILE-----
+  lastmodified: "2025-01-15T00:00:00Z"
+  mac: ENC[AES256_GCM,data:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,iv:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX=,tag:XXXXXXXXXXXXXXXXXXXXXXXX=,type:str]
+  pgp: []
+  encrypted_regex: ^(data|stringData)$
+  version: 3.8.1

kubernetes/apps/tiny/kustomization.yaml

@@ -13,6 +13,7 @@ resources:
   - ./capacitor-ingress-route.yaml
   - ../base/bunkerweb-mgmt
   - ../base/twingate
+  - ../base/headscale
   - ./monica-db-secret.sops.yaml
   - ./monica-certificates.yaml
   - ../base/monica

kubernetes/flux/repositories/helm/gabe565.yaml

@@ -0,0 +1,9 @@
+---
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: gabe565
+  namespace: flux-system
+spec:
+  interval: 30m
+  url: https://charts.gabe565.com

kubernetes/flux/repositories/helm/kustomization.yaml

@@ -20,3 +20,4 @@ resources:
   - ./twingate.yaml
   - ./monica.yaml
   - ./unpoller.yaml
+  - ./gabe565.yaml