K8SPG-911 Documented TDE

docs/assets/fragments/vault-enable-kv.txt

@@ -0,0 +1,41 @@
+At this step you need to configure Vault and enable secrets engine within it. To do so you must first authenticate in Vault.
+
+When you started Vault, it generates and starts with a [root token :octicons-link-external-16:](https://developer.hashicorp.com/vault/docs/concepts/tokens) that provides full access to Vault. Use this token to authenticate.
+
+Run the following command on a leader node. The remaining ones will synchronize from the leader.
+
+1. Extract the Vault root token from the file where you saved the init response output:
+
+    ```bash
+    cat ${WORKDIR}/vault-init | jq -r ".root_token"
+    ```
+
+    ??? example "Sample output"
+
+        ```{.text .no-copy}
+        hvs.*************Jg9r
+        ```
+
+2. Connect to Vault Pod:
+
+    ```bash
+    kubectl exec -it vault-0 -n $NAMESPACE -- /bin/sh
+    ```
+
+3. Authenticate in Vault with this token:
+
+    ```bash
+    vault login hvs.*************Jg9r
+    ```
+
+4. Enable the secrets engine at the mount path. The following command enables KV secrets engine v2 at the `tde` mount-path:
+
+    ```bash
+    vault secrets enable --version=2 -path=tde kv
+    ```
+
+    ??? example "Sample output"
+
+        ```{.text .no-copy}
+        Success! Enabled the kv secrets engine at: tde/
+        ```

docs/assets/fragments/vault-generate-tls-certs.txt

@@ -0,0 +1,158 @@
+## Generate TLS certificates 
+
+To use TLS, you'll need the following certificates:
+
+* A private key for the Vault server
+* A certificate for the Vault server signed by the Kubernetes CA
+* The Kubernetes CA certificate
+
+These files store sensitive information. Make sure to keep them in a secure location.
+
+### Generate the private key
+
+Generate a private key for the Vault server:
+
+```bash
+openssl genrsa -out ${WORKDIR}/vault.key 2048
+```
+
+### Create the Certificate Signing Request (CSR)
+
+A Certificate Signing Request (CSR) is a file that contains information about your server and the certificate you need. You create it using your private key, and then submit it to Kubernetes to get a certificate signed by the Kubernetes Certificate Authority (CA). The signed certificate proves your server's identity and enables secure TLS connections.
+
+1. Create the Certificate Signing Request configuration file:
+
+    Specify the certificate details that Kubernetes needs to sign your certificate:
+
+    * **Request settings** (`[req]`): References the sections for certificate extensions and distinguished name. The distinguished name section is left empty as Kubernetes will populate it automatically.
+    * **Certificate extensions** (`[v3_req]`): Defines how the certificate can be used. `serverAuth` allows the certificate for server authentication, while `keyUsage` specifies the cryptographic operations the certificate supports (non-repudiation, digital signature, and key encipherment).
+    * **Subject Alternative Names** (`[alt_names]`): Lists all DNS names and IP addresses where your Vault service can be accessed. This includes the service name, fully qualified domain names (FQDNs) for different Kubernetes DNS contexts (namespace-scoped, cluster-scoped with `.svc`, and fully qualified with `.svc.cluster.local`), and the localhost IP address.
+
+    ```bash
+    cat > "${WORKDIR}/csr.conf" <<'EOF'
+    [req]
+    default_bits = 2048
+    prompt = no
+    encrypt_key = yes
+    default_md = sha256
+    distinguished_name = kubelet_serving
+    req_extensions = v3_req
+    [ kubelet_serving ]
+    O = system:nodes
+    CN = system:node:*.vault.svc.cluster.local
+    [ v3_req ]
+    basicConstraints = CA:FALSE
+    keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
+    extendedKeyUsage = serverAuth, clientAuth
+    subjectAltName = @alt_names
+    [alt_names]
+    DNS.1 =*.vault-internal
+    DNS.2 = *.vault-standby
+    DNS.3 =*.vault-internal.vault.svc.cluster.local
+    DNS.4 = *.vault-standby.vault.svc.cluster.local
+    DNS.5 =*.vault
+    DNS.6 = vault.vault.svc.cluster.local
+    IP.1 = 127.0.0.1
+    EOF
+    ```
+
+2. Generate the CSR. The following command creates the Certificate Signing Request file using your private key and the configuration file.
+
+    The `-subj` parameter specifies the distinguished name directly: the Common Name (CN) identifies your Vault service using the Kubernetes node naming convention (`system:node:${SERVICE}.${NAMESPACE}.svc`), and the Organization (O) field is set to `system:nodes`, which Kubernetes requires to recognize and sign the certificate. The command combines these subject fields with the certificate extensions defined in the configuration file to produce the complete CSR.
+
+    ```bash
+    openssl req -new -key $WORKDIR/vault.key \
+      -subj "/CN=system:node:${SERVICE}.${NAMESPACE}.svc;/O=system:nodes" \
+      -out $WORKDIR/server.csr -config $WORKDIR/csr.conf
+    ```
+
+### Issue the certificate
+
+To get your certificate signed by Kubernetes, you need to submit the CSR through the Kubernetes API. The CSR file you generated with OpenSSL must be wrapped in a Kubernetes CertificateSigningRequest resource.
+
+1. Create the CSR YAML file to send it to Kubernetes:
+
+    This YAML file creates a Kubernetes CertificateSigningRequest object that contains your CSR. The file embeds the base64-encoded CSR content and specifies:
+
+    * The signer name (`kubernetes.io/kubelet-serving`) that tells Kubernetes which CA should sign the certificate
+    * The groups field (`system:authenticated`) that identifies who can approve this CSR
+    * The certificate usages that define how the certificate can be used (digital signature, key encipherment, and server authentication)
+
+    ```bash
+    cat > $WORKDIR/csr.yaml <<EOF
+    apiVersion: certificates.k8s.io/v1
+    kind: CertificateSigningRequest
+    metadata:
+      name: ${CSR_NAME}
+    spec:
+      groups:
+      - system:authenticated
+      request: $(cat $WORKDIR/server.csr | base64 | tr -d '\n')
+      signerName: kubernetes.io/kubelet-serving
+      usages:
+      - digital signature
+      - key encipherment
+      - server auth
+    EOF
+    ```
+
+2. Create the CertificateSigningRequest (CSR) object:
+
+    ```bash
+    kubectl create -f ${WORKDIR}/csr.yaml
+    ```
+
+3. Approve the CSR in Kubernetes:
+
+    ```bash
+    kubectl certificate approve ${CSR_NAME}
+    ```
+
+4. Confirm the certificate was issued:
+
+    ```bash
+    kubectl get csr ${CSR_NAME}
+    ```
+
+    ??? example "Sample output"
+
+        ```{.text .no-copy}
+        NAME        AGE   SIGNERNAME                      REQUESTOR       REQUESTEDDURATION   CONDITION
+        vault-csr   16s   kubernetes.io/kubelet-serving   minikube-user   <none>              Approved,Issued
+        ```
+
+### Retrieve the certificates
+
+After Kubernetes approves and signs your CSR, you need to retrieve the signed certificate and the Kubernetes CA certificate. These certificates are required to configure TLS for your Vault server.
+
+1. Retrieve the signed certificate from the CertificateSigningRequest object. The certificate is base64-encoded in Kubernetes, so you decode it and save it to a file.
+
+    ```bash
+    kubectl get csr ${CSR_NAME} -o jsonpath='{.status.certificate}' | base64 -d > $WORKDIR/vault.crt
+    ```
+
+2. Retrieve Kubernetes CA certificate:
+
+    This command retrieves the Kubernetes cluster's Certificate Authority (CA) certificate from your `kubeconfig` file. The CA certificate is needed to verify that the signed certificate is valid and was issued by the Kubernetes CA. The command uses `kubectl config view` with flags to get the raw, flattened configuration and extract the CA certificate data, which is also base64-encoded.
+
+    ```bash
+    kubectl config view \
+      --raw \
+      --minify \
+      --flatten \
+      -o jsonpath='{.clusters[].cluster.certificate-authority-data}' \
+      | base64 -d > ${WORKDIR}/vault.ca
+    ```
+
+### Store certificates in Kubernetes secrets
+
+Create a TLS secret in Kubernetes to store the certificates and key:
+
+```bash
+kubectl create secret generic ${SECRET_NAME_VAULT} \
+  --namespace ${NAMESPACE} \
+  --from-file=vault.key=$WORKDIR/vault.key \
+  --from-file=vault.crt=$WORKDIR/vault.crt \
+  --from-file=vault.ca=$WORKDIR/vault.ca
+```
+

docs/assets/fragments/vault-install-tls.txt

@@ -0,0 +1,197 @@
+## Install Vault with TLS
+
+For this setup, we install Vault in Kubernetes using the [Helm 3 package manager :octicons-link-external-16:](https://helm.sh/) in High Availability (HA) mode with Raft storage backend and with TLS enabled.
+
+1. Add and update the Vault Helm repository:
+
+    ```bash
+    helm repo add hashicorp https://helm.releases.hashicorp.com
+    helm repo update
+    ```
+
+2. Install Vault with TLS enabled:
+
+    ```bash
+    helm upgrade --install ${SERVICE} hashicorp/vault \
+      --disable-openapi-validation \
+      --version ${VAULT_HELM_VERSION} \
+      --namespace ${NAMESPACE} \
+      --set "global.enabled=true" \
+      --set "global.tlsDisable=false" \
+      --set "global.platform=kubernetes" \
+      --set server.extraEnvironmentVars.VAULT_CACERT=/vault/userconfig/${SECRET_NAME_VAULT}/vault.ca \
+      --set "server.extraEnvironmentVars.VAULT_TLSCERT=/vault/userconfig/${SECRET_NAME_VAULT}/vault.crt" \
+      --set "server.extraEnvironmentVars.VAULT_TLSKEY=/vault/userconfig/${SECRET_NAME_VAULT}/vault.key" \
+      --set "server.volumes[0].name=userconfig-${SECRET_NAME_VAULT}" \
+      --set "server.volumes[0].secret.secretName=${SECRET_NAME_VAULT}" \
+      --set "server.volumes[0].secret.defaultMode=420" \
+      --set "server.volumeMounts[0].mountPath=/vault/userconfig/${SECRET_NAME_VAULT}" \
+      --set "server.volumeMounts[0].name=userconfig-${SECRET_NAME_VAULT}" \
+      --set "server.volumeMounts[0].readOnly=true" \
+      --set "server.ha.enabled=true" \
+      --set "server.ha.replicas=3" \
+      --set "server.ha.raft.enabled=true" \
+      --set "server.ha.raft.setNodeId=true" \
+      --set-string "server.ha.raft.config=cluster_name = \"vault-integrated-storage\"
+    ui = true
+    listener \"tcp\" {
+      tls_disable = 0
+      address = \"[::]:8200\"
+      cluster_address = \"[::]:8201\"
+      tls_cert_file = \"/vault/userconfig/${SECRET_NAME_VAULT}/vault.crt\"
+      tls_key_file  = \"/vault/userconfig/${SECRET_NAME_VAULT}/vault.key\"
+      tls_client_ca_file = \"/vault/userconfig/${SECRET_NAME_VAULT}/vault.ca\"
+    }
+    storage \"raft\" {
+      path = \"/vault/data\"
+    }
+    disable_mlock = true
+    service_registration \"kubernetes\" {}"
+    ```
+
+    This command does the following:
+
+    * Installs HashiCorp Vault in High Availability (HA) mode with secure TLS enabled in your Kubernetes cluster.
+    * Configures Vault pods to use certificates from a Kubernetes Secret via volume mounts for secure HTTPS communication between Vault and clients.
+    * Sets up Raft as the backend storage with three replicas for fault tolerance, and configures the Vault TCP listener to enforce TLS with your specified certificate files.
+
+
+    ??? example "Sample output"
+
+        ```{.text .no-copy}
+        NAME: vault
+        LAST DEPLOYED: Wed Aug 20 12:55:38 2025
+        NAMESPACE: vault
+        STATUS: deployed
+        REVISION: 1
+        NOTES:
+        Thank you for installing HashiCorp Vault!
+
+        Now that you have deployed Vault, you should look over the docs on using
+        Vault with Kubernetes available here:
+
+        https://developer.hashicorp.com/vault/docs
+        ```
+
+3. Retrieve the Pod name where Vault is running:
+
+    ```bash
+    kubectl -n $NAMESPACE get pod -l app.kubernetes.io/name=${SERVICE} -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}'
+    ```
+
+    ??? example "Sample output"
+
+        ```{.text .no-copy}
+        vault-0
+        vault-1
+        vault-2
+        ```
+
+## Initialize and unseal Vault
+
+1. After Vault is installed, you need to initialize it. Run the following command to initialize the first pod:
+
+    ```bash
+    kubectl exec -it pod/vault-0 -n $NAMESPACE -- vault operator init -key-shares=1 -key-threshold=1 -format=json > ${WORKDIR}/vault-init
+    ```
+
+    The command does the following:
+
+    * Connects to the Vault Pod
+    * Initializes Vault server with TLS enabled
+    * Creates 1 unseal key share which is required to unseal the server
+    * Outputs the init response to a local file. The file includes unseal keys and root token.
+
+2. Vault is started in a sealed state. In this state Vault can access the storage but it cannot decrypt data. In order to use Vault, you need to unseal it.
+
+    Retrieve the unseal key from the file:
+
+    ```bash
+    unsealKey=$(jq -r ".unseal_keys_b64[]" < ${WORKDIR}/vault-init)
+    ```
+
+    Now, unseal Vault. Run the following command:
+
+    ```bash
+    kubectl exec -it pod/vault-0 -n $NAMESPACE -- vault operator unseal "$unsealKey"
+    ```
+
+    ??? example "Sample output"
+
+        ```{.text .no-copy}
+        Key                     Value
+        ---                     -----
+        Seal Type               shamir
+        Initialized             true
+        Sealed                  false
+        Total Shares            1
+        Threshold               1
+        Version                 1.19.0
+        Build Date              2025-03-04T12:36:40Z
+        Storage Type            raft
+        Cluster Name            vault-integrated-storage
+        Cluster ID              ed275c91-e227-681b-5aaa-f7a9fc19e37e
+        Removed From Cluster    false
+        HA Enabled              true
+        HA Cluster              <https://vault-0.vault-internal:8201>
+        HA Mode                 active
+        Active Since            2025-12-15T13:36:42.542059059Z
+        Raft Committed Index    37
+        Raft Applied Index      37
+        ```
+
+3. Add the remaining Pods to the Vault cluster. If you have another secret name, replace the `vault-secret` with your value in the following for loop:
+
+    ```bash
+    for POD in vault-1 vault-2; do
+      kubectl -n "$NAMESPACE" exec $POD -- sh -c '
+        vault operator raft join -address=https://${HOSTNAME}.vault-internal:8200 \
+          -leader-ca-cert="$(cat /vault/userconfig/vault-secret/vault.ca)" \
+          -leader-client-cert="$(cat /vault/userconfig/vault-secret/vault.crt)" \
+          -leader-client-key="$(cat /vault/userconfig/vault-secret/vault.key)" \
+          https://vault-0.vault-internal:8200;
+      '
+    done
+    ```
+
+    The command connects to each Vault Pod (`vault-1` and `vault-2`) and issues the `vault operator raft join` command, which:
+    * Joins the Pods to the Vault Raft cluster, enabling HA mode.
+    * Uses the necessary TLS certificates to securely connect to the cluster leader (`vault-0`).
+    * Ensures all nodes participate in the Raft consensus and share storage responsibilities.
+
+    ??? example "Sample output"
+
+        ```{.text .no-copy}
+        Key       Value
+        ---       -----
+        Joined    true
+        ```
+
+4. Unseal the remaining Pods. Use this for loop:
+
+    ```bash
+    for POD in vault-1 vault-2; do
+        kubectl -n "$NAMESPACE" exec $POD -- sh -c "
+            vault operator unseal \"$unsealKey\"
+        "
+    done
+    ```
+
+    ??? example "Expected output"
+
+        ```{.text .no-copy}
+        Key                     Value
+        ---                     -----
+        Seal Type               shamir
+        Initialized             true
+        Sealed                  false
+        Total Shares            1
+        Threshold               1
+        Unseal Progress         1/1
+        Unseal Nonce            n/a
+        Version                 1.19.0
+        Build Date              2025-03-04T12:36:40Z
+        Storage Type            raft
+        Removed From Cluster    false
+        HA Enabled              true
+        ```