-
Notifications
You must be signed in to change notification settings - Fork 0
GraphQL Gateway for Milo APIServer #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
e52b928
1c89b01
74a8f36
56082f5
aa10d42
5bb9ebe
e565e32
103ce8b
712229a
6d44dac
969c44c
4961812
9f0c804
82f2d3b
8ee7c42
c0c9de8
9480d7b
6d6086c
00ff661
70ff4cf
dc498d8
fdad017
8f4f663
712f235
2739ef2
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,3 @@ | ||
| ./node_modules/ | ||
| .env | ||
| supergraph.graphql | ||
| ./dist |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| name: Build and Publish Docker Image | ||
|
|
||
| on: | ||
| push: | ||
| release: | ||
| types: ['published'] | ||
|
|
||
| jobs: | ||
| publish-container-image: | ||
| permissions: | ||
| id-token: write | ||
| contents: read | ||
| packages: write | ||
| attestations: write | ||
| uses: datum-cloud/actions/.github/workflows/publish-docker.yaml@v1.7.2 | ||
| with: | ||
| image-name: graphql-gateway | ||
| secrets: inherit | ||
|
|
||
| publish-kustomize-bundles: | ||
| permissions: | ||
| id-token: write | ||
| contents: read | ||
| packages: write | ||
| uses: datum-cloud/actions/.github/workflows/publish-kustomize-bundle.yaml@v1.7.2 | ||
| with: | ||
| bundle-name: ghcr.io/datum-cloud/graphql-gateway-kustomize | ||
| bundle-path: config | ||
| image-overlays: config/base | ||
| image-name: ghcr.io/datum-cloud/graphql-gateway | ||
| secrets: inherit |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,3 @@ | ||
| node_modules | ||
| .env | ||
| supergraph.graphql | ||
| dist/ |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| node_modules/ | ||
| dist/ | ||
| src/mesh/gen/ | ||
| *.graphql | ||
|
|
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| { | ||
| "semi": false, | ||
| "singleQuote": true, | ||
| "tabWidth": 2, | ||
| "trailingComma": "es5", | ||
| "printWidth": 100 | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,34 +1,42 @@ | ||
| FROM node:20-alpine AS builder | ||
| # --- Stage 1: Build --- | ||
| FROM node:22-slim AS builder | ||
|
|
||
| WORKDIR /app | ||
|
|
||
| ARG DATUM_TOKEN | ||
| ENV DATUM_TOKEN=$DATUM_TOKEN | ||
|
|
||
| ARG DATUM_BASE_URL | ||
| ENV DATUM_BASE_URL=$DATUM_BASE_URL | ||
|
|
||
| # Install dependencies (needed for mesh compose) | ||
| # Copy package files | ||
| COPY package.json package-lock.json ./ | ||
|
|
||
| # Install all dependencies (including devDependencies for build) | ||
| RUN npm ci | ||
|
|
||
| # Copy composition config and API descriptions | ||
| COPY mesh.config.ts ./ | ||
| COPY config/apis.yaml ./config/apis.yaml | ||
| # Copy source code and config | ||
| COPY tsconfig.json ./ | ||
| COPY src/ ./src/ | ||
|
|
||
| # Build the application | ||
| RUN npm run build | ||
|
|
||
| # --- Stage 2: Runtime --- | ||
| FROM node:22-slim | ||
|
|
||
| # Generate the supergraph at build time | ||
| RUN npx mesh-compose -o supergraph.graphql | ||
| WORKDIR /app | ||
|
|
||
| # Set the environment to production | ||
| ENV NODE_ENV=production | ||
|
|
||
| # Copy package files | ||
| COPY package.json package-lock.json ./ | ||
|
|
||
| # --- Runtime image: Hive Gateway --- | ||
| FROM ghcr.io/graphql-hive/gateway:2.1.19 | ||
| RUN npm i @graphql-mesh/transport-rest | ||
| # Install only production dependencies | ||
| RUN npm ci --omit=dev | ||
|
|
||
| WORKDIR /gateway | ||
| # Copy compiled gateway and shared code from builder | ||
| COPY --from=builder /app/dist/gateway ./dist/gateway | ||
| COPY --from=builder /app/dist/shared ./dist/shared | ||
|
|
||
| # Copy generated supergraph and gateway configuration | ||
| COPY --from=builder /app/supergraph.graphql ./supergraph.graphql | ||
| COPY gateway.config.ts ./gateway.config.ts | ||
| # Copy runtime configuration | ||
| COPY config/ ./config/ | ||
|
|
||
| EXPOSE 4000 | ||
|
|
||
| CMD ["supergraph", "--hive-router-runtime"] | ||
| CMD ["node", "dist/gateway/index.js"] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,103 +1,86 @@ | ||
| ## GraphQL Gateway for Milo APIServer | ||
|
|
||
| This project provides a **GraphQL gateway** that sits in front of Milo APIServer REST/OpenAPI services and exposes a unified GraphQL API. It uses **Hive Mesh** at build time to compose a supergraph from OpenAPI specs, and **Hive Gateway** to serve the federated GraphQL endpoint. | ||
| This project provides a **GraphQL gateway** that sits in front of Milo APIServer REST/OpenAPI services and exposes a unified GraphQL API. It uses **Hive Gateway** with dynamic supergraph composition from OpenAPI specs at runtime. | ||
|
|
||
| > **Status**: This gateway is in an **initial, non‑production stage**. It is intended **only for local testing by the Datum team** and **is not production ready**. | ||
|
|
||
| ### What Hive Mesh is | ||
| ### Architecture | ||
|
|
||
| - **Hive Mesh**: a composition/mesh tool that can read multiple upstream sources (OpenAPI, REST, GraphQL, etc.) and generate a single federated GraphQL schema (a **supergraph**). | ||
| - In this project, Hive Mesh: | ||
| - Reads Milo OpenAPI definitions listed in `config/apis.yaml`. | ||
| - Uses `mesh.config.ts` to map those APIs into subgraphs. | ||
| - Outputs `supergraph.graphql`, which Hive Gateway then runs. | ||
| - **Dynamic Supergraph Composition**: The gateway dynamically fetches OpenAPI specs from the Milo APIServer and composes a unified GraphQL supergraph at runtime. | ||
| - **mTLS Authentication**: Uses client certificates (mTLS) to authenticate with the Kubernetes API server. | ||
| - **Polling**: The supergraph is recomposed periodically based on the `POLLING_INTERVAL` environment variable. | ||
|
|
||
| ### What Hive Gateway is | ||
|
|
||
| - **Hive Gateway**: a production-ready GraphQL gateway/router from the GraphQL Hive ecosystem. | ||
| - It: | ||
| - Loads the `supergraph.graphql` produced by Hive Mesh. | ||
| - Dynamically composes a supergraph from OpenAPI specs fetched from Milo APIServer. | ||
| - Executes incoming GraphQL operations by delegating to the underlying Milo APIs. | ||
| - Handles concerns like header propagation, TLS, caching, and observability. | ||
|
|
||
| ### Why `--hive-router-runtime` | ||
| ### Environment Variables | ||
|
|
||
| - **`--hive-router-runtime`** tells Hive Gateway to use the **Hive Router runtime** as its execution engine. | ||
| - In practice this: | ||
| - Enables the newer router-style unified graph execution path. | ||
| - Improves compatibility with Mesh-based supergraphs and transport plugins (such as HTTP/REST). | ||
| - Keeps the gateway aligned with the same runtime used by the standalone Hive Router. | ||
| - Increases execution speed | ||
| | Variable | Description | Default | | ||
| | --------------------- | -------------------------------------------------- | ----------------- | | ||
| | `PORT` | Port the gateway listens on | `4000` | | ||
| | `KUBECONFIG` | Path to kubeconfig file for K8s authentication | Required | | ||
| | `POLLING_INTERVAL` | Interval (ms) between supergraph recomposition | `120000` (2 min) | | ||
| | `LOGGING` | Log level (`debug`, `info`, `warn`, `error`) | `info` | | ||
| | `NODE_EXTRA_CA_CERTS` | Path to CA certificate for trusting K8s API server | Required for mTLS | | ||
|
|
||
| ### Project scripts | ||
| ### Project Scripts | ||
|
|
||
| Defined in `package.json`: | ||
|
|
||
| - **`npm run supergraph:compose`** | ||
| - Uses Hive Mesh (`mesh-compose`) and `mesh.config.ts` to generate `supergraph.graphql` locally. | ||
| - Reads API groups/versions from `config/apis.yaml` and environment such as `DATUM_TOKEN` and `DATUM_BASE_URL`. | ||
| - **`npm run build`** | ||
| - Compiles TypeScript source code to JavaScript. | ||
|
|
||
| - **`npm run dev`** | ||
| - Runs Hive Gateway directly against `supergraph.graphql` in the local working directory. | ||
| - Useful for quick local development when you already composed the supergraph. | ||
| - **`npm run dev`** | ||
| - Runs the gateway in development mode with hot reloading. | ||
|
|
||
| - **`npm run start:gateway`** | ||
| - Builds a Docker image (`graphql-gateway`) using the provided `Dockerfile`: | ||
| - Passes `DATUM_TOKEN` and `DATUM_BASE_URL` as build arguments for Mesh composition. | ||
| - Runs `mesh-compose` in the build stage to bake `supergraph.graphql` into the image. | ||
| - Starts the container on port `4000` and removes the image when the container exits. | ||
| - Intended as a simple “build-and-run” entrypoint for local or ad‑hoc environments. | ||
| - **`npm run start`** | ||
| - Runs the compiled gateway from `dist/`. | ||
|
|
||
| ### Querying Milo through the gateway | ||
| - **`npm run lint`** | ||
| - Runs ESLint on the codebase. | ||
|
|
||
| - **Using Postman with `supergraph.graphql`**: | ||
| - After running `npm run supergraph:compose`, you will have a local `supergraph.graphql` file. | ||
| - In Postman, create a new GraphQL request and **import** or **paste** the contents of `supergraph.graphql` as the schema. | ||
| - Point the request URL to your running gateway (for example `http://127.0.0.1:4000/graphql`) and you can start querying Milo through the GraphQL gateway. | ||
| ### Local Testing | ||
|
|
||
| - **Using the built‑in GraphQL UI**: | ||
| - When you run the gateway locally (via `npm run dev` or `npm run start:gateway`), it exposes a GraphQL endpoint at `http://127.0.0.1:4000/graphql`. | ||
| - Open `http://127.0.0.1:4000/graphql` in your browser to use the UI for exploring the schema and running queries against Milo. | ||
| A comprehensive local testing script is provided at `scripts/local-test.sh`. This script: | ||
|
|
||
| ### Basic usage | ||
| 1. Verifies the correct kubectl context is active | ||
| 2. Generates client certificates using cert-manager | ||
| 3. Creates a local kubeconfig with mTLS credentials | ||
| 4. Sets up port-forwarding to the Milo APIServer | ||
| 5. Runs the gateway locally | ||
|
|
||
| #### Datum access token (`DATUM_TOKEN`) | ||
| **Prerequisites**: | ||
|
|
||
| `DATUM_TOKEN` is a **Datum access token**. The easiest way to obtain it is with the `datumctl` CLI (see the official [datumctl docs](https://www.datum.net/docs/quickstart/datumctl/)). | ||
| - `kubectl` configured with access to the staging cluster | ||
| - `cert-manager` installed in the cluster | ||
| - Entry in `/etc/hosts`: `127.0.0.1 milo-apiserver` | ||
|
|
||
| - **Get a production token**: | ||
| **Usage**: | ||
|
|
||
| ```bash | ||
| datumctl auth get-token | ||
| ``` | ||
| ```bash | ||
| ./scripts/local-test.sh | ||
| ``` | ||
|
|
||
| - **Get a staging token**: | ||
| ### Kubernetes Deployment | ||
|
|
||
| ```bash | ||
| datumctl auth login --hostname auth.staging.env.datum.net | ||
| datumctl auth get-token | ||
| ``` | ||
| The gateway is deployed to Kubernetes using the manifests in `config/base/`. Key components: | ||
|
|
||
| Use the resulting token value as the `TOKEN` environment variable in the commands below. | ||
| - **deployment.yaml**: Gateway deployment with mTLS volume mounts | ||
| - **service.yaml**: ClusterIP service exposing port 4000 | ||
| - **http-route.yaml**: HTTPRoute for ingress configuration | ||
| - **milo-control-plane-kubeconfig.yaml**: ConfigMap with kubeconfig template | ||
|
|
||
| - **Compose the supergraph locally**: | ||
| ### Querying Milo through the Gateway | ||
|
|
||
| ```bash | ||
| export DATUM_TOKEN=your-token | ||
| export DATUM_BASE_URL=https://api.staging.env.datum.net | ||
| npm run supergraph:compose | ||
| ``` | ||
| - **Using the built‑in GraphQL UI**: | ||
| - When running the gateway, it exposes a GraphQL endpoint at `http://127.0.0.1:4000/graphql`. | ||
| - Open this URL in your browser to use the UI for exploring the schema and running queries against Milo. | ||
|
|
||
| - **Run the gateway locally (no Docker)**: | ||
|
|
||
| ```bash | ||
| npm run dev | ||
| ``` | ||
|
|
||
| - **Build and run via Docker**: | ||
|
|
||
| ```bash | ||
| export DATUM_TOKEN=your-token | ||
| export DATUM_BASE_URL=https://api.staging.env.datum.net | ||
| npm run start:gateway | ||
| ``` | ||
| - **Health Endpoints**: | ||
| - `/healthcheck` - Liveness probe | ||
| - `/readiness` - Readiness probe (checks if K8s auth is initialized) | ||
This file was deleted.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| apiVersion: apps/v1 | ||
| kind: Deployment | ||
| metadata: | ||
| labels: | ||
| app.kubernetes.io/component: backend | ||
| app.kubernetes.io/instance: graphql-gateway | ||
| app.kubernetes.io/name: graphql-gateway | ||
| app.kubernetes.io/version: 1.0.0 | ||
| name: graphql-gateway | ||
| spec: | ||
| progressDeadlineSeconds: 600 | ||
| replicas: 2 | ||
| revisionHistoryLimit: 5 | ||
| selector: | ||
| matchLabels: | ||
| app.kubernetes.io/component: backend | ||
| app.kubernetes.io/instance: graphql-gateway | ||
| app.kubernetes.io/name: graphql-gateway | ||
| strategy: | ||
| rollingUpdate: | ||
| maxSurge: 1 | ||
| maxUnavailable: 0 | ||
| type: RollingUpdate | ||
| template: | ||
| metadata: | ||
| labels: | ||
| app.kubernetes.io/component: backend | ||
| app.kubernetes.io/instance: graphql-gateway | ||
| app.kubernetes.io/name: graphql-gateway | ||
| spec: | ||
| containers: | ||
| - name: graphql-gateway | ||
| image: ghcr.io/datum-cloud/graphql-gateway:latest | ||
| imagePullPolicy: IfNotPresent | ||
| ports: | ||
| - containerPort: 4000 | ||
| name: http | ||
| protocol: TCP | ||
| env: | ||
| - name: LOGGING | ||
| value: debug | ||
| - name: KUBECONFIG | ||
| value: /etc/milo-control-plane/config/kubeconfig | ||
| volumeMounts: | ||
| - name: milo-control-plane-kubeconfig | ||
| mountPath: /etc/milo-control-plane/config | ||
| readOnly: true | ||
| - name: milo-control-plane-certs | ||
| mountPath: /etc/kubernetes/pki/client | ||
| readOnly: true | ||
| - name: trust-bundle | ||
| mountPath: /etc/kubernetes/pki/trust | ||
| readOnly: true | ||
| resources: | ||
| limits: | ||
| cpu: 1 | ||
| memory: 1Gi | ||
| livenessProbe: | ||
| failureThreshold: 3 | ||
| initialDelaySeconds: 5 | ||
| periodSeconds: 5 | ||
| timeoutSeconds: 10 | ||
| httpGet: | ||
| path: /healthcheck | ||
| port: 4000 | ||
| scheme: HTTP | ||
| readinessProbe: | ||
| failureThreshold: 3 | ||
| initialDelaySeconds: 5 | ||
| periodSeconds: 5 | ||
| timeoutSeconds: 10 | ||
| httpGet: | ||
| path: /readiness | ||
| port: 4000 | ||
| scheme: HTTP | ||
| volumes: | ||
| - name: milo-control-plane-kubeconfig | ||
| configMap: | ||
| name: milo-control-plane-kubeconfig | ||
| items: | ||
| - key: kubeconfig | ||
| path: kubeconfig | ||
| - name: milo-control-plane-certs | ||
| csi: | ||
| driver: csi.cert-manager.io | ||
| readOnly: true | ||
| volumeAttributes: | ||
| csi.cert-manager.io/issuer-name: datum-control-plane | ||
| csi.cert-manager.io/issuer-kind: ClusterIssuer | ||
| csi.cert-manager.io/common-name: graphql-gateway | ||
| csi.cert-manager.io/fs-group: '65534' | ||
| csi.cert-manager.io/key-usages: client auth | ||
| - name: trust-bundle | ||
| configMap: | ||
| name: datum-control-plane-trust-bundle | ||
| items: | ||
| - key: ca.crt | ||
| path: ca.crt | ||
|
Comment on lines
+83
to
+98
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd normally recommend leaving the environment specific details like the cluster issuer name / trust bundle name out of the base repo and apply those through overrides in the infra repo. Don't have to fix this as part of this PR, but something to consider for the future. |
||
| dnsPolicy: ClusterFirst | ||
| restartPolicy: Always | ||
| terminationGracePeriodSeconds: 30 | ||
Uh oh!
There was an error while loading. Please reload this page.