Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 51 additions & 0 deletions docs/deployment/docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
# Docker

## Run as a HTTP Service


1. Start service on port 8000

```bash
docker run -p 8000:8000 ghcr.io/google/garf:latest
```

2. Call it

```bash
curl -X 'POST' \
'http://0.0.0.0:8000/api/execute' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"source": "rest",
"title": "test",
"query": "SELECT id AS device_id, name AS device_name, data.color AS device_color FROM objects",
"context": {
"fetcher_parameters": {
"endpoint": "https://api.restful-api.dev"
}
}
}'
```

## Run as a gRPC Service


1. Start service on port 50051

```bash
docker run -p 50051:50051 ghcr.io/google/garf:latest \
python -m garf.executors.entrypoints.grpc_server
```



## Run as a CLI tool

```bash
docker run ghcr.io/google/garf:latest \
'SELECT id AS device_id, name AS device_name, data.color AS device_color FROM objects' \
--input console --output console --source rest \
--rest.endpoint=https://api.restful-api.dev \
--logger rich
```
86 changes: 86 additions & 0 deletions docs/deployment/google-cloud.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
`garf` can be easily deployed to Google Cloud as a Cloud Run service or as a part of
Cloud Workflows.

## Cloud Run

Before using `garf` in Google Cloud we need to submit an image to Cloud Build.

### Prerequisites

Ensure that service account has necessary permissions:

```bash
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
--member="serviceAccount:$(gcloud projects describe $GOOGLE_CLOUD_PROJECT \
--format='value(projectNumber)')@cloudbuild.gserviceaccount.com" \
--role="roles/run.admin"

gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
--member="serviceAccount:$(gcloud projects describe $GOOGLE_CLOUD_PROJECT \
--format='value(projectNumber)')@cloudbuild.gserviceaccount.com" \
--role="roles/iam.serviceAccountUser"
```


The following command builds the image and create a Cloud Run service `garf`.

```bash
gcloud builds submit
--config <(curl -s https://raw.githubusercontent.com/google/garf/refs/heads/main/gcp/cloudbuild.yaml) \
--substitutions=_EXTRA_LIBS="garf-youtube garf-google-ads garf-executors[bq]" .
```

!!!note
You can provide additional libraries via `_EXTRA_LIBS` substitutions.

For example to interact with Google Ads API you need to install `garf-google-ads` library.


## Cloud Workflows

If you have a Cloud Run service deployed you can use it a [Cloud Workflows](https://docs.cloud.google.com/workflows/docs).

### Prerequisites

1. Workflows API enabled

```bash
gcloud services enable workflows.googleapis.com
```

2. LogWriter role granted to default service account

```bash
PROJECT_ID=$(gcloud projects describe $GOOGLE_CLOUD_PROJECT \ --format='value(projectNumber)')
gcloud projects add-iam-policy-binding $GOOGLE_CLOUD_PROJECT \
--member="serviceAccount:$PROJECT_ID-compute@developer.gserviceaccount.com" \
--role=roles/logging.logWriter
```

### Prepare

!!!important
You can convert your existing [workflow file](../usage/workflows.md) into
Cloud Workflow yaml file with a helper command:

```bash
grf worflow deploy -f path/to/workflow.yaml -o path/to/google-cloud-workflow.yaml
```

Once you have a `google-cloud-workflow.yaml` file you can deploy Cloud Workflow
based on it.


### Deploy

```bash
GARF_CLOUD_RUN_URL=$(gcloud run services describe garf --region=us-central1 --format='value(status.url)')

gcloud workflows deploy WORKFLOW_NAME \
--source=path/to/google-cloud-workflow.yaml \
--set-env-vars GARF_ENDPOINT=$GARF_CLOUD_RUN_URL
```


After workflow is deployed you can visit Cloud Workflow page in Google Cloud Console
and trigger or schedule it.
77 changes: 77 additions & 0 deletions docs/deployment/kubernetes.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
# Kubernetes


## Deployment and service

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: garf
labels:
app: garf
spec:
replicas: 1
selector:
matchLabels:
app: garf
template:
metadata:
labels:
app: garf
spec:
containers:
name: garf
image: ghcr.io/google/garf:latest
ports:
- containerPort: 8000

---
apiVersion: v1
kind: Service
metadata:
name: garf
spec:
selector:
app: garf
type: NodePort
ports:
- protocol: TCP
port: 30003
targetPort: 8000
```


## Cron job


```yaml

apiVersion: batch/v1
kind: CronJob
metadata:
name: garf-rest
spec:
schedule: "0 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: garf
image: ghcr.io/google/garf:latest
imagePullPolicy: IfNotPresent
command: ["garf"]
args:
- "'SELECT id AS device_id, name AS device_name, data.color AS device_color FROM objects'"
- "--input"
- "console"
- "--source"
- "rest"
- "--source.endpoint=https://api.restful-api.dev"
- "--output"
- "csv"
- "--logger"
- "local"
restartPolicy: OnFailure
```
40 changes: 40 additions & 0 deletions gcp/cloudbuild.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
steps:
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-c'
- |
docker build \
--build-arg="_EXTRA_LIBS=${_EXTRA_LIBS}" \
-t "gcr.io/$PROJECT_ID/${_IMAGE}:latest" \
-f - . <<EOF
FROM ${_BASE_IMAGE}
ARG _EXTRA_LIBS=""
RUN if [ -n "$_EXTRA_LIBS" ]; then \
uv pip install --no-cache-dir $_EXTRA_LIBS; \
fi
EOF

- name: 'gcr.io/cloud-builders/docker'
args: ['push', 'gcr.io/$PROJECT_ID/${_IMAGE}:latest']

- name: 'gcr.io/google.com/cloudsdktool/cloud-sdk'
entrypoint: gcloud
args: [
'run', 'deploy', '${_SERVICE_NAME}',
'--image', 'gcr.io/$PROJECT_ID/${_IMAGE}:latest',
'--region', '${_REGION}',
'--platform', 'managed', '--port', '8000',
'--set-env-vars',
'OTEL_EXPORTER_OTLP_ENDPOINT=telemetry.googleapis.com,OTEL_SERVICE_NAME=garf',
]

substitutions:
_EXTRA_LIBS: "garf-executors"
_BASE_IMAGE: "ghcr.io/google/garf:latest"
_IMAGE: "garf"
_SERVICE_NAME: "garf"
_REGION: "us-central1"

images:
- 'gcr.io/$PROJECT_ID/${_IMAGE}:latest'
2 changes: 1 addition & 1 deletion libs/executors/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -9,4 +9,4 @@ RUN if [ -n "$EXTRA_LIBS" ]; then \
uv pip install --no-cache-dir $EXTRA_LIBS; \
fi

ENTRYPOINT ["uvicorn", "garf.executors.entrypoints.server:app", "--host", "0.0.0.0"]
CMD ["uvicorn", "garf.executors.entrypoints.server:app", "--host", "0.0.0.0"]
4 changes: 4 additions & 0 deletions mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,10 @@ nav:
- Create API client: development/create-api-client.md
- Create API response parser: development/create-api-response-parser.md
- Create a library: development/create-garf-library.md
- Deployment:
- Docker: deployment/docker.md
- Kubernetes: deployment/kubernetes.md
- Google Cloud: deployment/google-cloud.md
- Fetchers:
- fetchers/overview.md
- REST: fetchers/rest.md
Expand Down