Skip to content

Add Prometheus Metric QOL improvements to local env #198

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
liam-lowe merged 2 commits into from
Feb 24, 2026
Merged

Add Prometheus Metric QOL improvements to local env #198

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bd2ab7a
Add Prometheus Metric QOL improvements to local env
liam-lowe Feb 20, 2026
81dc755
Format awk nonsense into something sensible
liam-lowe Feb 20, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 17 additions & 2 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,20 +152,35 @@ DOCKER_COMPOSE = docker compose --file $(DOCKER_COMPOSE_FILE) --env-fil

PROXY_LEFT_CONFIG_TMPL = develop/docker-compose/develop.proxy-left.tmpl.yaml
PROXY_RIGHT_CONFIG_TMPL = develop/docker-compose/develop.proxy-right.tmpl.yaml
PROMETHEUS_CONFIG_TMPL = develop/docker-compose/develop.prometheus.tmpl.yaml
PROXY_LEFT_CONFIG = develop/docker-compose/tmp/develop.proxy-left.yaml
PROXY_RIGHT_CONFIG = develop/docker-compose/tmp/develop.proxy-right.yaml
PROMETHEUS_CONFIG = develop/docker-compose/tmp/develop.prometheus.yaml

.PHONY: generate-configs
generate-configs:
mkdir -p develop/docker-compose/tmp
set -a && . $(DEVELOP_ENV_FILE) && set +a && \
envsubst < $(PROXY_LEFT_CONFIG_TMPL) > $(PROXY_LEFT_CONFIG) && \
envsubst < $(PROXY_RIGHT_CONFIG_TMPL) > $(PROXY_RIGHT_CONFIG)
envsubst < $(PROXY_LEFT_CONFIG_TMPL) > $(PROXY_LEFT_CONFIG) && \
envsubst < $(PROXY_RIGHT_CONFIG_TMPL) > $(PROXY_RIGHT_CONFIG) && \
envsubst < $(PROMETHEUS_CONFIG_TMPL) > $(PROMETHEUS_CONFIG)

.PHONY: show-dependencies-ports
show-dependencies-ports:
Copy link
Contributor Author

@liam-lowe liam-lowe Feb 20, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Opted to derive from env file as source of truth. Result is a more readable, and links are cmd+clickable:

Exposed localhost ports:
  temporal-left                      http://localhost:4000
  temporal-right                     http://localhost:5000
  proxy-left                         http://localhost:4001
  proxy-right                        http://localhost:5001
  proxy-left-metrics                 http://localhost:4091
  proxy-right-metrics                http://localhost:5091
  prometheus                         http://localhost:9090
  temporal-left-ui                   http://localhost:4233
  temporal-right-ui                  http://localhost:5233

The "more-correct" approach would be to derive from running containers directly - something like docker compose ps --format "table {{.Service}}\t{{.Ports}}"

But the result is actually less-usable and descriptive imo.

SERVICE          PORTS
prometheus       0.0.0.0:9090->9090/tcp, [::]:9090->9090/tcp
proxy-left       0.0.0.0:4001->6233/tcp, [::]:4001->6233/tcp, 0.0.0.0:4091->9091/tcp, [::]:4091->9091/tcp
proxy-right      0.0.0.0:5001->6333/tcp, [::]:5001->6333/tcp, 0.0.0.0:5091->9091/tcp, [::]:5091->9091/tcp
temporal-left    0.0.0.0:4000->7233/tcp, [::]:4000->7233/tcp, 0.0.0.0:4233->8233/tcp, [::]:4233->8233/tcp
temporal-right   0.0.0.0:5000->7233/tcp, [::]:5000->7233/tcp, 0.0.0.0:5233->8233/tcp, [::]:5233->8233/tcp

The downside of implemented approach is that the make implementation isn't particularly easy to read, and isn't deriving from more reliable source of truth.

@echo 'Exposed localhost ports:'
@grep '_EXTERNAL_PORT=' $(DEVELOP_ENV_FILE) | \
awk -F= '{ \
name = $$1; \
gsub(/_EXTERNAL_PORT$$/, "", name); \
gsub(/_/, "-", name); \
printf " %-34s http://localhost:%s\n", tolower(name), $$2; \
}'

.PHONY: start-dependencies
start-dependencies: generate-configs
$(DOCKER_COMPOSE) up --detach --build --wait --wait-timeout 120
@echo >&2 'Dependencies ready!'
@$(MAKE) --no-print-directory show-dependencies-ports

.PHONY: stop-dependencies
stop-dependencies:
Expand Down
29 changes: 25 additions & 4 deletions develop/docker-compose/develop.docker-compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,15 @@ services:
- "${TEMPORAL_INTERNAL_PORT}"
- --ip
- "0.0.0.0"
- --headless
- --ui-port
- "${TEMPORAL_UI_INTERNAL_PORT}"
- --namespace
- default
- --log-level
- warn
ports:
- "${TEMPORAL_LEFT_EXTERNAL_PORT}:${TEMPORAL_INTERNAL_PORT}"
- "${TEMPORAL_LEFT_UI_EXTERNAL_PORT}:${TEMPORAL_UI_INTERNAL_PORT}"
networks:
- develop
healthcheck:
Expand All @@ -43,13 +45,15 @@ services:
- "${TEMPORAL_INTERNAL_PORT}"
- --ip
- "0.0.0.0"
- --headless
- --ui-port
- "${TEMPORAL_UI_INTERNAL_PORT}"
- --namespace
- default
- --log-level
- warn
ports:
- "${TEMPORAL_RIGHT_EXTERNAL_PORT}:${TEMPORAL_INTERNAL_PORT}"
- "${TEMPORAL_RIGHT_UI_EXTERNAL_PORT}:${TEMPORAL_UI_INTERNAL_PORT}"
networks:
- develop
healthcheck:
Expand All @@ -75,6 +79,7 @@ services:
volumes: *x-volumes
ports:
- "${PROXY_RIGHT_EXTERNAL_PORT}:${PROXY_RIGHT_INTERNAL_PORT}"
- "${PROXY_RIGHT_METRICS_EXTERNAL_PORT}:${PROXY_METRICS_INTERNAL_PORT}"
networks:
- develop
depends_on:
Expand All @@ -83,7 +88,7 @@ services:
healthcheck: # TODO: Replace with healthcheck endpoint.
test:
- CMD-SHELL
- "wget -q -O /dev/null -T 3 http://localhost:6060/debug/pprof/ || exit 1"
- "wget -q -O /dev/null -T 3 http://localhost:${PROXY_PPROF_INTERNAL_PORT}/debug/pprof/ || exit 1"
interval: 5s
timeout: 5s
retries: 30
Expand All @@ -98,6 +103,7 @@ services:
volumes: *x-volumes
ports:
- "${PROXY_LEFT_EXTERNAL_PORT}:${PROXY_LEFT_INTERNAL_PORT}"
- "${PROXY_LEFT_METRICS_EXTERNAL_PORT}:${PROXY_METRICS_INTERNAL_PORT}"
networks:
- develop
depends_on:
Expand All @@ -108,12 +114,27 @@ services:
healthcheck: # TODO: Replace with healthcheck endpoint.
test:
- CMD-SHELL
- "wget -q -O /dev/null -T 3 http://localhost:6060/debug/pprof/ || exit 1"
- "wget -q -O /dev/null -T 3 http://localhost:${PROXY_PPROF_INTERNAL_PORT}/debug/pprof/ || exit 1"
interval: 5s
timeout: 5s
retries: 30
start_period: 10s

prometheus:
image: prom/prometheus:latest
command:
- --config.file=/etc/develop/docker-compose/tmp/develop.prometheus.yaml
volumes: *x-volumes
ports:
- "${PROMETHEUS_EXTERNAL_PORT}:${PROMETHEUS_INTERNAL_PORT}"
networks:
- develop
depends_on:
proxy-left:
condition: service_healthy
proxy-right:
condition: service_healthy
Comment on lines +123 to +136
Copy link
Collaborator

@temporal-nick temporal-nick Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we want a prometheus server in here?

Copy link
Contributor Author

@liam-lowe liam-lowe Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Intent is to facilitate faster and more-reliable local testing, specifically for metrics - which is where I'm making changes next.

There are 3 approaches I see:

  • Grep via /metrics (either inside container or exposed localhost port)
  • Prometheus
  • Grafana

I think Prometheus is faster to validate locally than exposing each /metrics/ port and grepping logs, and lighter weight with less configuration than Grafana.

There could be use case for Grafana in the future, but at the moment, I don't believe we need it.


smoke-test:
image: temporalio/temporal:1.5.0
entrypoint: /bin/sh
Expand Down
9 changes: 9 additions & 0 deletions develop/docker-compose/develop.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,18 @@ TEMPORAL_LEFT_EXTERNAL_PORT=4000
TEMPORAL_RIGHT_EXTERNAL_PORT=5000
PROXY_LEFT_EXTERNAL_PORT=4001
PROXY_RIGHT_EXTERNAL_PORT=5001
PROXY_LEFT_METRICS_EXTERNAL_PORT=4091
Copy link
Collaborator

@hehaifengcn hehaifengcn Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i thought we want to use different name other than left/right? not need for this PR. just want to understand if we still plan to do so.

Copy link
Contributor Author

@liam-lowe liam-lowe Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. We will standardize - but outside scope of this PR.

PROXY_RIGHT_METRICS_EXTERNAL_PORT=5091
PROMETHEUS_EXTERNAL_PORT=9090
TEMPORAL_LEFT_UI_EXTERNAL_PORT=4233
TEMPORAL_RIGHT_UI_EXTERNAL_PORT=5233

# Internal Ports (exposed on containers in docker network)
TEMPORAL_INTERNAL_PORT=7233
TEMPORAL_UI_INTERNAL_PORT=8233
PROXY_LEFT_INTERNAL_PORT=6233
PROXY_RIGHT_INTERNAL_PORT=6333
PROXY_MUX_INTERNAL_PORT=6334
PROXY_PPROF_INTERNAL_PORT=6060
PROXY_METRICS_INTERNAL_PORT=9091
PROMETHEUS_INTERNAL_PORT=9090
11 changes: 11 additions & 0 deletions develop/docker-compose/develop.prometheus.tmpl.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
global:
scrape_interval: 15s

scrape_configs:
- job_name: proxy-left
static_configs:
- targets: ["proxy-left:${PROXY_METRICS_INTERNAL_PORT}"]

- job_name: proxy-right
static_configs:
- targets: ["proxy-right:${PROXY_METRICS_INTERNAL_PORT}"]
5 changes: 4 additions & 1 deletion develop/docker-compose/develop.proxy-left.tmpl.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,7 @@ clusterConnections:
muxAddressInfo:
address: "proxy-right:${PROXY_MUX_INTERNAL_PORT}"
profiling:
pprofAddress: "0.0.0.0:6060"
pprofAddress: "0.0.0.0:${PROXY_PPROF_INTERNAL_PORT}"
metrics:
prometheus:
listenAddress: "0.0.0.0:${PROXY_METRICS_INTERNAL_PORT}"
5 changes: 4 additions & 1 deletion develop/docker-compose/develop.proxy-right.tmpl.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,7 @@ clusterConnections:
muxAddressInfo:
address: "0.0.0.0:${PROXY_MUX_INTERNAL_PORT}"
profiling:
pprofAddress: "0.0.0.0:6060"
pprofAddress: "0.0.0.0:${PROXY_PPROF_INTERNAL_PORT}"
metrics:
prometheus:
listenAddress: "0.0.0.0:${PROXY_METRICS_INTERNAL_PORT}"