feat(otel): Add OpenTelemetry distributed tracing and ctutils logging integration #3

himaschal · 2025-12-08T14:01:16Z

Summary

This PR integrates the digicert/ctutils shared logging library to enable OpenTelemetry-compliant distributed tracing across the Trillian log server and signer components.

Changes

config/config.go: New centralized logging configuration with InitLogging()
cmd/trillian_log_server/main.go: Initialize OTEL logging on startup
cmd/trillian_log_signer/main.go: Initialize OTEL logging on startup
cmd/internal/serverutil/main.go: Add chained gRPC interceptors for trace propagation
examples/deployment/docker/log_server/Dockerfile.unified: Docker build with private ctutils dependency
README.md: Add Observability section documenting OTEL configuration
go.mod/go.sum: Add ctutils v0.1.6 and OTEL dependencies

Configuration

Tracing is controlled via environment variables:

Variable	Description	Default
`OTEL_ENABLED`	Enable OpenTelemetry	`false`
`OTEL_EXPORTER`	Exporter type (`otlp` or `stdout`)	`stdout`
`OTEL_COLLECTOR_ENDPOINT`	OTLP collector URL	`localhost:4317`
`OTEL_SERVICE_NAME`	Service name for traces	service-specific
`OTEL_SAMPLE_RATIO`	Sampling ratio (0.0-1.0)	`1.0`

Related PRs

ctutils#1 - Shared logging library
trillian-ctlog#32 - Helm/CI configuration

Testing

Manual testing with local OTEL collector. Full integration testing planned.

Refs: RA-8279

… integration This change integrates the digicert/ctutils shared logging library to enable OpenTelemetry-compliant distributed tracing across the Trillian log server and signer components. Key changes: - Add config/config.go with InitLogging() for centralized OTEL configuration - Update log_server and log_signer main.go to call config.InitLogging() - Add chained gRPC interceptors for trace context propagation - Add Dockerfile.unified with SSH access for private ctutils dependency - Update go.mod/go.sum for ctutils v0.1.6 and OTEL dependencies The logging configuration is driven by environment variables: - OTEL_ENABLED: Enable/disable OpenTelemetry (default: false) - OTEL_EXPORTER: Exporter type ('otlp' or 'stdout') - OTEL_COLLECTOR_ENDPOINT: OTLP collector URL - OTEL_SERVICE_NAME: Service name for traces - OTEL_SAMPLE_RATIO: Sampling ratio (0.0-1.0) This enables end-to-end request tracing from CTFE through Trillian backends, allowing operators to correlate logs and traces across the CT infrastructure. Refs: RA-8279

chore(deps): update ctutils

Copilot

Pull request overview

This PR integrates OpenTelemetry distributed tracing into Trillian log server and signer components through the digicert/ctutils shared logging library, enabling end-to-end request tracing and structured logging.

Changes:

Added OTEL-compliant distributed tracing via ctutils dependency
Configured gRPC and HTTP middleware for trace propagation and logging
Added environment-variable based OTEL configuration

Reviewed changes

Copilot reviewed 16 out of 18 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
config/config.go	New centralized logging initialization with OTEL setup
cmd/trillian_log_server/main.go	Initialize logging on server startup
cmd/trillian_log_signer/main.go	Initialize logging on signer startup
cmd/internal/serverutil/main.go	Add gRPC/HTTP interceptors for trace propagation
go.mod/go.sum	Add ctutils v0.1.13-test and updated OTEL dependencies
examples/deployment/docker//Dockerfile	Docker build configuration for private ctutils dependency
.github/workflows/*.yaml	CI authentication for private ctutils repository
README.md	Documentation for OTEL configuration
experimental/batchmap/batchmap.shims.go	Auto-generated code formatting updates

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-01-13T16:14:06Z

config/config.go

+func InitLogging() {
+	// Initialize OpenTelemetry with config struct
+	logging.InitOpenTelemetry(logging.TelemetryConfigFromEnv())
+
+	// Example: select logger backend via config, env var, or flag
+	logCfg := logging.Config{Format: "json"}
+	adapter := adapters.NewLogrusAdapter(logCfg)
+	logging.SetLoggerAdapter(adapter)
+}


The new config package and InitLogging() function lack test coverage. Given that this is a critical initialization function that sets up OpenTelemetry and logging adapters for the entire application, it should have unit tests to verify proper initialization, error handling, and configuration loading from environment variables.

Copilot · 2026-01-13T16:14:06Z

cmd/internal/serverutil/main.go

+		http.Handle("/metrics", logging.Middleware(promhttp.Handler()))
+		http.HandleFunc("/healthz", logging.MiddlewareFunc(m.healthz))


The HTTP middleware wrapping (logging.Middleware and logging.MiddlewareFunc) adds observability to HTTP endpoints, but there's no error handling if these functions fail or return errors. Consider adding error checking or at least documenting the error handling behavior of these middleware functions.

Copilot · 2026-01-13T16:14:06Z

examples/deployment/docker/log_server/Dockerfile.unified

+# Use Go proxy for faster module downloads (can be overridden at build time)
+ENV GOPROXY=https://proxy.golang.org,direct
+
+# Always copy main repo and ctutils


The Dockerfile expects both trillian/. and ctutils/. directories to exist in the build context (lines 17-18), but this assumes a specific directory structure that may not be clear to users. Consider documenting this requirement in the README or adding a comment explaining the expected build context structure and how to set it up.

Suggested change

# Always copy main repo and ctutils

# Always copy main repo and ctutils.

# NOTE: The Docker build context must contain `trillian/` and `ctutils/` directories

# at its root. Typically this means running `docker build` from the repository root,

# e.g.: `docker build -f examples/deployment/docker/log_server/Dockerfile.unified .`

Copilot · 2026-01-13T16:14:07Z

examples/deployment/docker/log_signer/Dockerfile

+# Download dependencies first - this should be cacheable.
+COPY go.mod go.sum ./
+ARG GITHUB_TOKEN
+ENV GITHUB_TOKEN=${GITHUB_TOKEN}
+ENV GOPRIVATE=github.com/digicert/ctutils
+ENV GONOSUMDB=github.com/digicert/ctutils
+RUN git config --global url."https://${GITHUB_TOKEN}:x-oauth-basic@github.com/".insteadOf "https://github.com/"
+
 # Download dependencies first - this should be cacheable.
 COPY go.mod go.sum ./


Lines 8-9 and lines 16-17 are duplicated. The comment "Download dependencies first - this should be cacheable." and the COPY go.mod go.sum ./ command appear twice. The first occurrence (lines 8-9) should be removed since the authentication setup (lines 10-14) needs to happen before copying the go.mod/go.sum files for the authentication to be effective during go mod download.

Copilot · 2026-01-13T16:14:07Z

go.mod

 toolchain go1.23.4

 require (
+	// require github.com/digicert/ctutils v0.1.2 // commented out for localreplace: use local ctutils for development


The comment on line 8 is misleading. It says "commented out for localreplace: use local ctutils for development" but the actual require statement for ctutils is on line 54 and is not commented out. Either remove this comment or clarify the intended development workflow. Additionally, having a commented require statement can be confusing - consider using go.mod replace directives for local development instead.

Suggested change

// require github.com/digicert/ctutils v0.1.2 // commented out for localreplace: use local ctutils for development

// For local development of github.com/digicert/ctutils, use a go.mod replace directive instead of a commented-out require.

Copilot · 2026-01-13T16:14:08Z

config/config.go

+	logging.InitOpenTelemetry(logging.TelemetryConfigFromEnv())
+
+	// Example: select logger backend via config, env var, or flag
+	logCfg := logging.Config{Format: "json"}


The hardcoded logger configuration Format: "json" is not aligned with the environment-based configuration approach mentioned in the PR description. Consider making the log format configurable via environment variables (e.g., LOG_FORMAT) to maintain consistency with the OTEL configuration pattern and allow flexibility in different deployment environments.

Copilot · 2026-01-13T16:14:09Z

examples/deployment/docker/log_server/Dockerfile

+ARG GITHUB_TOKEN
+ENV GITHUB_TOKEN=${GITHUB_TOKEN}
+ENV GOPRIVATE=github.com/digicert/ctutils
+ENV GONOSUMDB=github.com/digicert/ctutils
+RUN git config --global url."https://${GITHUB_TOKEN}:x-oauth-basic@github.com/".insteadOf "https://github.com/"
+


The token is stored in an environment variable GITHUB_TOKEN and then used in git config. This approach exposes the token in the environment and in the git config. Consider using --mount=type=secret to handle the token more securely, especially since this is a private repository token that should be protected.

Copilot · 2026-01-13T16:14:09Z

examples/deployment/docker/log_signer/Dockerfile

+ARG GITHUB_TOKEN
+ENV GITHUB_TOKEN=${GITHUB_TOKEN}
+ENV GOPRIVATE=github.com/digicert/ctutils
+ENV GONOSUMDB=github.com/digicert/ctutils
+RUN git config --global url."https://${GITHUB_TOKEN}:x-oauth-basic@github.com/".insteadOf "https://github.com/"


The token is stored in an environment variable GITHUB_TOKEN and then used in git config. This approach exposes the token in the environment and in the git config. Consider using --mount=type=secret to handle the token more securely, especially since this is a private repository token that should be protected.

Copilot · 2026-01-13T16:14:09Z

cmd/trillian_log_server/main.go


+	// Set up logging adapter via config logic
+	// This can be extended to use env vars, flags, or config files
+	config.InitLogging()


The config.InitLogging() call has no error handling. If the initialization fails (e.g., due to invalid configuration, missing environment variables, or OTEL setup issues), the server will continue running with potentially incorrect logging/tracing configuration. Consider either: 1) Having InitLogging() return an error and handle it here, or 2) Documenting that InitLogging() will panic/exit on critical errors.

Suggested change

config.InitLogging()

if err := config.InitLogging(); err != nil {

klog.Exitf("Failed to initialize logging: %v", err)

}

Copilot · 2026-01-13T16:14:10Z

cmd/internal/serverutil/main.go

+	// Initialize OpenTelemetry for the Trillian server
+	//config.InitLogging()
+


The OpenTelemetry initialization is commented out. If this initialization is not needed here (because it's already called in the main functions), the commented code should be removed. If it is needed, it should be uncommented. Leaving commented code can cause confusion about whether this initialization is intentional or was forgotten.

Suggested change

// Initialize OpenTelemetry for the Trillian server

//config.InitLogging()

chore(deps): update ctutils

himaschal mentioned this pull request Dec 8, 2025

feat(otel): Add OpenTelemetry distributed tracing and ctutils logging to CTFE digicert/certificate-transparency-go#3

Open

himaschal and others added 17 commits December 10, 2025 17:35

Update ctutils to v0.1.7

518b1d8

ci: add workflow to update ctutils dependencies

c64d10e

fix(ci): use oauth2 syntax for authentication and debug token

86dc36a

fix(ci): use GITHUB_TOKEN for PR creation

fea9c9d

fix(ci): auth to private ctutils for lint and vuln checks

c370095

fix(ci): auth to private ctutils for database integration tests

0991a95

fix(ci): auth to private ctutils and set GOPRIVATE for CodeQL

03dc543

test: clean run trigger for v0.1.11-test

13a414e

chore(deps): update ctutils to latest version

c71aacb

Merge pull request #4 from digicert/chore/update-ctutils

6ae67f5

chore(deps): update ctutils

chore: test pipeline run with ctutils v0.1.12-test

febb1a9

chore(deps): update ctutils to latest version

683f18f

Merge pull request #5 from digicert/chore/update-ctutils

f04e183

chore(deps): update ctutils

chore: test pipeline run with ctutils v0.1.13-test

66422c4

chore(deps): update ctutils to latest version

53ae555

Merge pull request #6 from digicert/chore/update-ctutils

c2fab3b

chore(deps): update ctutils

chore: revert pipeline testing configurations to production ready state

d81bf44

himaschal requested a review from Copilot January 13, 2026 16:03

Copilot started reviewing on behalf of himaschal January 13, 2026 16:04 View session

Copilot AI reviewed Jan 13, 2026

View reviewed changes

himaschal and others added 8 commits January 14, 2026 18:56

RA-8279: Modified to use github app

996c9dc

ci: use github app for private repo access

ac7a3ca

ci: update all workflows to use github app token

8448274

chore(deps): update ctutils to latest version

4300dd0

Merge pull request #8 from digicert/chore/update-ctutils-test

80f9e8f

chore(deps): update ctutils

chore(deps): update ctutils to latest version

221044d

Merge pull request #9 from digicert/chore/update-ctutils-test

ef8cfbb

chore(deps): update ctutils

chore(deps): update ctutils to latest version

c16b7e8

himaschal added 2 commits January 15, 2026 19:13

Merge pull request #10 from digicert/chore/update-ctutils-test

6704de0

chore(deps): update ctutils

fix(ci): remove hardcoded base branch reference

81f2939

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(otel): Add OpenTelemetry distributed tracing and ctutils logging integration #3

feat(otel): Add OpenTelemetry distributed tracing and ctutils logging integration #3

Uh oh!

himaschal commented Dec 8, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Copilot AI Jan 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		http.Handle("/metrics", logging.Middleware(promhttp.Handler()))
		http.HandleFunc("/healthz", logging.MiddlewareFunc(m.healthz))

-# Always copy main repo and ctutils
+# Always copy main repo and ctutils.
+# NOTE: The Docker build context must contain `trillian/` and `ctutils/` directories
+# at its root. Typically this means running `docker build` from the repository root,
+# e.g.: `docker build -f examples/deployment/docker/log_server/Dockerfile.unified .`

	// require github.com/digicert/ctutils v0.1.2 // commented out for localreplace: use local ctutils for development
	// For local development of github.com/digicert/ctutils, use a go.mod replace directive instead of a commented-out require.

-	config.InitLogging()
+	if err := config.InitLogging(); err != nil {
+		klog.Exitf("Failed to initialize logging: %v", err)
+	}

		// Initialize OpenTelemetry for the Trillian server
		//config.InitLogging()

feat(otel): Add OpenTelemetry distributed tracing and ctutils logging integration #3

Are you sure you want to change the base?

feat(otel): Add OpenTelemetry distributed tracing and ctutils logging integration #3

Uh oh!

Conversation

himaschal commented Dec 8, 2025

Summary

Changes

Configuration

Related PRs

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jan 13, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant