💥Add worker heartbeat support by yuandrew · Pull Request #2186 · temporalio/sdk-go

yuandrew · 2026-02-10T22:00:12Z

NOTE: This is a re-creation of #2148, making a newer, clean PR as the resolved comment count was getting very high and making it hard to navigate.

What was changed

Key changes:

Add HeartbeatMetricsHandler to capture metrics needed for heartbeats
Add internal_worker_heartbeat.go for heartbeat worker management
💥 Removed contrib/resourcetuner lib, resource tuner portion has been moved into the SDK and is exposed through the worker package, while a new contrib/sysinfo package was created to handle system resource measurement
- We never released a v1.0.0 of contrib/resourcetuner, so this should be okay.
Plumb heartbeat data through workers (workflow, activity, nexus, local activity)
always send ShutdownWorker RPC to give final heartbeat, even if not using sticky queues
Move verifyNamespaceExist calls from each specific worker, and instead call at the AggregatedWorker level and cache for worker heartbeating

Why?

New feature!

Checklist

Closes Worker Heartbeating #2094
How was this tested:

Added a bunch of tests to worker_heartbeat_test.go

Add integration tests for worker heartbeat functionality

Any docs updates needed?

Note

Medium Risk
Touches core worker start/stop and polling paths and adds new background goroutines/RPCs for heartbeating; failures could impact worker shutdown behavior or add load if misconfigured, but the feature is gated by an experimental interval option and server capabilities.

Overview
Adds experimental worker heartbeats: clients can now be configured with WorkerHeartbeatInterval to periodically call RecordWorkerHeartbeat and to send a final heartbeat via ShutdownWorker on stop, including worker/poller/slot metrics, plugin names, deployment version, and optional CPU/memory usage.

Refactors tuning/sysinfo: moves the resource-based tuner into the SDK worker API (requiring an injected SysInfoProvider), introduces new contrib/sysinfo (gopsutil + cgroup-aware) implementation, and wires poll-success tracking plus heartbeat-specific metrics capturing across workflow/activity/nexus pollers.

Updates dev/test configs to enable WorkerHeartbeatsEnabled/ListWorkersEnabled, adjusts integration tests accordingly, and adds a large new worker_heartbeat_test.go coverage suite.

^{Written by Cursor Bugbot for commit 1cb7c90. This will update automatically on new commits. Configure here.}

Implement worker heartbeats that report worker status, slot usage, and metrics to the server via the WorkerHeartbeat RPC. Key changes: - Add HeartbeatMetricsHandler to capture metrics needed for heartbeats - Add internal_worker_heartbeat.go for heartbeat worker management - Add hostmetrics package for CPU/memory usage reporting - Plumb heartbeat data through workers (workflow, activity, nexus, local activity) - Add integration tests for worker heartbeat functionality - Fix nil pointer in shutdownWorker when heartbeating is disabled

… AggregatedWorker.start()

…rrent-safe

… clarify code

…t of metrics

…sion

…eTracker more idiomatic, move namespace capabilities to AW, remove Get from func names

# Conflicts: # contrib/sysinfo/go.sum # internal/cmd/build/main.go

…tbeat worker creation and callback registration

…passes CI

…cheHit checks

internal/internal_worker_heartbeat.go

internal/client.go

yuandrew · 2026-02-10T22:21:21Z

internal/resource_tuner.go

+// SysInfoContext provides context for SysInfoProvider calls.
+//
+// Exposed as: [go.temporal.io/sdk/worker.SysInfoContext]
+type SysInfoContext struct {


#2148 (comment)

cretz
Consider embedding context.Context in here even if just context.Background() (but not that important)

yuandrew
keeping out for now, localActivityContext, NexusOperationContext and testContext are all *Context structs that are missing context.Context, if we want to add it in later to use, we can

cretz
Then IMO we should consider passing a Go context into the calls alongside where this is passed (even if it's not used for anything yet)

yuandrew
I'm planning on leaving it out for now, and adding in the context into the SysInfoContext struct when we need the context. It seems a little redundant to pass in a Context and SysInfoContext together, any reason we'd need to add it into the calls alongside right now?

Go API that is invoked that others implement basically all need a context.Context always even if you don't do anything with it yet. At this point it might as well be part of Go API heh. You can wait and embed later if you want, though harmless to do now.

internal/internal_worker.go

yuandrew · 2026-02-11T18:45:55Z

internal/internal_worker.go

 	}
 	proto.Merge(aw.capabilities, capabilities)

+	if _, err := aw.client.loadNamespaceCapabilities(aw.executionParams.MetricsHandler); err != nil {


@jmaeagle99 this is similar to your change in external storage, we can coordinate

cretz

Nothing blocking, just a few minor comments, though worth looking at them and seeing if we want to address

internal/client.go

internal/internal_worker.go

cretz · 2026-02-12T13:50:07Z

internal/internal_worker.go

 				NumNexusSlots:         defaultMaxConcurrentTaskExecutionSize,
 			})
 	}
+	if params.pollTimeTracker == nil {


What is the situation where this is not always nil? Should this be set where params is created and not here?

It's always nil, but this is the most common place to nil check, mainly for unit tests sake. I had this set where params are created in NewAggregatedWorker, but I'd still like to keep this check here for tests

cretz · 2026-02-12T13:53:59Z

internal/internal_worker.go

+		stickyTaskQueue = getWorkerTaskQueue(aw.workflowWorker.stickyUUID)
+	}
+
+	_, err := aw.client.workflowService.ShutdownWorker(grpcCtx, &workflowservice.ShutdownWorkerRequest{


Can we pass the worker instance key here and in polls?Or if these should be done in separate issue, can we make it and put it up for triage?

It looks like it's still not on shutdown here, but is on polls?

cretz · 2026-02-12T13:55:17Z

internal/internal_worker.go

+	}
+
+	if err != nil {
+		aw.logger.Debug("ShutdownWorker rpc errored during worker shutdown.", tagError, err)


I know this was debug in old code, but we should consider making this a warning, but can be done in a separate issue. This failing has negative user effects, so effectively swallowing this error and not telling user can be rough. But we have to think about situations where this is just noise.

will address separately

cretz · 2026-02-12T14:01:59Z

internal/resource_tuner.go

+// SysInfoContext provides context for SysInfoProvider calls.
+//
+// Exposed as: [go.temporal.io/sdk/worker.SysInfoContext]
+type SysInfoContext struct {


Go API that is invoked that others implement basically all need a context.Context always even if you don't do anything with it yet. At this point it might as well be part of Go API heh. You can wait and embed later if you want, though harmless to do now.

…w/activity poll requests, make WorkerHeartbeatInterval not a pointer

cursor · 2026-02-13T00:11:28Z

internal/internal_worker_heartbeat_metrics.go

+		TotalFailedTasks:           int32(totalFailed),
+		LastIntervalProcessedTasks: int32(intervalProcessed),
+		LastIntervalFailureTasks:   int32(intervalFailed),
+	}


Silent int64-to-int32 truncation corrupts heartbeat task counters

Medium Severity

buildSlotsInfo truncates int64 counter values to int32 via int32(totalProcessed) and int32(totalFailed) when populating the WorkerSlotsInfo proto. The internal tracking uses atomic.Int64, which can grow beyond math.MaxInt32. For a worker processing ~1000 tasks/second, this overflows in ~25 days, silently producing negative or wrapped values for TotalProcessedTasks and TotalFailedTasks. The same applies to TotalStickyCacheHit/TotalStickyCacheMiss in PopulateHeartbeat. Clamping to MaxInt32 would prevent silent corruption.

Additional Locations (1)

internal/internal_worker_heartbeat_metrics.go#L166-L170

Alternatively these should just be stored as int32 since that's what we have to send them as eventually anyway

internal/internal_worker_heartbeat.go

test/worker_heartbeat_test.go

Sushisource

Overall makes sense to me. I didn't do a super detailed review but the high level and tests look reasonable. A few small things.

test/worker_heartbeat_test.go

Sushisource · 2026-02-13T00:46:38Z

internal/internal_worker.go

+		stickyTaskQueue = getWorkerTaskQueue(aw.workflowWorker.stickyUUID)
+	}
+
+	_, err := aw.client.workflowService.ShutdownWorker(grpcCtx, &workflowservice.ShutdownWorkerRequest{


It looks like it's still not on shutdown here, but is on polls?

internal/internal_worker_heartbeat_metrics.go

Sushisource · 2026-02-13T00:51:57Z

internal/internal_worker_heartbeat_metrics.go

+		TotalFailedTasks:           int32(totalFailed),
+		LastIntervalProcessedTasks: int32(intervalProcessed),
+		LastIntervalFailureTasks:   int32(intervalFailed),
+	}


Alternatively these should just be stored as int32 since that's what we have to send them as eventually anyway

internal/internal_worker.go

cursor · 2026-02-13T19:08:55Z

internal/internal_task_pollers.go

+	} else {
+		wtp.pollTimeTracker.recordPollSuccess(metrics.PollerTypeWorkflowTask)
+	}
+


Idle pollers report stale heartbeat timestamps

Medium Severity

pollTimeTracker.recordPollSuccess is only called when a poll returns a task token. Successful empty polls are skipped, so LastSuccessfulPollTime in worker heartbeats can remain stale or zero while pollers are healthy and actively polling. This makes heartbeat poller health data inaccurate during idle periods.

Additional Locations (2)

internal/internal_task_pollers.go#L1182-L1190

internal/internal_nexus_task_poller.go#L87-L95

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

cursor · 2026-02-13T19:42:59Z

internal/internal_worker.go

 	}

+	aw.shutdownWorker()
+


Shutdown sent for never-started workers

Medium Severity

AggregatedWorker.Stop() now always calls shutdownWorker() even when the worker was never started. This can send a ShutdownWorker RPC with a synthesized heartbeat for a worker that never polled, creating misleading worker state and adding avoidable network/retry delay during startup-failure cleanup.

yuandrew added 29 commits January 14, 2026 11:50

vendor gopsutil

6af4db8

PR feedback

04ff20f

Sort plugin names

819080b

Create new hostinfo package

c156e0d

make methods/structs private, remove aw.workerHeartbeatManager

ebf1064

tighten lock, consolidate describeNamespace calls to a single call in…

b8893b9

… AggregatedWorker.start()

simplify heartbeat metrics, decouple poller/worker type from WithTags()

a6135de

remove unused nexus worker, tighten heartbeat callback and make concu…

4f43e75

…rrent-safe

Merge branch 'master' into worker-heartbeat

54ddf1f

Fix tests

e7fbc03

Fix cursor discovered bugs, fix integ tests

edf6e11

Rename hostinfo to sysinfo, add interval enforcement, rename mutexes,…

73f4a10

… clarify code

fix bugs cursor found, sync.oncevalue, separate poll time tracking ou…

972555a

…t of metrics

Add back resource tuner tests that got dropped

f952732

Fix tests

a25d85d

Fix tests, disable heartbeating for normal tests, bump dev server ver…

53da340

…sion

Finish renames of sysInfoProvider, handle Time.IsZero(), make pollTim…

2155206

…eTracker more idiomatic, move namespace capabilities to AW, remove Get from func names

Fix tests

dd02159

remove extra default logger addition, remove dead code

bb556cb

Merge branch 'master' into worker-heartbeat

136d311

# Conflicts: # contrib/sysinfo/go.sum # internal/cmd/build/main.go

forgot a change..

da40521

fix unit tests

04f5d4d

Fix eventually expectation for slower CI machines, fix race with hear…

a5c85d0

…tbeat worker creation and callback registration

Merge branch 'master' into worker-heartbeat

b68132e

Gate all sticky cache tests behind maxWorkflowCacheSize checks so it …

c5b49db

…passes CI

loosen workerInfo.CurrentStickyCacheSize and workerInfo.TotalStickyCa…

faeba63

…cheHit checks

Fix up TestWorkerHeartbeatStickyCacheMiss

004032a

Add comment, minor fix

257e264

yuandrew commented Feb 11, 2026

View reviewed changes

yuandrew marked this pull request as ready for review February 11, 2026 17:24

yuandrew requested a review from a team as a code owner February 11, 2026 17:24

cursor bot reviewed Feb 11, 2026

View reviewed changes

internal/internal_worker.go Show resolved Hide resolved

yuandrew commented Feb 11, 2026

View reviewed changes

cretz approved these changes Feb 12, 2026

View reviewed changes

Make SHUTTING_DOWN status atomic, plumb workerInstanceKeys to workflo…

8d7aa2e

…w/activity poll requests, make WorkerHeartbeatInterval not a pointer

cursor bot reviewed Feb 13, 2026

View reviewed changes

Sushisource approved these changes Feb 13, 2026

View reviewed changes

yuandrew and others added 2 commits February 12, 2026 18:15

Merge branch 'master' into worker-heartbeat1, PR feedback

24e102a

Merge branch 'master' into worker-heartbeat1

300c71d

cursor bot reviewed Feb 13, 2026

View reviewed changes

internal/internal_worker.go Show resolved Hide resolved

internal/internal_worker.go Show resolved Hide resolved

bring back listworkers dynamic config, fix identity in heartbeat

41d6afd

cursor bot reviewed Feb 13, 2026

View reviewed changes

Add dynamic config for listWorkers for docker test

e297ae1

cursor bot reviewed Feb 13, 2026

View reviewed changes

server v1.29.1 still requires dynamic config for heartbeating

1cb7c90

yuandrew merged commit 3f64d34 into temporalio:master Feb 14, 2026
31 of 34 checks passed

yuandrew deleted the worker-heartbeat1 branch February 14, 2026 02:17

yuandrew mentioned this pull request Feb 17, 2026

Change ShutdownWorker err log from Debug to Warn #2195

Merged

yuandrew changed the title ~~Add worker heartbeat support~~ 💥Add worker heartbeat support Mar 5, 2026

Conversation

yuandrew commented Feb 10, 2026 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was changed

Why?

Checklist

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cretz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot Feb 13, 2026

Choose a reason for hiding this comment

Silent int64-to-int32 truncation corrupts heartbeat task counters

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Sushisource left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

cursor bot Feb 13, 2026

Choose a reason for hiding this comment

Idle pollers report stale heartbeat timestamps

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor bot Feb 13, 2026

Choose a reason for hiding this comment

Shutdown sent for never-started workers

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yuandrew commented Feb 10, 2026 •

edited by cursor bot

Loading