Skip to content

Conversation

@joshuaokolo
Copy link

Screenshot 2025-12-19 at 12 35 23 AM

Summary

This PR fixes intermittent panics and goroutine leaks in the Node.js runtime worker's log streaming functionality that occur when workers are stopped or killed during development.

Problem

The stack trace you provided shows panics occurring in (*Worker).Logs when workers are terminated:

github.com/sst/sst/v3/pkg/runtime/node.(*Worker).Logs.func2()
/home/runner/work/sst/sst/pkg/runtime/node/node.go:104 +0x84

The root cause was the use of io.Copy() for streaming stdout/stderr. When a worker process is killed:

  1. The stdout/stderr pipes close abruptly
  2. io.Copy doesn't gracefully handle write errors to closed pipes
  3. This causes panics and leaves goroutines hanging

Solution

Replaced io.Copy with a manual buffered read/write loop that:

  • Explicitly checks for write errors and exits cleanly when the pipe is closed
  • Properly handles process termination by detecting EOF and returning gracefully
  • Prevents goroutine leaks by ensuring all copy goroutines exit when errors occur
  • Fixes cleanup order by moving writer.Close() after wg.Wait() (was incorrectly using defer before the wait)

Changes

  • Modified pkg/runtime/node/node.go:
    • Replaced two io.Copy goroutines with a reusable copyStream function
    • Added explicit error handling for both read and write operations
    • Fixed writer cleanup order to prevent premature pipe closure

Testing

  • Verified the package compiles successfully
  • Code follows the same pattern used in pkg/runtime/python/python.go which doesn't have these issues
  • Matches the interface contract defined in pkg/runtime/runtime.go

This should eliminate the intermittent panics you're experiencing during sst dev when functions are rebuilt or workers are restarted.

The previous implementation using io.Copy could panic when the worker
process was killed, as io.Copy doesn't handle write errors gracefully
when the destination pipe is closed.

This fix:
- Uses a buffered read/write loop with explicit error checking
- Properly handles pipe closure and process termination
- Prevents goroutine leaks by ensuring all copy goroutines exit cleanly
- Moves defer writer.Close() after wg.Wait() for proper cleanup order

Fixes intermittent panics in github.com/sst/sst/v3/pkg/runtime/node.(*Worker).Logs
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant