Resource leak with Node.js 20

[oporkka]

I am leaving this as information for the community. I understand this library might not be anymore supported, so I'm not expecting a fix.

We are running a Node.js application which is a GraphQL aggregator, with each incoming request creating a lot of client spans for each dependency. We were running lightstep-tracer 0.35.0-no-protobuf when we tried to upgrade our Node.js version from latest version 18 to latest version 20.

What we observed after switching some canary traffic to the deployment running on Node.js 20 was that the pod counts started fluctuating up and down

This was caused by some pods having high CPU usage (yellow line for max CPU) which increased the average CPU usage (orange line) until the autoscaling happened. After some time the max CPU usage went down close to average and the fleet scaled in again.

We observed that garbage collection had difficulties freeing space in heap

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Another observed new message was MaxListenersExceededWarning at pod startup even before it got any traffic.

(node:1) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 timeout listeners added to [TLSSocket]. Use emitter.setMaxListeners() to increase limit

We experimented by running our application with noopTracer with Node.js 20 and the problems went away. After that we also updated to OpenTelemetry libraries, which caused us to report spans again, and the problem did not reproduce anymore.

We concluded that there is a resource leak with lightstep-tracer with Node.js 20. This conclusion is further supported by the fact that lightstep-tracer is doing a hacky wrapper over EventEmitter (mentioned in MaxListenersExceededWarning: Possible EventEmitter memory leak detected)

lightstep-tracer-javascript/src/imp/tracer_imp.js

Lines 187 to 217 in 52ebe87

	// Morally speaking, Tracer also inherits from EventEmmiter, but we must
	// fake it via composition.
	//
	// If not obvious on inspection: a hack.
	_delegateEventEmitterMethods() {
	let self = this;
	this._ee = new EventEmitter();
	// The list of methods at https://nodejs.org/api/events.html
	_each([
	'addListener',
	'emit',
	'eventNames',
	'getMaxListeners',
	'listenerCount',
	'listeners',
	'on',
	'once',
	'prependListener',
	'prependOnceListener',
	'removeAllListeners',
	'removeListener',
	'setMaxListeners',
	], (methodName) => {
	self[methodName] = function () {
	if (self._ee[methodName]) {
	// eslint-disable-next-line prefer-spread
	self._ee[methodName].apply(self._ee, arguments);
	}
	};
	});
	}

.

Moreover, the support for Node.js 20 in lightstep-tracer seems unverified, as the test suite does not test with Node.js 18 or 20

lightstep-tracer-javascript/.circleci/config.yml

Lines 137 to 152 in 52ebe87

	- "node-12":
	filters:
	branches:
	only: /.*/
	- "node-14":
	filters:
	branches:
	only: /.*/
	- "node-16":
	filters:
	branches:
	only: /.*/
	- "node-latest":
	filters:
	branches:
	only: /.*/