Failed to start HTTP/2 stream

[richardwu]

I get the the following error in rare occurrences, at least in dev (have not observed yet in prod):

/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call.ts:81
  return Object.assign(new Error(message), status);
                       ^
Error: 13 INTERNAL: Failed to start HTTP/2 stream
    at Object.callErrorFromStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call.ts:81:24)
    at Object.onReceiveStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/client.ts:334:36)
    at Object.onReceiveStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/client-interceptors.ts:434:34)
    at Object.onReceiveStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/client-interceptors.ts:397:48)
    at Http2CallStream.outputStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call-stream.ts:230:22)
    at Http2CallStream.maybeOutputStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call-stream.ts:280:14)
    at Http2CallStream.endCall (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call-stream.ts:263:12)
    at Http2CallStream.cancelWithStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call-stream.ts:604:10)
    at /mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/channel.ts:349:34
    at runMicrotasks (<anonymous>)

Is this simply just intermittent flakiness of trying to initiate a websocket connection with Google PubSub?

[richardwu]

Looks like this may be related to losing internet connection (e.g., putting my dev machine to sleep). Another error may be:

/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call.ts:81
  return Object.assign(new Error(message), status);
                       ^
Error: 13 INTERNAL: Received RST_STREAM with code 2
    at Object.callErrorFromStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call.ts:81:24)
    at Object.onReceiveStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/client.ts:334:36)
    at Object.onReceiveStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/client-interceptors.ts:434:34)
    at Object.onReceiveStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/client-interceptors.ts:397:48)
    at Http2CallStream.outputStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call-stream.ts:230:22)
    at Http2CallStream.maybeOutputStatus (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call-stream.ts:280:14)
    at Http2CallStream.endCall (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call-stream.ts:263:12)
    at ClientHttp2Stream.<anonymous> (/mnt/ubuntu/home/richardwu/code/topspin-apollo-backend/app/node_modules/google-gax/node_modules/@grpc/grpc-js/src/call-stream.ts:552:14)
    at ClientHttp2Stream.emit (node:events:327:20)
    at ClientHttp2Stream.EventEmitter.emit (node:domain:486:12)
[nodemon] app crashed - waiting for file changes before starting...

Not sure if there could be some clever retry logic added in here.

[richardwu]

So it looks like I'm running into this in prod as well: my temporary hack is to ensure I always have 2+ replicas and have k8s restart the servers.

This seems to be related to googleapis/nodejs-pubsub#437 and googleapis/nodejs-vision#785: perhaps a version bump of google-pubsub/grpc will solve this?