Client unable to close connection to cluster

[rmfaheem]

Description
The client enters into a seemingly endless loop trying to close a connection to a cluster after a node was restarted.

To Reproduce
Uncertain at this moment, aside from restarting a node to which a client is connected.

Expected behavior
Successfully closing the connection and re-connecting to the cluster.

Actual behavior
The connection fails to close and this is retried several times, failing each time.

Config/Logs/Screenshots
Connection Settings:

  "MaxRetries": -1, // forever
  "MaxReconnections": -1, // forever
  "ReconnectionDelay": 200,
  "HeartbeatInterval": 1500,
  "HeartbeatTimeout": 5000,
  "OperationTimeout": 20000,
  "EventsReadBatchSize": 25,
  "DisableServerCertificateValidation": true,
  "KeepAlivePingDelaySeconds": 60 
  "ConnectionString": "ConnectTo=tcp://user:pass@x.x.x.x:1113",
  "EventsReadBatchSize": 250

Logs:

May 31st 2022, 16:59:27.605 a b EventStoreConnection 'connection-d': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38224, 86d2eba7-ed69-4b10-aa32-fb8c03574bef] due to HEARTBEAT TIMEOUT at pkgNum 43825.
May 31st 2022, 16:59:27.579 a b EventStoreConnection 'connection-c': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38220, 32d3e74b-dc83-4524-94a7-b393e8bf5920] due to HEARTBEAT TIMEOUT at pkgNum 43817.
May 31st 2022, 16:59:27.468 a b EventStoreConnection 'connection-a': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38200, 99b1e87e-ce30-498b-87eb-9b2539718fe5] due to HEARTBEAT TIMEOUT at pkgNum 43867.
May 31st 2022, 16:59:27.406 a b EventStoreConnection 'connection-d': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38224, 86d2eba7-ed69-4b10-aa32-fb8c03574bef] due to HEARTBEAT TIMEOUT at pkgNum 43825.
May 31st 2022, 16:59:27.383 a b EventStoreConnection 'connection-c': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38220, 32d3e74b-dc83-4524-94a7-b393e8bf5920] due to HEARTBEAT TIMEOUT at pkgNum 43817.
May 31st 2022, 16:59:27.271 a b EventStoreConnection 'connection-a': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38200, 99b1e87e-ce30-498b-87eb-9b2539718fe5] due to HEARTBEAT TIMEOUT at pkgNum 43867.
May 31st 2022, 16:59:27.207 a b EventStoreConnection 'connection-d': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38224, 86d2eba7-ed69-4b10-aa32-fb8c03574bef] due to HEARTBEAT TIMEOUT at pkgNum 43825.
May 31st 2022, 16:59:27.182 a b EventStoreConnection 'connection-c': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38220, 32d3e74b-dc83-4524-94a7-b393e8bf5920] due to HEARTBEAT TIMEOUT at pkgNum 43817.
May 31st 2022, 16:59:27.067 a b EventStoreConnection 'connection-a': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38200, 99b1e87e-ce30-498b-87eb-9b2539718fe5] due to HEARTBEAT TIMEOUT at pkgNum 43867.
May 31st 2022, 16:59:27.007 a b EventStoreConnection 'connection-d': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38224, 86d2eba7-ed69-4b10-aa32-fb8c03574bef] due to HEARTBEAT TIMEOUT at pkgNum 43825.
May 31st 2022, 16:59:26.983 a b EventStoreConnection 'connection-c': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38220, 32d3e74b-dc83-4524-94a7-b393e8bf5920] due to HEARTBEAT TIMEOUT at pkgNum 43817.
May 31st 2022, 16:59:26.867 a b EventStoreConnection 'connection-a': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38200, 99b1e87e-ce30-498b-87eb-9b2539718fe5] due to HEARTBEAT TIMEOUT at pkgNum 43867.
May 31st 2022, 16:59:26.805 a b EventStoreConnection 'connection-d': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38224, 86d2eba7-ed69-4b10-aa32-fb8c03574bef] due to HEARTBEAT TIMEOUT at pkgNum 43825.
May 31st 2022, 16:59:26.779 a b EventStoreConnection 'connection-c': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38220, 32d3e74b-dc83-4524-94a7-b393e8bf5920] due to HEARTBEAT TIMEOUT at pkgNum 43817.
May 31st 2022, 16:59:26.669 a b EventStoreConnection 'connection-a': closing TCP connection [Unspecified/x.x.x.x:1113, x.x.x.x:38200, 99b1e87e-ce30-498b-87eb-9b2539718fe5] due to HEARTBEAT TIMEOUT at pkgNum 43867.

Actual logs in the support ticket.

EventStore details

• EventStore server version: 20.10
• Operating system: Linux
• EventStore client version: 21.2.2.
• .NET Runtime: 5

Additional context
Support Ticket: 110334

[ylorph]

@rmfaheem : the connection string is to a specific node,

"ConnectionString": "ConnectTo=tcp://user:pass@x.x.x.x:1113",

To be able to connect to any node a gossip seed or DNS discovery should be used
e.g : GossipSeeds=127.0.0.1:2113,127.0.0.2:2113,127.0.0.3:2113

[timothycoleman]

is the problem though that it fails to reconnect to the original node even after it has come back up