A ton of socket closed errors on prod

[sescobb27]

Hi there, We're seeing thousands of errors (5-10% of errors, and when we increase load we can even crash the connection pool and crash our server due to these errors) in prod that happens when we lose our connection to Clickhouse Cloud. We have confirmed with them that these errors don't show up in their graphs, so, something is going on with the connection/reconnection handling, We're trying to figure out why we are loosing so many connections. ** (Mint.TransportError) socket closed is the error we are having, in some places we have had to add retries and that worked, but in some cases even 10 retries won't work.

reading the code, it seems the error may be in maybe_reconnect/1 as it returned conn (the broken connection) on reconnect failure, so subsequent requests would use a dead socket and fail with socket closed / timeout errors, maybe we should return something like {:disconnect, reason, conn} and update callers to handle it so it can be propagated to DBConnection so it knows the connection fails and drops that connection instead of using it?

we are using

ch 0.6.1
elixir 1.18.3-otp-27
erlang 27.3.3

[sescobb27]

maybe related too #210 and #208 but not sure 🤔

[hkrutzer]

We also started seeing a lot of these a few months ago. We are not using cloud, but we are updating Clickhouse monthly, so I am not sure it's something that changed in ch or if it's in Clickhouse.

[ruslandoga]

👋 @sescobb27 @hkrutzer

Could be plausible/ecto_ch#252 if you have insert ... select queries.

But there are still quite a few things we could do to handle closed connections better :)

[sescobb27]

we are seeing a ton of The connection was closed by the server; a new connection has been successfully reestablished. so its really weird that the connection was closed and then we re-establish it correctly, return it but still we get the error 🤔

[sescobb27]

ahh!!!! @ruslandoga yes!!!! we are facing that error with the insert selects "mosly" but also in some other queries like alter partitions and move partitions

[hkrutzer]

Could be plausible/ecto_ch#252 if you have insert ... select queries.

For me, we don't have those queries. The error I am seeing on the Clickhouse side is Net Exception: Socket is not connected, while reading from socket.