TCP keepalive support - Traefik unusable in Azure

[joonas-fi]

Possibly related to #727, but am not sure if that issue is only about HTTP keepalives (which is different than TCP keepalive).

I haven't seen TCP keepalive documentation (or other issues) in Traefik?

Why? 1

The other useful goal of keepalive is to prevent inactivity from disconnecting the channel. It's a very common issue, when you are behind a NAT proxy or a firewall, to be disconnected without a reason.

I am experiencing this exact issue with Azure. Even though I use a public IP which is directly attached to my VM, for absolutely no reason they have a forced NAT gateway in front of it (it's a public IP with 1:1 relationship with the VM!!!) which I cannot do anything about. This issue only manifests when I'm using Cloudflare in front of Traefik - and to optimize latency they keep the connection open. Azure's "SNAT" implementation1 kills the connection @ 4 minute idle mark. I verified this with netcat & tcpdump.

Microsoft's cloud is actually pretty disgusting compared to any other sane alternatives like AWS or DigitalOcean in many respects (I'll probably write a blog post about it), but I have to use them for business reasons.

Would it be possible to add TCP keepalive support to Traefik? Any pointers on which file the patch should go into? It's probably at the point when the server accepts a new connection on the socket, but I didn't find it (perhaps it's somewhere at mailgun/manners).

For reference: Nginx keepalive switches: http://nginx.org/en/docs/http/ngx_http_core_module.html (search for so_keepalive)

Linux kernel also has keepalive settings, but the keepalive feature is opt-in per-socket, so the app has to do the equivalent of setsockopt(so, SOL_SOCKET, SO_KEEPALIVE, 1) before the kernel assists in sending keepalive probes. More of this in link 2.

[joonas-fi]

It seems that mailgun/manners uses TCPKeepAliveListener for TLS sockets (https://github.com/mailgun/manners/blob/master/server.go#L188)

Quick glance didn't make me confident that TCPKeepAliveListener is used for plaintext sockets. I only see it used from ListenAndServeTLSWithConfig but not from ListenAndServe (= plaintext counterpart). Is this a bug?

[emilevauge]

@joonas-fi can you confirm it works with TLS?

[joonas-fi]

@emilevauge: yes, good idea to test TLS vs. plaintext! :) I just now tested and confirmed that TLS works while plaintext doesn't. On Azure:

• over port 80 Traefik became unresponsive at 4m30s of radio silence.
• over port 443 (TLS) Traefik responded to me after 10 mins of radio silence.

=> looks like a bug in mailgun/manners, as it doesn't seem to make sense that TLS actively uses TCPKeepAliveListener but plaintext socket does not.

edit: if it helps, I can provide for a Docker-enabled box on Azure for (short-term)testing of this issue's resolution.

[joonas-fi]

It seems to be a bug, as in this commit it was documented that it was supposted to be implemented for plaintext sockets as well: mailgun/manners@448d7b5#diff-804732da4aad8a84ebedfb545342f187L60

Don't ask me why that commit is about removing the feature and why some newer commit introduced it again..

[joonas-fi]

BTW if somebody else wants to reproduce this bug, just telnet (Windows) or nc (Linux) to either the plaintext or TLS port, wait a desired amount of time and whack Enter key three times. Both plaintext and TLS connection should result in error from three newlines, if the connection is still alive. If it is not alive, nothing happens and it's just stuck waiting (at least in Azure case) because the TCP connection broke down.

[joonas-fi]

I've implemented the fix and tested it as working: https://github.com/function61/manners/commit/3d17f4a196bc7cec444d281b54217a5f1ef705c5

I now got a response from Traefik (on Azure) after 11 minutes of radio silence on the plaintext socket. I'll probably post a reliability difference comparison after a few days of more data.

I've submitted a PR to mailgun/manners#13

But in the meantime, if you'd like to integrate that fix, patch:

• server.go: mailgun/manners => function61/manners
• glide.yaml: mailgun/manners => function61/manners
• glide.lock: mailgun/manners => function61/manners
• glide.lock: a585afd9d65c0e05f6c003f921e71ebc05074f4f => 3d17f4a196bc7cec444d281b54217a5f1ef705c5

I could've submitted a PR to Traefik as well, but I don't know how to update the hash and timestamp in glide.lock and cannot be bothered to learn it just for this simple patch.

[joonas-fi]

I wrote the blog post: https://joonas.fi/2017/01/23/microsoft-azures-networking-is-fundamentally-broken/

[joonas-fi]

The reliability comparison I promised:

prod5 was fitted with the keepalive patch on the day with score of 97.032 %. Both prod4 and prod5 uptimes are monitored with Cloudflare proxying Traefik.

I think it is safe to say that with this change Traefik now works on Azure.

@emilevauge: any ETA on including this in Traefik?

[joonas-fi]

@emilevauge ?

[joonas-fi]

@emilevauge I'm sorry for spamming, but I'd very much like to see this implemented in milestone:1.2.

It seems to be a clear bug (with an easy, already implemented fix) in manners because:

• keepalive is enabled for TLS sockets
• but not plaintext sockets.

Upstream (mailgun/manners) seems unresponsive towards PRs, so I hope you'll at least temporarily point to my fork. I wouldn't want to maintain a fork of Traefik just for Azure compatibility.

[emilevauge]

@joonas-fi
Sorry for my late answer :/

1. As go 1.8 now supports graceful connections shutdown, we will not need manners anymore. Do you know if this would be difficult to do the same using the std lib?
2. We are planning to release more often, if this is not in 1.2, the next release should not wait too much ;)
3. Looking at this project https://github.com/felixge/tcpkeepalive, where the author seems to have worked with the help of multiple contributors, it says:

Known Issues: Some problems with the implementation were reported, I'll try to fix them when I get a chance, or if somebody sends a PR.

I wonder if this change is that safe 🤔

WDYT?

[joonas-fi]

@emilevauge

1. Ok that's great, one dependency less to worry about :)
   Yes, this is very easy to do with std lib. If you look at the keepalive implementation in manners: https://github.com/mailgun/manners/blob/master/listener.go#L133
   => You'll see that all it does is wrap Accept() and call SetKeepAlive(true) and SetKeepAlivePeriod(3 * time.Minute) on the underlying connection (net.Conn).

2. Great :)

3. I'm pretty sure that if there's an issue it's related to their implementation (perhaps the stdlib's API for keepalives was not there at the time because they access file descriptor and different APIs) - not TCP keepalive in general as an idea. And Traefik has had keepalive by using Manners for the TLS socket always without issues, so extending the keepalive to plaintext socket is a non-issue in my opinion. :)
   Though you could make it configurable, by adding a conf parameter keepalive_interval (safe default 180 secs (3 mins) so nothing changes in Traefik), so users can either disable keepalive (keepalive_interval=0) or tune it.

[ldez]

Do you still have the same problem with the latest version? releases