Limit the number of requests that are accepted #10
Description
From https://groups.google.com/d/msg/thin-ruby/7p5BHt5j7M4/KtubwDr0wakJ
Depending on exactly how the Heroku load balancers are implemented (particularly what layer), you probably have no real defense against this.
Your best option is to tune the accept backlog (assuming a low layer load balancer), and that should inform the OS, and respectively the LB when your process is "full". We never exposed an API in EventMachine for this tunable, and as such, you may have trouble.
I'd be happy to accept a patch to EM to allow for this, it's presently hardcoded here:
https://github.com/eventmachine/eventmachine/blob/master/ext/em.cpp#L1547
The other problem here, is that EM being concurrency focused, preferences accepting more than just a single socket per "request" (the per-request aspect is related to how a webserver is implemented on top of EM (being blocking for a whole request at a time)):
https://github.com/eventmachine/eventmachine/blob/master/ext/ed.cpp#L1415
Again, this could be made tunable. For a single-threaded Thin style use case, this loop essentially would want to be eradicated, as concurrency is not desired in this way.
What this all really means, is this is the direct cause as to why low layer load balancers cannot balance Thin in a very "fair" way today. Thin will gobble up connections faster than it can service them, so under more than your serviceable load, you will get very unfair scheduling at the load balancer. It is as always, worth noting that this happens when you have more than your serviceable load, so you should really be provisioning new backends already.
What does this all mean for heroku-forward? If heroku can manage expectations with the downstream servers, heroku-forward could be a single point where an accept is or is not happening, a busy response is sent to the LB, etc.