Least Request is not working as expected above 1.30 versions

[paramjeet01]

Title: Least request load balancer policy is not working as expected

Description:

What issue is being seen?
After upgrading the envoy proxy version to 1.30 and above , the least request's behaviour has changed while using in large scale.

Repro steps:

Upgrade envoy proxy to v1.30 or above versions.
Create 50 application pods and 18 load testing tool pods with vegeta serving 60k RPS
Each application pod can handle upto 20-25k RPS and load testing tool will generate 60k RPS per pod.
Now, when we scale up the application pods from 50 to 60 pods , we will notice that the requests for few pods are not actually least request load balanced.

[paramjeet01]

Please find the supporting metrics

v1.30

v1.29

[paramjeet01]

This PR merged into 1.30 version could have caused the issue : #31507

[wbpcode]

It's weird becase if you doesn't enable the full scan, the #31507 should change nothing. Could you also share your cluster here? Thanks.

[bparamjeet]

@wbpcode , please find the cluster here.

    {
     "version_info": "c5b83a8872029242ac5492a833468005aa624f0d4d55765784cc2525ff0f4dfb",
     "cluster": {
      "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
      "name": "httproute/eg-test/lor-test-1-internal/rule/0",
      "type": "EDS",
      "eds_cluster_config": {
       "eds_config": {
        "ads": {},
        "resource_api_version": "V3"
       },
       "service_name": "httproute/eg-test/lor-test-1-internal/rule/0"
      },
      "connect_timeout": "10s",
      "per_connection_buffer_limit_bytes": 32768,
      "lb_policy": "LEAST_REQUEST",
      "circuit_breakers": {
       "thresholds": [
        {
         "max_connections": 36000,
         "max_pending_requests": 1024,
         "max_requests": 4096,
         "max_retries": 1024
        }
       ]
      },
      "dns_lookup_family": "V4_ONLY",
      "ignore_health_on_host_removal": true,
      "typed_extension_protocol_options": {
       "envoy.extensions.upstreams.http.v3.HttpProtocolOptions": {
        "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions",
        "common_http_protocol_options": {
         "idle_timeout": "70s"
        },
        "explicit_http_config": {
         "http_protocol_options": {}
        }
       }
      },
      "least_request_lb_config": {
       "slow_start_config": {
        "slow_start_window": "60s"
       }
      }
     },
     "last_updated": "2025-02-21T11:29:56.370Z"
    },

[adisuissa]

cc @tonya11en who may have some insights here.

[tonya11en]

@paramjeet01 I'm not sure what http_requests_total is exactly, since it's not a metric emitted by the Envoys. It should be obvious because of the name, but I can't look at the code that emits it and I've learned not to make assumptions in these kinds of investigations.

For those same time ranges you show, let's get screenshots of the following metrics emitted by the Envoys and grouped by pod. We'll want to just see one metric at a time, so one screenshot for each metric/version combo:

• upstream_rq_total
• upstream_rq_active
• upstream_rq_retry
• max_host_weight

Thanks

[tonya11en]

/assign @tonya11en

[bparamjeet]

@tonya11en , http_requests_total is exposed by the service application pod which represents the total number of HTTP requests received. We have used this metric to validate the behaviour of the LOR. We can clearly see the differences in the behaviour of LOR between 1.29 and 1.30 - above versions where more requests are being sent to specific pod. We have also done the tests multiple time to validate the same. Please find the screenshots requested

metrics for v1.29

metrics for v1.30