getting large resources on slow connection is slow

[grosser]

Version

version 4.11.0 + master (054bff2)

Problem

getting many resources is slow compared to kubeclient --raw, especially when going cross-region, cluster-local is much better (local: 6s vs 2s remote: 50s vs 12s)

Solution

enable gzip or recommend in the readme to enable gzip

# gem "faraday-gzip"
require 'faraday/gizp'
client.configure_faraday { |conn| conn.request :gzip }

... no idea for memory

Background

comparing

x=nil; Benchmark.realtime { x = JSON.parse(`kubectl --context foo get --raw '/api/v1/pods?resourceVersion=0'`, symbolize_names: true) }`

with ps -o rss= -p 74108 and GC.start
alternative: GC.start; sleep 10; (Integer(File.read("/proc/#{Process.pid}/statm").split(" ")[1]) * 4096.0 / 1024 / 1024).round
kubectl in ruby takes ~12s and 50mb of memory
kubeclient takes 40s and 600mb initial + ~300 inc of memory (leaking but eventually gets gced, unclear why it takes so long)
with

config = Kubeclient::Config.read(File.expand_path('~/.kube/kubeconfig_zendesk.yaml'))
context = config.context 'foo'
c = Kubeclient::Client.new(
  context.api_endpoint,
  'v1',
  ssl_options: context.ssl_options,
  auth_options: context.auth_options,
  as: :parsed_symbolized
)
x = nil; Benchmark.realtime { x = c.get_entities(nil, 'pods', resource_version: 0) }

using without parsed_symbolized is 5s slower and uses even more (1GB)

data = nil
time = Benchmark.realtime do
  uri = URI.parse("#{host}#{path}")
  http = Net::HTTP.new(uri.host, uri.port)
  http.use_ssl = true
  http.verify_mode = OpenSSL::SSL::VERIFY_NONE # You may want to remove this for production
  request = Net::HTTP::Get.new(uri.request_uri)
  request["Authorization"] = "Bearer #{api_token}"
  response = http.request(request)
  data = JSON.parse(response.body, symbolize_names: true)
  GC.start
end
puts time

require 'curb'
url = "#{host}#{path}"

data = nil
time = Benchmark.realtime do
  http = Curl.get(url) do |http|
    http.headers['Authorization'] = "Bearer #{api_token}"
    http.headers['Accept'] = "application/json, */*"
    http.ssl_verify_peer = false # You may want to remove this for production
  end
  data = JSON.parse(http.body_str, symbolize_names: true)
  GC.start
end
puts time

# --compressed reduces by ~50% .. http2 seems to not help much
data = nil
time = Benchmark.realtime do
  `curl --compressed --http2 -k -H "Authorization: Bearer #{api_token}" "#{url}" > cache`
  data = JSON.parse(File.read("cache"), symbolize_names: true)
  GC.start
end
puts time

data = nil
time = Benchmark.realtime do
  `kubectl --context foo get --raw '#{path}' > cache`
  data = JSON.parse(File.read("cache"), symbolize_names: true)
  GC.start
end
puts time

FYI

• upgrading from 4.9.3->4.11.0 reduced the time from 55s -> 45s (could be randomness, not sure)
• as: :raw did not help either
• comparing to a raw net/http: memory growth is similar, time is similar
• comparing to curb: time is similar (40s) and memory growth is similar
• shelling out to curl is even slower and crashes at ~90s
• tried curl with --http2 same result
• curl with --compressed makes that ~17s