Support for providing kubeclient config for core and group APIs?

[joeauty]

We are trying to use this gem using the kubeclient instructions for connecting inside a cluster (using a service account and its token), and this is mostly working, but we can only get Kubeclient get_jobs calls working with client configured as:

config.kubeclient = Kubeclient::Client.new(
    'https://kubernetes.default.svc/apis/batch',
    'v1',
    auth_options: auth_options,
    ssl_options:  ssl_options
  )

and can only get get_pods calls working with the client configured as follows:

config.kubeclient = Kubeclient::Client.new(
    'https://kubernetes.default.svc',
    'v1',
    auth_options: auth_options,
    ssl_options:  ssl_options
  )

It looks like resque-kubernetes is not built to support the core and group APIs when a custom kubeclient config is provided (we do not have a production ~/.kube/config volume mount)? I'm wondering if this assessment is correct, if any users have found workarounds, or whether this is an appropriate thing to issue a patch for?

[jeremywadsack]

@joeauty I think you right. Thanks for uncovering this oversight.

In this code, we assume a single configured client can handle all scopes:

resque-kubernetes/lib/resque/kubernetes/jobs_manager.rb

Lines 60 to 65 in 601901f

	def client(scope)
	return RetriableClient.new(Resque::Kubernetes.kubeclient) if Resque::Kubernetes.kubeclient
	client = build_client(scope)
	RetriableClient.new(client) if client
	end

But the build_client method clearly produces a different client depending on the scope provided.

resque-kubernetes/lib/resque/kubernetes/jobs_manager.rb

Lines 67 to 74 in 601901f

	def build_client(scope)
	context = ContextFactory.context
	return unless context
	@default_namespace = context.namespace if context.namespace
	Kubeclient::Client.new(context.endpoint + scope, context.version, **context.options)
	end

Do you have any thoughts about how to change the configuration process so this works better for you?

[joeauty]

@joeauty I think you right. Thanks for uncovering this oversight.

In this code, we assume a single configured client can handle all scopes:

resque-kubernetes/lib/resque/kubernetes/jobs_manager.rb

Lines 60 to 65 in 601901f

	def client(scope)
	return RetriableClient.new(Resque::Kubernetes.kubeclient) if Resque::Kubernetes.kubeclient
	client = build_client(scope)
	RetriableClient.new(client) if client
	end

But the build_client method clearly produces a different client depending on the scope provided.

resque-kubernetes/lib/resque/kubernetes/jobs_manager.rb

Lines 67 to 74 in 601901f

	def build_client(scope)
	context = ContextFactory.context
	return unless context
	@default_namespace = context.namespace if context.namespace
	Kubeclient::Client.new(context.endpoint + scope, context.version, **context.options)
	end

Do you have any thoughts about how to change the configuration process so this works better for you?

This might not be the most elegant approach, but an approach that should work:

Resque::Kubernetes.configuration do |config|
  config.enabled = Rails.env.production?

  auth_options = {
    bearer_token_file: '/var/run/secrets/kubernetes.io/serviceaccount/token'
  }
  ssl_options = {}
  if File.exist?("/var/run/secrets/kubernetes.io/serviceaccount/ca.crt")
    ssl_options[:ca_file] = "/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
  end
  config.kubeclient[:core] = Kubeclient::Client.new(
    'https://kubernetes.default.svc',
    'v1',
    auth_options: auth_options,
    ssl_options:  ssl_options
  )
  config.kubeclient[:group] = Kubeclient::Client.new(
    'https://kubernetes.default.svc/apis/batch',
    'v1',
    auth_options: auth_options,
    ssl_options:  ssl_options
  )
end

The reason I'm not sure if something like this would be the best approach is because I don't know if there is a use case for wanting to separate hosts/auth/ssl options per client like this, but I guess this might also be better than manipulating the client's path at execution time?

[jeremywadsack]

Right, the client has the facility to use different endpoints, auth, etc. for each service but I don't know how much that's done in practice. As a reference point, the standard Kubernetes cluster has a single client authorization. Same is true for Google's default application credentials as well as their service credential approach. Given those it seems that's a very distant use case that would have separate clients.

At the same time manipulating the path (or anything) about the provided client would be poor.

I'd like to keep the configuration as simple as possible for the simplest use cases, but I don't see a way of getting around providing two clients (as we do in the gem in the default auth cases).

Thanks for identifying this and providing a suggestion. Would you be able to submit a PR for this (given that you have a system to test it in)?

[jeremywadsack]

Also, I just re-read your opening sentence and you should be able to get the gem to work without configuring any client. It should auto-detect that you are in a standard cluster and handle it for you.

[joeauty]

Yup, it does work without the kubeclient config. I went down this path because I was getting RBAC errors, but I should have taken them at face value and noticed that was due to my RBAC not being cluster scoped.

I just need to figure out how to pass the command's context into the manifest so it can dynamically set the right command to run, but that is obviously well out-of-scope for this ticket.

Hopefully this conversation inspires visibility into what might go wrong with a user that actually needs a custom client config so I haven't wasted your time!