Can spire agent send its build version to spire server for telemetry?

[nikotih]

Background
In our setup, we are managing spire servers and our customers are managing spire agents. It will be useful to know which agent versions are currently in use e.g. last 30 days. We don't have access to customer runtimes apart from what is needed for challenges.

Agent versions are needed for risk assessment for server upgrades or as a reminder to upgrade agents for the customers.

Our specific example:
we have spire agent plugin - svid store - version X. It was having a bug and we fixed it in agent plugin version Y. We announced that everyone should move to new agent image but overall we have no clue how many customers did that. (i guess we can see image pull statistics?)
3 months later we performed data migration that broke some customers who were still running on older version due to mentioned bug. In here of course svid store plugin is purely in agent side so we have no idea what versions people run, but we can weakly link to spire agent version e.g. 1.13.1

Other example:
We run spire server 1.12.* and have agents connected with versions 1.11.* and 1.12.* (+-1 minor version compatibility). We bump spire server to 1.13.* and any agents running on 1.11.* might get into compatibility issues.

Proposal
Agent sends its build version as part of:
a) node attestation only?
b) with every request as user-agent gRPC header? e.g. (User-Agent: spire-agent/1.13.1)

Spire server can log this information or emit in grpc metric? By default this information is untrusted but it wont be used by attestation or authorizations, only for telemetry

[sorindumitru]

One way we could easily report this is through metrics from the agent. For example we periodically report the uptime in ms. We could include tags for the version and the trust domain, which are currently only reported at start up. Would that work in your case? We think that having the metrics exported from you users environment would be useful to you in other ways too, so even if you don't have access to them at the moment, it would be good to get access to them.

Now for the actual request in this issue, we had some discussion points:

• Which RPCs are safe? Some of the RPCs are unauthenticated so we might not want to receive untrusted data and log it or emit it as metrics directly. This makes it hard to use for the most logical place to report this, which is node attestation since that's usually unauthenticated. It may be possible to use SyncAuthorizedEntries for this since it happens periodically and it is an authenticated call.
• How the request is passed. We had preference for using RPC fields for this instead of the user agent since it allows us to better control the shape of that information
• Should we store the version information? This might be useful, for example having a command to report the minimum agent version that was previously reported.
• Could we include external plugin information in the metric/user agent? From your report it seemed like your issue was more about upgrading an external plugin. Reporting the agent version might not give any indication of the plugin version. We could include information about external plugins in the reported data. For example the checksum of the plugins, if configured.

We'll have to discuss this some more, but in principle this would be nice to have in some kind of form. @nikotih, let us know if you have any views on the points above.

[nikotih]

• Agent metric sounds reasonable but wont work for us as server only operators. typically people want to protect metrics endpoint from outside calls (e.g. kube-rbac-proxy or only internal network address) so i dont think we would ever get a hold on that information. Our max usage projection is ~10_000 tenants and close to 500_000 agents, i think polling or asking for this information would be harder than if agent sending it in some way

• Sanitizing value is always an option, but if version is available via SyncAuthorizedEntries only i think its fine too. We currently utilize experimental.sync_interval to reduce amount of calls but overall our range is 60s to 5m (instead of default 5s)

• gRPC field or header - what better fits into spire design is fine with us. I suppose currently user-agent value is grpc/version

• Persisting might be tricky no? if you bind it to SyncAuthorizedEntries there might be a lot of concurrent write attempts (at least during first startup). For us its fine if its just in metric / log. we can persist and query those outside of spire

• External plugin information is tricky. We dont utilize plugin checksums anymore, those were causing more pain and confusion than help. We mostly rely on overall package (e.g. docker) tag/checksum/signature. If we of course start versioning plugins too and agent can ask plugins for its versions - that would work but its even more complicated. For the start i suggest we can just go with agent version

[sorindumitru]

• Agent metric sounds reasonable but wont work for us as server only operators. typically people want to protect metrics endpoint from outside calls (e.g. kube-rbac-proxy or only internal network address) so i dont think we would ever get a hold on that information. Our max usage projection is ~10_000 tenants and close to 500_000 agents, i think polling or asking for this information would be harder than if agent sending it in some way

Pushing or pulling stats depends on the telemetry sink config. Prometheus would require you to pull, but the others can be configured to push the metrics to some endpoint. Does that change anything for you?

[nikotih]

its an option of course, but would require us to spin up entire new metrics system with all its complexity (auth, scaling etc). Is that something you recommend?

Prometheus technically also has push gateway but i guess it does not really fit this use case.

[sorindumitru]

I would say that the metrics (and logs) are not useful just for the version, they're useful in general. You want to have access to telemetry to be able to debug any issues your users encounter. This probably depends a bit on the support model you provide, but I think it's generally useful. The fact that you also get information about the version is an added bonus.

[sorindumitru]

We discussed this yesterday again. @rturner3 remembered about #2125, from a while ago. We propose we effectively do what discussed there somewhat:

• When the agent starts up it will use the PostStatus API to notify the server about it's current status. That API was initially going to be used for tracking bundle propagation, but it's unused and unimplemented now. We propose adding the version to the request and implementing the API in spire-server.
• The server will update the status of the agent in the datastore in the same table that is currently tracking the rest of the agent information.
• ListAgents and GetAgent should also return the version
• The CLI will report the version of the agents.

I believe this should also cover your use case. Would you be open to help implement this?

[nikotih]

hi @sorindumitru

we are definitely interested - i will create a backlog item on our side, but it might be we wont find time for it this year. So no strong commitments