opkssh verify is very slow

[clementbrls]

When connecting via SSH using a private key generated by the opkssh tool, the connection and authentication process is slow, often taking approximately 10 seconds to establish the shell session.

I am using the command: ssh root@hostname -i .\.ssh\id_ecdsa -o "IdentitiesOnly=yes"
I have tested this across different servers and with various OIDC providers, and the delay is consistently present. Am I the only one experiencing this issue?

[EthanHeilman]

Thanks for reporting this.

I'll run some tests on my end but it should be as fast as regular SSH and I have no noticed any slowdowns. It should not be that slow!

Some questions to help me understand what is happening:

• Did you just start using opkssh and noticed this or did it become slow recently?
• What version are of opkssh are you running on client and server? opkssh --version
• How many entries do you have in your policy file /etc/opk/auth_id or ~/etc/opk/auth_id?
• How many users do you have on this system?
• What OpenID Provider are you using?

[EthanHeilman]

@clementbrls Is this still an issue?

[clementbrls]

I'm running into a performance issue that I can't determine is new, as I've never used opkssh before.

Key Details:

• I'm using v0.10.0 on both the clients and the server.
• I only have 2 entries for auth_id and one provider configured.
• I have 10 total users, but the problem persists even when testing with a single user.
• I initially used a custom OpenID Provider (Authentik), but to rule out my identity provider as the bottleneck, I also tested with Google. The slow performance issue remains the same across both providers.

Thanks in advance for taking the time to look into this problem!

[EthanHeilman]

I wasn't able to replicate on ubuntu.

Can you post the log files at /var/log/opkssh.log? If you don't feel comfortable posting them, you can email them to ethan.r.heilman@gmail.com

What is the latency from your host to the SSH server (ping <server>)? Is it just as slow when you ssh using an public key not associated with opkssh?

[EthanHeilman]

Any update on this or more information? I want to fix the issue you are having but I can't replicate it on my side

[EthanHeilman]

I was going to close this since I can't replicate, but I'm assigning to copilot to see if it finds anything.

[EthanHeilman]

@clementbrls

Copilot didn't find the issue, but the PR is generated got me thinking about how long it takes to download the JWKS keys and I think I've found the issue.

Your command doesn't do what you intended to do only send the SSH key .\.ssh\id_ecdsa
ssh root@hostname -i .\.ssh\id_ecdsa -o "IdentitiesOnly=yes"

I tested this on windows and found the following behavior:

1. If .\.ssh\id_ecdsa is not found, it sends all your SSH keys. If this takes 1.5 seconds per SSH key and you have 7 SSH keys, it may take 10.5 seconds (worst case).

2. If .\.ssh\id_ecdsa is found, but you have other SSH keys configured in your SSH config at ~/.ssh/config or ~/.ssh/opkssh/config, OpenSSH will send these keys as well.

Either of these two issues could be causing your problem.

To confirm run ssh with `-v- flag and it will show you the keys it sends to the server.

ssh -v root@hostname -i .\.ssh\id_ecdsa -o "IdentitiesOnly=yes"

If you can confirm you are still sending one key and seeing this latency, please let me know. Also provide the logs for ssh -vvv root@hostname -i .\.ssh\id_ecdsa -o "IdentitiesOnly=yes"

Solutions

1. You can fix this by reducing the OPKSSH SSH keys on your system (if this is the cause)

2. I can fix this by speeding up JWKS access via caching in openpubkey. Which I should do anyways.

Notes

You can ignore this, these are just my notes on latency.

When looking up the JWKS keys in OpenPubkey https://github.com/openpubkey/openpubkey/blob/main/discover/discover.go#L95C6-L95C21

1. We do a DNS lookup for the well-known URI,
2. Then we make a TLS connection to the well-known URI to get the JWKS URI,
3. We do a DNS lookup for the JWKS URI,
4. Then we make a new TLS connection to download the JWKS from the URI.
5. Then we download the JWKS URI.

All of these are blocking calls. If we assume 150ms per call, that's 750ms.

OpenSSH always calls AuthorizedKeysCommand twice, this is an intended feature of OpenSSH. So it should take about 1.5 seconds. This is the time I see.

However if your latency between your server and the Google was greater it could be much higher. TLS is a bunch round trips 200ms ping latency could increase this to something approaching 10 seconds.

[minstrike520]

@EthanHeilman Hello, I just encountered some problems similar to this, so I guess I can provide some valid context :D

Answering some questions above:

Some questions to help me understand what is happening: (rest omitted)

• I just started using opkssh today and it's laggy since first use
• 0.11.0
• I had only one entry in my /etc/opk/auth_id and one policy script:

  #!/usr/bin/env sh
  # /etc/opk/plug-cmd.sh
  SUPPOSED_EMAIL="${OPKSSH_PLUGIN_U}@gmail.com"
  
  if [ "${OPKSSH_PLUGIN_EMAIL}" = $SUPPOSED_EMAIL  ]; then
       echo "allow"
       exit 0
  fi
  

• I have roughly 40 users in my system
• I'm using Authentik too. Haven't tested the Google one on my system yet

ssh host@hostname -vvv -i ~/.ssh/id_ecdsa -o IdentitiesOnly=yes

(...OMITTED)
debug1: Offering public key: /home/bladeisoe/.ssh/id_ecdsa ECDSA SHA256:dYsltOzDCDok/Lnuu9z2zBhtFEsiU2Gdk8PQxgGVZOQ  
debug3: send packet: type 50  
debug2: we sent a publickey packet, wait for reply  
debug3: receive packet: type 51  
debug1: Authentications that can continue: publickey,password  
debug1: Offering public key: /home/bladeisoe/.ssh/id_ecdsa ECDSA-CERT SHA256:dYsltOzDCDok/Lnuu9z2zBhtFEsiU2Gdk8PQxgGVZOQ  
debug3: send packet: type 50  
debug2: we sent a publickey packet, wait for reply  

(ABOUT 7 SECONDS PASSED)
  
debug3: receive packet: type 60  
debug1: Server accepts key: /home/bladeisoe/.ssh/id_ecdsa ECDSA-CERT SHA256:dYsltOzDCDok/Lnuu9z2zBhtFEsiU2Gdk8PQxgGVZOQ  
debug3: sign_and_send_pubkey: using publickey-hostbound-v00@openssh.com with ECDSA-CERT SHA256:dYsltOzDCDok/Lnuu9z2zBhtFE  
siU2Gdk8PQxgGVZOQ  
debug2: sign_and_send_pubkey: using private key "/home/bladeisoe/.ssh/id_ecdsa" for certificate  
debug3: sign_and_send_pubkey: signing using ecdsa-sha2-nistp256-cert-v01@openssh.com SHA256:dYsltOzDCDok/Lnuu9z2zBhtFEsiU  
2Gdk8PQxgGVZOQ  
debug3: send packet: type 50  
  
 (ABOUT 10 SECONDS PASSED)
  
debug3: receive packet: type 52  
Authenticated to <hostname> ([<hostname>]:22) using "publickey".  
debug1: channel 0: new session [client-session] (inactive timeout: 0)  
debug3: ssh_session2_open: channel_new: 0 (tty)  
debug2: channel 0: send open
(OMITTED...)

[EthanHeilman]

@minstrike520 Thanks for this information, it is very helpful to understand this issue.

What is the latency from your server to your Authentik server?

If you run ping what do you get for latency?

ping {YOUR AUTHENTIK SERVER}

Also check the latency of the JWKS URI of your Authentik server. You can find the JWKS URI at the well-known URI for your authentik server, the well-known URI should be something like: https://{YOUR AUTHENTIK SERVER}/application/o/{APP SLUG)/.well-known/openid-configuration.

Then plug the JWKS URI into curl to measure the time it takes to resolve JWKS

time wget -o /dev/null -S https://YOUR AUTHENTIK JWKS URI

Tell me know what the time to wget the JWKS is.