Sporadic tracebacks when attempting to submit a job #675
Description
I've recently encountered the following traceback when attempting to submit jobs:
$ testflinger-cli submit -p aitken.yaml
Traceback (most recent call last):
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connectionpool.py", line 534, in _make_request
response = conn.getresponse()
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connection.py", line 516, in getresponse
httplib_response = super().getresponse()
File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
response.begin()
File "/usr/lib/python3.10/http/client.py", line 318, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.10/http/client.py", line 279, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.10/socket.py", line 705, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3.10/ssl.py", line 1303, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib/python3.10/ssl.py", line 1159, in read
return self._sslobj.read(len, buffer)
TimeoutError: The read operation timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/adapters.py", line 667, in send
resp = conn.urlopen(
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connectionpool.py", line 841, in urlopen
retries = retries.increment(
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/util/retry.py", line 474, in increment
raise reraise(type(error), error, _stacktrace)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/util/util.py", line 39, in reraise
raise value
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
response = self._make_request(
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connectionpool.py", line 536, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='testflinger.canonical.com', port=443): Read timed out. (read timeout=15)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/snap/testflinger-cli/201/bin/testflinger-cli", line 8, in <module>
sys.exit(cli())
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/__init__.py", line 55, in cli
tfcli.run()
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/__init__.py", line 180, in run
sys.exit(self.args.func())
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/__init__.py", line 538, in submit
self.check_online_agents_available(queue)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/__init__.py", line 574, in check_online_agents_available
agents = self.client.get_agents_on_queue(queue)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/client.py", line 294, in get_agents_on_queue
data = self.get(endpoint)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/testflinger_cli/client.py", line 59, in get
req = requests.get(uri, timeout=timeout, headers=headers)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/api.py", line 73, in get
return request("get", url, params=params, **kwargs)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/snap/testflinger-cli/201/lib/python3.10/site-packages/requests/adapters.py", line 713, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='testflinger.canonical.com', port=443): Read timed out. (read timeout=15)
I've seen this after resetting offline Testflinger agents, and now also after submitting a new job just after releasing a node that I'd been testing via a direct (non-Testflinger) MAAS deployment. The job runs normally after one (or, in the latest case, two) failures. In the specific case just shown, here's the aitken.yaml file I was using:
job_queue: aitken
provision_data:
distro: noble
reserve_data:
ssh_keys:
- lp:bladernr
- lp:rodsmith
timeout: 21600
My Testflinger snap information:
$ snap info testflinger-cli
name: testflinger-cli
summary: Testflinger CLI
publisher: Canonical Certification Team (ce-certification-qa)
store-url: https://snapcraft.io/testflinger-cli
contact: https://github.com/canonical/testflinger/issues
license: GPL-3.0-only
description: |
The testflinger-cli tool is used for interacting with the testflinger
server for submitting test jobs, checking status, getting results, and
streaming output.
commands:
- testflinger-cli
snap-id: O4E2kMeKVNOoXBW7r8IeiAHk2Un4UbfM
tracking: latest/stable
refresh-date: 14 days ago, at 17:43 CEST
channels:
latest/stable: 20250408 2025-04-10 (201) 6MB -
latest/candidate: 20250408 2025-04-10 (201) 6MB -
latest/beta: 20250428 2025-04-28 (205) 569kB -
latest/edge: 20250428 2025-04-28 (205) 569kB -
installed: 20250408 (201) 6MB -
The host on which I'm running testflinger-cli is running Ubuntu 25.04.