feat: add exclude_agents field to job data to allow jobs to skip specific agents

[ajzobro]

Description

Added the ability to specify that select agents are not to be used to run a given job, even if they are in the specified queue.

Agents will now be required to POST to /agents/data/<agent_name> before they are able to GET a /job to work. The response message from registering the agent will return a cookie which will identify the agent on subsequent calls to fetch a job to work on.

Resolved issues

Resolves Jira CERTTF-697

Documentation

Examples and references data updated in docs/.

Web service API changes

Job data may now include exclude_agents at the top level, e.g. alongside job_queue, for example:

job_queue: my_queue
exclude_agents:
    - agent_007
    - agent_002
provision_data:
  distro: noble
reserve_data:
  ssh_keys:
    - lp:bond
  timeout: 3600

Added a cookie in the response to POST /agents/data/<agent_name> which is required to then GET a /job. This is done to ensure that agents getting work are who they say they are and without introducing any new agent code to do so. The requests library is already configured to respond with cookies it it given and the agents are already configured to register before they look for work.

exclude_agents was added to the database job schema in support of this.

When there are excluded agents, they will be considered when checking against the relevant queues upon job scheduling and assignment so that jobs with no agents will be prevented from being scheduled and agents will not be blocked from doing other work for which they are not excluded from running.

Tests

Tests were added to CLI and server to ensure new functionality is tested.
cd cli && uvx tox
cd server && uvx tox

[codecov]

Codecov Report

❌ Patch coverage is 96.82540% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.46%. Comparing base (e8f6add) to head (3714ee5).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #850      +/-   ##
==========================================
+ Coverage   71.31%   71.46%   +0.14%     
==========================================
  Files         108      108              
  Lines        9651     9707      +56     
  Branches      858      864       +6     
==========================================
+ Hits         6883     6937      +54     
- Misses       2595     2596       +1     
- Partials      173      174       +1     

Flag	Coverage Δ		*Carryforward flag
agent	72.85% <ø> (ø)		Carriedforward from b08c9b2
cli	87.98% <100.00%> (+0.19%)	⬆️
device	55.21% <ø> (ø)		Carriedforward from b08c9b2
server	87.71% <90.47%> (-0.01%)	⬇️

*This pull request uses carry forward flags. Click here to find out more.

Components	Coverage Δ
Agent	72.85% <ø> (ø)
CLI	87.98% <100.00%> (+0.19%)	⬆️
Common	∅ <ø> (∅)
Device Connectors	55.21% <ø> (ø)
Server	87.71% <90.47%> (-0.01%)	⬇️

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ajzobro]

This appeared to be a mistake and was getting reformatted by uvx ruff format

[ajzobro]

#852 has many of the same changes, but also changes webdav to WebDAV (appropriately)