Prioritize solver already based on requirements

[harshad16]

Is your feature request related to a problem? Please describe.
Our current model executes the solvers for packages at the same time.When a new solver is included, it brings in a lot of solvers execution to be done, based on the number of packages yet to be solved, which creates a bottle neck for the old solver.

For example:
Lets say:

• pypi has 1million packages.
• Thoth-station has three solvers, for say:
  • ubi8-py38
  • f34-py39
  • f35-py310
    And each solver is already enabled for execution. Assuming that all the solver have solved up to 98% of packages.
    Now in this mix, if we include ubi9-py39, then it would fill up the pipelines as they are more package yet to be solved.
    The package-release-job, would still look for new packages to solved, however they would be in queue.
    The wait of each solver depends on one another.

If we have consumer requesting a specific solver, we only have the option to halt the rest by removing them from the mix
and wait for the specific solver be up-to-date.

The goal is to find a solution for solver prioritization.

Describe the solution you'd like
None as of now.

Describe alternatives you've considered
Halting other solvers for specific solver.

Additional context

• thoth-application/core/overlays/ocp4-stage/common/configmaps.yaml

  Line 37 in a13add7

  solvers: |

• https://docs.google.com/document/d/1jz4j07Q8gkv9H-23BgOp_4qN3uI29JsuP-_kTKEj-e8/edit#heading=h.geg4jwhxw684

/priority important-longterm
/sig devsecops

Acceptance criteria

• [spike] Explore the solution for this issue and create an ADR to discuss with the team #2694

[harshad16]

Context from the tech call:

As we are running multiple solvers at the same time, it takes time for newer packages to sync for all the solvers. any suggestion on the changing the priority of this action:

• select a set of solver that are most used (via configmap)
  • disable some solver(s)? concern: if we keep supporting them, we should keep solving
• priority in solver workflow
  • idea: configmap with : map ?
• changes in consumer logic, by having solver based topics and throttling of consumers?

Usage Statistic? Which (non integration tests) advises use what os+py?

How do we pick our focus?

• what runtime is used by RHODS/ODH: ex: ubi8-py38
• what runtime is release and maintained by Red Hat: ex: ubi9-py39
• based on out research (py311 due to perf incr.), see also https://www.phoronix.com/review/python-311-performance

Questions

• are there particular components we could investigate to see if we have low hanging perf improvements possible to have faster solver/sync ?
  solvers, graph-refresh-job, package-release-job and investigator.

[harshad16]

Resources:
https://www.confluent.io/blog/prioritize-messages-in-kafka/