Skip to content

Migrate from Rye to uv, modernize Python 3.12+, comprehensive refactor#37

Merged
minhtribk12 merged 5 commits intomainfrom
refactor/migrate-to-uv
Mar 25, 2026
Merged

Migrate from Rye to uv, modernize Python 3.12+, comprehensive refactor#37
minhtribk12 merged 5 commits intomainfrom
refactor/migrate-to-uv

Conversation

@minhtribk12
Copy link
Copy Markdown
Contributor

@minhtribk12 minhtribk12 commented Mar 25, 2026

  • Replace Rye with uv as project manager (PEP 735 dependency-groups)
  • Drop Python 3.9/3.10/3.11, target 3.12+ with 3.13 in CI matrix
  • Rewrite CI workflows: lint/typecheck/test with astral-sh/setup-uv
  • Add PyPI publish workflow with trusted publishers (OIDC)
  • Delete tox.ini, Rye lock files; generate uv.lock
  • Modernize all type hints: Optional -> X | None, Union -> X | Y, str+Enum -> StrEnum, remove from future import annotations
  • Fix critical bugs: pydantic v1 .json() -> .model_dump_json(), .dict() -> .model_dump(), lock context manager, raise RuntimeWarning -> warnings.warn, enum typos and duplicates, method name typos
  • Replace bare except Exception with specific types throughout
  • Replace traceback.print_exception with qoa_logger.exception
  • Unify logging: replace print()/logging.error() with qoa_logger
  • Remove dead code: unused vars, outdated examples, legacy test dir
  • Archive cain_version_outdated examples, delete legacy BTS files
  • Define public API in init.py, add py.typed marker (PEP 561)
  • Add comprehensive test suite: 197 tests covering config, lang, reports, utils, connectors (up from 5 tests)
  • Update .gitignore for build artifacts and tool caches
  • Update pre-commit config for Python 3.12

@minhtribk12
Migrate from Rye to uv, modernize Python 3.12+, comprehensive refactor
cb264ff 
- Replace Rye with uv as project manager (PEP 735 dependency-groups)
- Drop Python 3.9/3.10/3.11, target 3.12+ with 3.13 in CI matrix
- Rewrite CI workflows: lint/typecheck/test with astral-sh/setup-uv
- Add PyPI publish workflow with trusted publishers (OIDC)
- Delete tox.ini, Rye lock files; generate uv.lock
- Modernize all type hints: Optional -> X | None, Union -> X | Y,
  str+Enum -> StrEnum, remove from __future__ import annotations
- Fix critical bugs: pydantic v1 .json() -> .model_dump_json(),
  .dict() -> .model_dump(), lock context manager, raise RuntimeWarning
  -> warnings.warn, enum typos and duplicates, method name typos
- Replace bare except Exception with specific types throughout
- Replace traceback.print_exception with qoa_logger.exception
- Unify logging: replace print()/logging.error() with qoa_logger
- Remove dead code: unused vars, outdated examples, legacy test dir
- Archive cain_version_outdated examples, delete legacy BTS files
- Define public API in __init__.py, add py.typed marker (PEP 561)
- Add comprehensive test suite: 197 tests covering config, lang,
  reports, utils, connectors (up from 5 tests)
- Update .gitignore for build artifacts and tool caches
- Update pre-commit config for Python 3.12
@minhtribk12 minhtribk12 requested a review from Copilot March 25, 2026 06:54
Copilot AI reviewed Mar 25, 2026
View reviewed changes
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR modernizes the qoa4ml Python package by migrating project management/CI from Rye/tox to uv, raising the supported Python baseline to 3.12+, refactoring code for newer typing + Pydantic v2 APIs, and expanding test coverage while removing legacy/outdated example assets.

Changes:

  • Migrate tooling to uv (dependency-groups, new CI workflows, publishing via trusted publishers) and drop Python <3.12 support.
  • Refactor runtime code for Pydantic v2 serialization, modern typing syntax, and unified logging/exception handling.
  • Add a significantly expanded pytest suite and remove legacy/tox + outdated example directories/files.

Reviewed changes

Copilot reviewed 114 out of 127 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
tox.ini Removed tox-based test orchestration.
pyproject.toml Python 3.12+ targeting, uv dependency-groups, updated deps/dev deps.
uv.lock New uv lockfile replacing Rye lockfiles.
requirements.lock Removed Rye lockfile.
requirements-dev.lock Removed Rye dev lockfile.
.python-version Bumped local Python version to 3.12.
.pre-commit-config.yaml Updated default Python to 3.12 for hooks.
.gitignore Added common tool/build caches and artifact directories.
.github/workflows/python-ci.yml Replaced Rye/tox CI with uv-based lint/typecheck/test (3.12/3.13).
.github/workflows/publish.yml Added PyPI publish workflow using OIDC trusted publishing.
.github/workflows/gh-page-ci.yml Switched docs deploy workflow to uv install/run.
src/qoa4ml/init.py Defined public API exports and removed __future__ annotations import.
src/qoa4ml/py.typed Added marker for PEP 561 typing support.
src/qoa4ml/qoa_client.py Pydantic v2 serialization updates, typing modernization, locking refactor.
src/qoa4ml/config/configs.py Typing modernization, Kafka config typo fix (poll_interval), forward refs rebuild.
src/qoa4ml/lang/attributes.py Modernized enums to StrEnum, safer docstring assignment loop.
src/qoa4ml/lang/common_models.py Updated type hints to `X
src/qoa4ml/lang/datamodel_enum.py Migrated many enums to StrEnum, fixed typos/values, updated MetricNameEnum alias.
src/qoa4ml/reports/abstract_report.py Updated generate_report signature typing to `str
src/qoa4ml/reports/ml_reports.py Warning semantics via warnings.warn, updated typing, Pydantic v2 API usage.
src/qoa4ml/reports/ml_report_model.py Typing modernization and schema model updates.
src/qoa4ml/reports/resources_report_model.py Removed __future__ annotations import.
src/qoa4ml/reports/rohe_reports.py Updated typing (`str
src/qoa4ml/utils/qoa_utils.py Narrowed exception handling, Pydantic v2-leaning serialization/logging changes.
src/qoa4ml/utils/gpu_utils.py Catch NVML-specific exceptions and formatting cleanup.
src/qoa4ml/utils/gpu_utils.py Adjusted NVML error handling to NVMLError.
src/qoa4ml/probes/probe.py Removed __future__ annotations import.
src/qoa4ml/probes/process_monitoring_probe.py Removed __future__ annotations import.
src/qoa4ml/probes/system_monitoring_probe.py Removed __future__ annotations import.
src/qoa4ml/probes/mlquality.py Added type hints and narrowed exception handling; logging modernization.
src/qoa4ml/connector/amqp_connector.py Typing modernization and minor comment typo fix.
src/qoa4ml/connector/mqtt_connector.py Unified logger usage (qoa_logger) and removed unused attribute.
src/qoa4ml/connector/socket_connector.py Replaced logging.error with qoa_logger, typing modernization.
src/qoa4ml/connector/prom_connector.py Removed unused constant.
src/qoa4ml/collector/socket_collector.py Updated Callable import to collections.abc.
src/qoa4ml/collector/kafka_collector.py Fixed poll_interval usage and replaced print with structured logging.
src/qoa4ml/collector/amqp_collector.py Typing modernization for optional host object.
src/qoa4ml/observability/odop_obs/embedded_database.py Renamed “lastest” → “latest” timestamp method.
src/qoa4ml/observability/odop_obs/node_aggregator.py Renamed route handler, switched .dict() → .model_dump().
tests/conftest.py Added shared fixtures for config/model tests.
tests/test_config/test_configs.py Added comprehensive config model validation tests.
tests/test_config/init.py Test package marker.
tests/test_connector/test_debug_connector.py Added DebugConnector behavior tests.
tests/test_connector/init.py Test package marker.
tests/test_lang/test_attributes.py Added enum/meta behavior tests.
tests/test_lang/test_common_models.py Added Pydantic model validation/serialization tests.
tests/test_lang/test_datamodel_enum.py Added enum uniqueness + expected values tests.
tests/test_lang/init.py Test package marker.
tests/test_reports/test_general_report.py Added tests for general report behaviors via concrete subclass.
tests/test_reports/test_ml_reports.py Added tests for MLReport behaviors and state transitions.
tests/test_reports/init.py Test package marker.
tests/test_utils/test_qoa_utils.py Added utility tests for config IO, conversions, and helpers.
tests/test_utils/init.py Test package marker.
example/simple/general_ml.py Removed __future__ annotations import.
example/simple/ensemble_ml.py Removed __future__ annotations import.
example/simple/observation_demo/config/observationConfig.yaml Added observation demo config.
example/simple/observation_demo/rohe_agent.ipynb Added observation demo notebook.
example/docker_report/docker_probe.py Removed __future__ annotations import.
example/dataquality/basic_dataquality.py Removed __future__ annotations import.
example/bts/bts_server/predictionService/powerGrid/LSTM_Prediction.py Formatting-only string interpolation cleanup.
example/bts/bts_server/predictionServerV1.py Removed legacy BTS server implementation.
example/bts/bts_client/client_v0.0.3/lstm_client.py Removed legacy BTS client example.
example/bts/bts_client/client_v0.0.3/client.json Removed legacy BTS client config.
example/bts/bts_client/client_v0.0.3/LSTM_Prediction_Client.py Removed legacy BTS client implementation.
example/object_detection/cain_version_outdated/worker/serve.sh Removed outdated object detection worker artifact.
example/object_detection/cain_version_outdated/worker/build_docker.sh Removed outdated object detection worker artifact.
example/object_detection/cain_version_outdated/worker/Dockerfile Removed outdated object detection worker artifact.
example/object_detection/cain_version_outdated/source/test.py Removed outdated object detection source artifact.
example/object_detection/cain_version_outdated/source/serve.sh Removed outdated object detection source artifact.
example/object_detection/cain_version_outdated/source/requirement.txt Removed outdated object detection source artifact.
example/object_detection/cain_version_outdated/source/readme.md Removed outdated object detection source artifact.
example/object_detection/cain_version_outdated/source/deployment.py Removed outdated object detection source artifact.
example/object_detection/cain_version_outdated/source/conf.json Removed outdated object detection source artifact.
example/object_detection/cain_version_outdated/source/build_docker.sh Removed outdated object detection source artifact.
example/object_detection/cain_version_outdated/source/aggregation.py Removed outdated object detection source artifact.
example/object_detection/cain_version_outdated/source/Dockerfile Removed outdated object detection source artifact.
example/object_detection/cain_version_outdated/ray_stop.sh Removed outdated Ray deployment script.
example/object_detection/cain_version_outdated/ray_serve.sh Removed outdated Ray deployment script.
example/object_detection/cain_version_outdated/ray_cluster/ray_worker.yml Removed outdated Ray manifests.
example/object_detection/cain_version_outdated/ray_cluster/ray_head_light.yml Removed outdated Ray manifests.
example/object_detection/cain_version_outdated/kube_ray/ray-cluster.heterogeneous.yaml Removed outdated KubeRay manifest.
example/object_detection/cain_version_outdated/kube_ray/config/rbac/service_account.yaml Removed outdated KubeRay RBAC.
example/object_detection/cain_version_outdated/kube_ray/config/rbac/role_binding.yaml Removed outdated KubeRay RBAC.
example/object_detection/cain_version_outdated/kube_ray/config/rbac/role.yaml Removed outdated KubeRay RBAC.
example/object_detection/cain_version_outdated/kube_ray/config/rbac/ray_rayservice_viewer_role.yaml Removed outdated KubeRay RBAC.
example/object_detection/cain_version_outdated/kube_ray/config/rbac/ray_rayservice_editor_role.yaml Removed outdated KubeRay RBAC.
example/object_detection/cain_version_outdated/kube_ray/config/rbac/ray_rayjob_viewer_role.yaml Removed outdated KubeRay RBAC.
example/object_detection/cain_version_outdated/kube_ray/config/rbac/ray_rayjob_editor_role.yaml Removed outdated KubeRay RBAC.
example/object_detection/cain_version_outdated/kube_ray/config/rbac/leader_election_role_binding.yaml Removed outdated KubeRay RBAC.
example/object_detection/cain_version_outdated/kube_ray/config/rbac/leader_election_role.yaml Removed outdated KubeRay RBAC.
example/object_detection/cain_version_outdated/kube_ray/config/rbac/kustomization.yaml Removed outdated KubeRay RBAC kustomization.
example/object_detection/cain_version_outdated/kube_ray/config/prometheus/monitor.yaml Removed outdated KubeRay Prometheus config.
example/object_detection/cain_version_outdated/kube_ray/config/prometheus/kustomization.yaml Removed outdated KubeRay Prometheus config.
example/object_detection/cain_version_outdated/kube_ray/config/manager/service.yaml Removed outdated KubeRay manager config.
example/object_detection/cain_version_outdated/kube_ray/config/manager/manager.yaml Removed outdated KubeRay manager config.
example/object_detection/cain_version_outdated/kube_ray/config/manager/kustomization.yaml Removed outdated KubeRay manager config.
example/object_detection/cain_version_outdated/kube_ray/config/default/namespace.yaml Removed outdated KubeRay default config.
example/object_detection/cain_version_outdated/kube_ray/config/default/kustomization.yaml Removed outdated KubeRay default config.
example/object_detection/cain_version_outdated/kube_ray/config/crd/kustomizeconfig.yaml Removed outdated KubeRay CRD config.
example/object_detection/cain_version_outdated/kube_ray/config/crd/kustomization.yaml Removed outdated KubeRay CRD config.
example/object_detection/cain_version_outdated/contract/object_detection.rego Removed outdated contract example.
example/object_detection/cain_version_outdated/contract/contract.json Removed outdated contract example.
example/object_detection/cain_version_outdated/client/test_worker.py Removed outdated client example.
example/object_detection/cain_version_outdated/client/sim_instance.py Removed outdated client example.
example/object_detection/cain_version_outdated/client/deployment/client.yml Removed outdated client deployment manifest.
example/object_detection/cain_version_outdated/client/conf.json Removed outdated client config.
example/object_detection/cain_version_outdated/client/client.py Removed outdated client script.
example/object_detection/cain_version_outdated/client/build_docker.sh Removed outdated client docker script.
example/object_detection/cain_version_outdated/client/Dockerfile Removed outdated client Dockerfile.
example/object_detection/cain_version_outdated/Readme.md Removed outdated example readme.
example/malware_detection/cain_version_outdated/server/serve.sh Removed outdated malware detection server artifact.
example/malware_detection/cain_version_outdated/server/detector.py Removed outdated malware detection server artifact.
example/malware_detection/cain_version_outdated/server/composition.py Removed outdated malware detection server artifact.
example/malware_detection/cain_version_outdated/server/build_docker.sh Removed outdated malware detection server artifact.
example/malware_detection/cain_version_outdated/server/aggregation.py Removed outdated malware detection server artifact.
example/malware_detection/cain_version_outdated/server/Dockerfile Removed outdated malware detection server artifact.
example/malware_detection/cain_version_outdated/contract/malware_detection.rego Removed outdated malware contract example.
example/malware_detection/cain_version_outdated/contract/contract.json Removed outdated malware contract example.
example/malware_detection/cain_version_outdated/client/test_worker.py Removed outdated malware client example.
example/malware_detection/cain_version_outdated/client/sim_instance.py Removed outdated malware client example.
example/malware_detection/cain_version_outdated/client/deployment/client.yml Removed outdated malware client deployment manifest.
example/malware_detection/cain_version_outdated/client/conf.json Removed outdated malware client config.
example/malware_detection/cain_version_outdated/client/client.py Removed outdated malware client script.
example/malware_detection/cain_version_outdated/client/build_docker.sh Removed outdated malware client docker script.
example/malware_detection/cain_version_outdated/client/Dockerfile Removed outdated malware client Dockerfile.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

example/simple/observation_demo/rohe_agent.ipynb
Comment on lines +37 to +41
"observation_config = qutils.load_config(config_file)\n",
"print(f\"observation_config: {observation_config}\")\n",
"collector_config = AMQPCollectorConfig(observation_config[\"collector\"][\"config\"])\n",
"\n",
"observation_collector = AmqpCollector(collector_config)"
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This notebook currently embeds a failing execution (traceback in outputs) caused by instantiating AMQPCollectorConfig with a positional dict argument. Update the demo code to use keyword expansion/model validation (e.g., AMQPCollectorConfig(**...)) and clear the stored outputs/execution metadata so the committed example runs cleanly.

Copilot uses AI. Check for mistakes.
src/qoa4ml/qoa_client.py Outdated
Comment on lines +381 to +412
@@ -396,32 +393,31 @@ def asyn_report(self, body_mess: str, connectors: Optional[list] = None) -> None
-----
Uses threading to send reports asynchronously.
"""
self.lock.acquire()
if connectors is None:
if self.default_connector:
chosen_connector = self.connector_list[self.default_connector]
if isinstance(chosen_connector, AmqpConnector):
if not chosen_connector.check_connection():
chosen_connector.reconnect()
with self.lock:
if connectors is None:
if self.default_connector:
chosen_connector = self.connector_list[self.default_connector]
if isinstance(chosen_connector, AmqpConnector):
if not chosen_connector.check_connection():
chosen_connector.reconnect()

chosen_connector.send_report(body_mess, corr_id=str(uuid.uuid4()))
chosen_connector.send_report(
body_mess, corr_id=str(uuid.uuid4())
)
else:
chosen_connector.send_report(body_mess)
else:
chosen_connector.send_report(body_mess)
else:
qoa_logger.error(
"No default connector, please specify the connector to use"
)
else:
pass
self.lock.release()
qoa_logger.error(
"No default connector, please specify the connector to use"
)
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

asyn_report ignores the connectors argument when a non-None list is provided, so report(..., submit=True, connectors=[...]) will silently not send anything. Either implement sending through the provided connectors list (and validate elements) or remove the connectors parameter entirely to avoid misleading API behavior.

Copilot uses AI. Check for mistakes.
src/qoa4ml/reports/ml_report_model.py
Comment on lines 82 to 84
class GeneralMlInferenceReport(MlQualityReport, BaseReport):
ml_inference: dict[str, InferenceInstance] = {}

Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

GeneralMlInferenceReport.ml_inference is annotated as dict[str, InferenceInstance], but the report logic stores/looks up entries by UUID keys. This mismatch can break (de)serialization and type checking; the field should be keyed by UUID consistently (or normalize to str everywhere).

Copilot uses AI. Check for mistakes.
src/qoa4ml/observability/odop_obs/embedded_database.py
Comment on lines +19 to 22
def get_latest_timestamp(self):
time_query = TimeQuery()
timestamp = datetime.fromtimestamp(math.floor(time.time()))
return self.db.search(time_query >= timestamp)
Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

get_latest_timestamp() currently queries for datapoints with time >= floor(now), which will often return an empty set and does not match the method name. If the intent is “latest datapoint”, fetch the max timestamp record (or query <= now and take the last) instead of filtering to only the current second.

Copilot uses AI. Check for mistakes.
src/qoa4ml/probes/mlquality.py Outdated
Comment on lines 65 to 75
def training_metric(model: Any) -> dict[str, list[float]] | None:
"""Retrieve the full training history from a Keras Sequential model."""
try:
if isinstance(model, tf.keras.Sequential):
return model.history.history
else:
return None
except Exception as e:
qoa_logger.exception(f"Error {type(e)} when querying training metrics")
except (AttributeError, RuntimeError, TypeError) as e:
qoa_logger.exception("Error %s when querying training metrics", type(e))
return {"Error": "Unable to get training metrics"}

Copy link

Copilot AI Mar 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

training_metric() is annotated to return dict[str, list[float]] | None, but on exception it returns {"Error": "..."} (a dict[str, str]). This will fail mypy in CI; either broaden the return type (e.g., dict[str, Any] | None) or return None/raise on error.

Copilot uses AI. Check for mistakes.
@minhtribk12
Fix CI: add mypy config, mark docker/gpu tests, fix missing return
546cbb2 
- Add [tool.mypy] config to pyproject.toml with relaxed error codes
  for pre-existing type issues (to be tightened incrementally)
- Add [tool.pytest.ini_options] with requires_docker/requires_gpu markers
- Skip docker and gpu tests in CI (no Docker daemon or NVIDIA in runners)
- Fix missing return statement in load_config() for unsupported formats
- Fix gpu_test.py to be a proper pytest test function
@minhtribk12
Fix lint formatting and address PR review comments
2fc9a2a 
- Format test_attributes.py and test_datamodel_enum.py (CI ruff check)
- Implement connectors parameter in asyn_report() instead of silently
  ignoring non-None connector lists
- Fix training_metric() return type: use dict[str, Any] | None and
  return None on error instead of mismatched dict[str, str]
@minhtribk12
Format test files with ruff 0.15 (match CI version)
99307d9
@minhtribk12
Pin ruff to 0.15.7 in pre-commit and dev deps for consistency
1d5d9ea 
Aligns pre-commit ruff version with the uv-installed version used in
CI to prevent formatting conflicts between local and CI environments.
@minhtribk12 minhtribk12 merged commit 1781412 into main Mar 25, 2026
4 checks passed
This was referenced Mar 25, 2026
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@minhtribk12