Skip to content

Parallelization#98

Merged
alex98247 merged 13 commits intoPySATL:mainfrom
quepum:main
Dec 22, 2025
Merged

Parallelization#98
alex98247 merged 13 commits intoPySATL:mainfrom
quepum:main

Conversation

@quepum
Copy link
Contributor

@quepum quepum commented Dec 17, 2025

Summary

Adding parallelization support to to speed up the execution of resource-intensive statistical experiments.

Solve the issue: #73

Quick changelog

  • Implemented runtime task scheduler Scheduler based on ProcessPoolExecutor
  • Introduced serializable task specification TaskSpec
  • Created universal worker universal_execute_task
  • Added buffered result saver BufferedSaver
  • Integrated parallel execution into all experiment types
  • Added parallel_workers parameter in experiment config
  • Added unit tests

What's new?

Now the experiment can be performed in parallel mode, which significantly reduces the total time it takes.

f1i3g3
f1i3g3 approved these changes Dec 17, 2025
View reviewed changes
Copy link
Contributor

@f1i3g3 f1i3g3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Мне нравится, ошибок пока не вижу, CI проходит.
Возможно, стоит ещё погонять на реальных данных.
Приглашаю остальных тоже глянуть.

pysatl_experiment/cli/commands/configure/parallel_workers/parallel_workers.py
Copy link
Contributor

@f1i3g3 f1i3g3 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Предлагаю поставить по умолчанию 1.

pysatl_experiment/cli/commands/create/create.py
Copy link
Contributor

@f1i3g3 f1i3g3 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

А, вижу :)

pysatl_experiment/factory/time_complexity/time_complexity.py
Copy link
Contributor

@f1i3g3 f1i3g3 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

А зачем тут строка для связи с БД?

pysatl_experiment/persistence/experiment/sqlite/sqlite.py
Copy link
Contributor

@f1i3g3 f1i3g3 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Надо будет остальным потом обновить для других PR.

tests/parallel/test_buffered_saver.py
Copy link
Contributor

@f1i3g3 f1i3g3 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Можно ещё тесты для очень большого числа потоков добавить (например, >100).

tests/parallel/test_scheduler.py
Copy link
Contributor

@f1i3g3 f1i3g3 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Можно ещё тесты для очень большого числа потоков добавить (например, >100).

tests/parallel/test_task_spec.py
Copy link
Contributor

@f1i3g3 f1i3g3 Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Можно ещё тесты для очень большого числа потоков добавить (например, >100).

@f1i3g3 f1i3g3 added enhancement New feature or request python Pull requests that update python code labels Dec 17, 2025
@alex98247 alex98247 merged commit 7cb5542 into PySATL:main Dec 22, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alex98247 alex98247 alex98247 approved these changes

@f1i3g3 f1i3g3 f1i3g3 approved these changes

@l-golofastov l-golofastov Awaiting requested review from l-golofastov

@faux66 faux66 Awaiting requested review from faux66

Assignees

@quepum quepum

Labels

enhancement New feature or request python Pull requests that update python code

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@quepum @alex98247 @f1i3g3