Skip to content

Airflow integration for data preparation #674

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
hasan7n wants to merge 181 commits into
base: main
Choose a base branch
from
Draft

Airflow integration for data preparation #674

hasan7n wants to merge 181 commits into from

Conversation

@hasan7n
Copy link
Contributor

@hasan7n hasan7n commented Jan 23, 2026

No description provided.

@RodriguesRBruno
separate airflow, synapse and base medperf requirements
e5de597
@RodriguesRBruno
Update dependencies in preparation for airflow
315863a
@RodriguesRBruno
Implement similar functionality to Pydantic v2 parse_datetime
9da3a3d
@RodriguesRBruno
Add explicit None default for Optionals, as required for Pydantic v2
04e7114
@RodriguesRBruno
Use default factories for default empty dicts
39eb3c1
@RodriguesRBruno
Use new implementation for parse_datetime
c7ce62b
@RodriguesRBruno
Add missing type hint required for pydantic v2
a314527
@RodriguesRBruno
General fixes for pydantic v2
38aedbd
@RodriguesRBruno
Update deprecated Pydantic functionality
6cf6ea7
@RodriguesRBruno
Add small tests config file to suppress yaspin color warnings
e9b622a
@RodriguesRBruno
Fix parse_datetime to support datetime objects directly
349d8b5
@RodriguesRBruno
Bump google-cloud-secret-manager version
f98b2fd
@RodriguesRBruno
Create DataPreparationWorkflow Object
afee7ff
@RodriguesRBruno
Add DataPrepWorkflow object to Dataset and Benchmark
6b2f855
@RodriguesRBruno
Migrate code from workflows repo into medperf
419dbec
@RodriguesRBruno
Standardize logging with Medperf
e304fbd
@RodriguesRBruno
Refactor code considering local airflow installation alongside MedPerf.
2c23d1b
@RodriguesRBruno
Convert mlcube_hash field from string do JSON (dict)
ee3b87c
@RodriguesRBruno
Update hashes on seed_utils.py
0c9add0
@RodriguesRBruno
Revert "Create DataPreparationWorkflow Object"
0e3f43d 
This reverts commit afee7ff.
@RodriguesRBruno
Revert "Add DataPrepWorkflow object to Dataset and Benchmark"
cbba26b 
This reverts commit 6b2f855.
@RodriguesRBruno
Revert "Bump google-cloud-secret-manager version"
30cde59 
This reverts commit f98b2fd.
@RodriguesRBruno
Reunify requirements using the synapse development build with better …
f5c32d7 
…psutil reqs
@RodriguesRBruno
Revert "Convert mlcube_hash field from string do JSON (dict)"
b4ea887 
This reverts commit ee3b87c.
@RodriguesRBruno
Revert "Update hashes on seed_utils.py"
180e977 
This reverts commit 0c9add0.
@RodriguesRBruno
Convert image_hash field to JSONField
766e5dd
@RodriguesRBruno
update server tests
637485a
@RodriguesRBruno
Update seed_utils.py to new image hash format
b2efff1
@RodriguesRBruno
Update client side code to expect hashes as dicts
186cc52
@RodriguesRBruno
Migrate airflow_runner files to subdirectory of container/runners
1519a5e
RodriguesRBruno and others added 28 commits December 9, 2025 13:34
@RodriguesRBruno
shorten imports for readability
dc10644
@RodriguesRBruno
Add auto-approval script to automatically do manual review for testing
b9e9a91
@RodriguesRBruno
refactoring
33af343
@RodriguesRBruno
Refactor download method so that Docker/singularity runner can take i…
0019709 
…n hashes as strings rather than dicts
@RodriguesRBruno
linting
3af7e5c
@RodriguesRBruno
Return hashes on AirflowRunner
abe808f
@RodriguesRBruno
Use "default" key always for hashes in simple containers
64ebf1d
@RodriguesRBruno
Refactor private model code paths into separate file for maintainability
63a60c5
@RodriguesRBruno
Simplify BearerAuth
2192f53
@RodriguesRBruno
Refactor download functionality for both Docker and Singularity so it…
48e21ec 
… can be reused for airflow
@RodriguesRBruno
Remove unused import
dfbc645
@RodriguesRBruno
fix import
958e97e
@RodriguesRBruno
Fix method call
6196f98
@RodriguesRBruno
Remove image_hash validator from Cube class. Convert string hashes in…
e8d5cb2 
…to dicts on SubmitCube.run
@RodriguesRBruno
linting
dd32afb
@RodriguesRBruno
Refactor AirflowClient. Make get_token public and allow it to be call…
fef44b3 
…ed without a base url
@RodriguesRBruno
Create auto-login plugin and make the link generated by SystemRunner …
240e6a5 
…point to that endpoint, so users can login with a single click
@RodriguesRBruno
Upgrade airflow version
c0c3ded
@RodriguesRBruno
Use self.port on healthcheck url
9273e2e
@RodriguesRBruno
Update auto-login plugin for airflow 3.1.15
5464aac
@RodriguesRBruno
linting
858d0e9
@RodriguesRBruno
Add small documentation about get_token_async
dff1aa3
@RodriguesRBruno
Update RANO README.md with auto_login
520aa67
@RodriguesRBruno
Modify instance name
824a7e1
@RodriguesRBruno
Fix errors in HEMnet README.md
916e91d
@RodriguesRBruno
Send correct email into airflow configuration
3db6eec
@RodriguesRBruno
Use the new ApprovalOperator for ManualApproval steps. Update RANO RE…
6e90053 
…ADME.md with this new implementation.
@hasan7n
Merge pull request #670 from RodriguesRBruno/add_airflow_support
c15d35e 
Add support for running Data Prep using Airflow
@hasan7n hasan7n requested a deployment to testing-external-code January 23, 2026 18:25 — with GitHub Actions Waiting
@github-actions
Copy link
Contributor

github-actions bot commented Jan 23, 2026

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@hasan7n @RodriguesRBruno