SG-37222 new way to handle python third parties #996

eduardoChaucaGallegos · 2024-12-31T15:43:07Z

Description

This pull request introduces a major overhaul to how third-party Python dependencies are bundled, loaded, and managed in the project. It centralizes dependency management using versioned pkgs.zip archives, implements a dynamic import system for the tank_vendor namespace, and updates documentation to reflect the new workflow. Additionally, it removes legacy code and files that are no longer compatible with the new approach.

Key changes include:

Dependency Management Overhaul

Introduced a new system where dependencies for each supported Python version are bundled in a pkgs.zip archive, located in the requirements/<version>/ directories. This ensures consistent, reproducible environments and simplifies dependency updates.
Updated documentation in developer/README.md to describe the new dependency workflow, including how to update, freeze, and bundle packages, as well as how to perform CVE checks using frozen_requirements.txt. [1] [2]

Dynamic Import System for `tank_vendor`

Rewrote python/tank_vendor/__init__.py to automatically discover, load, and alias all packages from the appropriate pkgs.zip for the running Python version. It installs a custom import hook to support lazy loading and seamless aliasing for tank_vendor.* imports, including submodules.
Added logic to patch specific packages (like shotgun_api3) to work correctly when loaded from a ZIP archive, e.g., handling SSL certificate files.
Ensured backward compatibility for environments that do not use pkgs.zip by falling back to pip-installed dependencies and still supporting the tank_vendor namespace.

Codebase Cleanup and Compatibility

Removed the legacy upgrade_pyyaml.py script, as the new workflow supersedes the need for manual upgrades of individual dependencies.
Deleted legacy files from the vendored packaging library that are now managed via the new bundling system. (python/tank_vendor/packaging/__init__.py, python/tank_vendor/packaging/_structures.py) [1] [2]
Updated the import handler in python/tank/bootstrap/import_handler.py to correctly delegate loading of modules from ZIP files to the appropriate import mechanism, ensuring compatibility with the new system.

References:
[1] [2] [3] [4] [5] [6] [7]

Resolves #914

developer/requirements.txt

python/tank_vendor/__init__.py

requirements/3.10/frozen_requirements.txt

update_python_packages.py

requirements/3.11/requirements.txt

update_python_packages.py

codecov · 2025-02-04T21:48:39Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.79%. Comparing base (67de82a) to head (c92a72b).
Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #996   +/-   ##
=======================================
  Coverage   79.79%   79.79%           
=======================================
  Files         198      198           
  Lines       20770    20760   -10     
=======================================
- Hits        16573    16566    -7     
+ Misses       4197     4194    -3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

developer/requirements.txt

python/tank/platform/environment.py

requirements/3.11/frozen_requirements.txt

update_python_packages.py

developer/README.md

python/tank_vendor/__init__.py

…n afteer 'try' import blcok

carlos-villavicencio-adsk

LGTM. Good stuff!

triley13 · 2025-03-19T18:11:54Z

ETA on this?

julien-lang · 2025-03-26T17:13:34Z

Hi @triley13, sorry, no, we don't have an ETA. We had to postpone this PR because of new priorities. We still plan to merge it eventually but not in the next few weeks.

Is this something you need? Feel free to share how this change could help you.

julien-lang · 2025-07-08T14:27:40Z

Replaces #914

triley13 · 2025-07-23T14:41:10Z

Hi @triley13, sorry, no, we don't have an ETA. We had to postpone this PR because of new priorities. We still plan to merge it eventually but not in the next few weeks.

Is this something you need? Feel free to share how this change could help you.

Right now we use Rez to manage our internal libraries. This helps to keep our repo clean of duplicating libraries and implementing CICD to package the site-packages with our internal libs. With this, we can pull from our py registry (github), to manage our library injections.

…_init__.py to best behavior with third parties

… imports

… hooks

…port handle them

julien-lang · 2025-12-11T17:30:01Z

python/tank/bootstrap/import_handler.py

+        # ZIP import handler (like zipimport or TankVendorMetaFinder) handle it.
+        # This is common for tank_vendor packages that come from pkgs.zip.
+        if package_path[0] and ".zip" in package_path[0]:
+            return None


Suggested change

return None

return

julien-lang · 2025-12-17T17:06:18Z

requirements/3.7/requirements.txt

+packaging==24.0
+pyyaml==5.4.1
+ruamel.yaml==0.18.13
+shotgun_api3==3.9.0


Why is this one not 3.9.1 like the other ones?

julien-lang · 2025-12-17T17:07:47Z

update_python_packages.py

Can we have this script either in the requirements folder or the developer folder instead?

julien-lang · 2025-12-17T17:11:18Z

python/tank_vendor/__init__.py

+# If pkgs.zip is not found, assume pip-style installation where dependencies
+# are installed directly in the Python environment. In this case, we still
+# install the import hook to enable tank_vendor.* aliasing for compatibility.
+if not _pkgs_zip_valid:
+    # Install import hook even without pkgs.zip for pip installations
+    _install_import_hook()


I don't understand why we want to handle this case.

eduardoChaucaGallegos added 2 commits December 31, 2024 10:30

first version of new way to handle python third parties

15ad5c5

test requirements for test

c5de133

julien-lang requested a review from a team January 6, 2025 16:29

eduardoChaucaGallegos added 2 commits January 16, 2025 11:31

changes update_python_packages

49c8b01

testing python3.7 with old requirements

7c56bcf

carlos-villavicencio-adsk reviewed Jan 28, 2025

View reviewed changes

julien-lang reviewed Jan 28, 2025

View reviewed changes

requirements/3.11/requirements.txt Outdated Show resolved Hide resolved

update_python_packages.py Outdated Show resolved Hide resolved

test

a2b9253

eduardoChaucaGallegos marked this pull request as draft January 28, 2025 17:30

eduardoChaucaGallegos added 7 commits January 28, 2025 13:07

test

8ab95c1

test

94fee7f

test

4949fe0

possible solution}

92880d2

six.py recovered

1a5800a

ruamel_yaml explicit recovered

04c6435

ruamel_yaml explicit recovered

2e7a75f

eduardoChaucaGallegos added 6 commits February 4, 2025 20:21

comments fixed and still using six and ruamel_yaml from tank_vendor

c013a93

black format

1f68472

requirements updated

c724756

requirements format

db21f6d

requirements updated

e6b3e34

correct distro version for python3.11

d001f67

eduardoChaucaGallegos marked this pull request as ready for review February 5, 2025 16:49

eduardoChaucaGallegos requested review from carlos-villavicencio-adsk and julien-lang February 5, 2025 16:49

julien-lang requested changes Feb 5, 2025

View reviewed changes

eduardoChaucaGallegos added 3 commits February 6, 2025 00:37

comments fixed

ffea9ba

hound recomendation changed

161cd66

black format

1ada3b1

julien-lang reviewed Feb 18, 2025

View reviewed changes

update_python_packages.py Outdated Show resolved Hide resolved

developer/README.md Outdated Show resolved Hide resolved

developer/README.md Outdated Show resolved Hide resolved

python/tank_vendor/__init__.py Outdated Show resolved Hide resolved

eduardoChaucaGallegos added 2 commits February 20, 2025 09:44

readme documentation updated, single import line updated and exceptio…

cadb4b9

…n afteer 'try' import blcok

blank line

1d7c227

eduardoChaucaGallegos requested review from carlos-villavicencio-adsk and julien-lang and removed request for carlos-villavicencio-adsk February 20, 2025 14:56

julien-lang approved these changes Feb 20, 2025

View reviewed changes

carlos-villavicencio-adsk approved these changes Feb 20, 2025

View reviewed changes

eduardoChaucaGallegos added 15 commits December 5, 2025 23:54

conflict fixed, branch updated from master and fixes on tank_vendor/_…

e9669f4

…_init__.py to best behavior with third parties

black format

5ee3377

requirements updated

9f9a6f2

tank_vendor __init__.py improved

b531ad9

tank_vendor __init__.py improved

f46ea03

shotgu_api3 from pip

0d277a1

test to fix tests

080985d

improvemetn to read certifi

c92f375

Register shotgun_api3.lib submodules in sys.modules to support nested…

9a3b6f9

… imports

Refactor tank_vendor to use dynamic package discovery and lazy import…

1a318b1

… hooks

RunTimeError added and black fixes

500ce3e

fiexs for integrationsion tests

a5ab5fb

fix(import_handler): Skip ZIP paths in CoreImportHandler to let zipim…

b5d8e39

…port handle them

integration test fixed

20a6651

last pkgs.zip updated

e14033b

eduardoChaucaGallegos requested review from carlos-villavicencio-adsk and julien-lang December 11, 2025 16:59

julien-lang reviewed Dec 17, 2025

View reviewed changes

SG-37222 new way to handle python third parties #996

Are you sure you want to change the base?

SG-37222 new way to handle python third parties #996

Uh oh!

Conversation

eduardoChaucaGallegos commented Dec 31, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Dependency Management Overhaul

Dynamic Import System for tank_vendor

Codebase Cleanup and Compatibility

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Feb 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

carlos-villavicencio-adsk left a comment

Choose a reason for hiding this comment

Uh oh!

triley13 commented Mar 19, 2025

Uh oh!

julien-lang commented Mar 26, 2025

Uh oh!

julien-lang commented Jul 8, 2025

Uh oh!

triley13 commented Jul 23, 2025

Uh oh!

julien-lang Dec 11, 2025

Choose a reason for hiding this comment

Uh oh!

julien-lang Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

julien-lang Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

julien-lang Dec 17, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

eduardoChaucaGallegos commented Dec 31, 2024 •

edited

Loading

Dynamic Import System for `tank_vendor`

codecov bot commented Feb 4, 2025 •

edited

Loading