SG-37222 new way to handle python third parties

developer/README.md

@@ -1,19 +1,31 @@
 # Flow Production Tracking Core API
 
-## How to upgrade pyyaml
+## The `requirements` folder
 
-This package is a vendor widely used by `sgtk` to parse configuration files
-in YAML format.
+The `requirements` folder contains subdirectories for different Python versions (e.g., `3.7`, `3.9`, `3.10`, and `3.11`). Each subdirectory includes the following files:
 
-This package is shipped in source format, that means that only `*.py` are
-included in `python/tank_vendor/yaml`.
+- **`requirements.txt`**: Specifies the dependencies for the corresponding Python version. This file is primarily used to document which packages are required for the application.
+- **`frozen_requirements.txt`**: A frozen version of the dependencies, capturing exact package versions installed, including sub-dependencies, to ensure consistent and reproducible environments.
+- **`pkgs.zip`**: A zip file containing the bundled packages for the corresponding Python version.
 
-If you need to upgrade this package you can use the script `upgrade_pyyaml.py`.
+### How bundled packages are used
 
-```shell
-cd tk-core/developer
-python upgrade_pyyaml.py
-```
+The `__init__.py` file in the `tank_vendor` folder dynamically references and loads packages from the appropriate `pkgs.zip` file in the `requirements` folder.
+
+This approach centralizes the management of dependencies, ensuring that packages are versioned and bundled consistently across different Python versions.
+
+### Updating and creating bundled packages
+
+The `update_python_packages.py` script automates the creation and maintenance of the `pkgs.zip` file.
+
+#### Workflow:
+
+1. Update the `requirements.txt` file for the desired Python version.
+2. Run the `update_python_packages.py` script to:
+   - Install the specified dependencies in a temporary directory.
+   - Create or update the `pkgs.zip` file with the required packages.
+   - Generate the `frozen_requirements.txt` file for consistency.
+3. Validate that the `pkgs.zip` file contains all necessary packages and matches the updated requirements.
 
 ## How to upgrade ruamel.yaml
 
@@ -29,11 +41,19 @@ pip install ruamel.yaml -t path/to/tank_vendor
 Then, let's remove all undesired directories and files, just leave the `ruamel` directory.
 We can automate this task later.
 
-## The requirements.txt file
+### Maintaining dependencies
+
+When adding new dependencies or updating existing ones:
+1. Update the `requirements.txt` file for the corresponding Python version.
+2. Regenerate the `pkgs.zip` and `frozen_requirements.txt` files using `update_python_packages.py`.
+3. Ensure the `pkgs.zip` file includes all necessary packages and modules.
+
+### Automated CVE checks
+
+The `frozen_requirements.txt` files enable automated checks for vulnerabilities (CVEs) in the bundled packages. These files capture the exact versions of dependencies included in the `pkgs.zip` files, ensuring the application remains secure by providing visibility into potential vulnerabilities.
+
+### Notes
 
-The file `developer/requirements.txt` is not used to install any packages,
-however exists so that automated checks for CVEs in dependencies will know about
-bundled packages in `python/tank_vendor`.
+The dynamic loading mechanism in `tank_vendor/__init__.py` ensures that bundled packages are accessed seamlessly from the `pkgs.zip` files, reducing duplication and simplifying dependency updates.
 
-For this reason, it's important to add any newly bundled packages to this file,
-and to keep the file up to date if the bundled version of a module changes.
+Careful attention to package structure and appropriate import mechanisms will help avoid runtime issues and ensure smooth integration of new dependencies.

python/tank/bootstrap/import_handler.py

@@ -275,6 +275,13 @@ def find_spec(self, module_fullname, package_path=None, target=None):
 
         module_name = module_path_parts.pop()
 
+        # Check if the package path is inside a ZIP file.
+        # If so, SourceFileLoader cannot handle it - we need to let the
+        # ZIP import handler (like zipimport or TankVendorMetaFinder) handle it.
+        # This is common for tank_vendor packages that come from pkgs.zip.
+        if package_path[0] and ".zip" in package_path[0]:
+            return None
+
         try:
             # find the module spec
             if os.path.isdir(os.path.join(package_path[0], module_name)):