refactor: Replace string-based import parsing with AST-based detection

[StefanoFioravanzo]

Summary

• Introduce a new imports.py module with proper AST-based import parsing
• Replace brittle hardcoded string matching in the compiler
• Add comprehensive test coverage (70 unit tests)

Problem

When Kale compiles a notebook into a Kubeflow Pipeline, it needs to determine which Python packages should be installed in the pipeline steps. This is done by analyzing the import statements in the notebook cells.

The previous implementation used simple string matching to parse imports:

kale/backend/kale/compiler.py

Lines 279 to 298 in 5c4a324

	for line in lines:
	line = line.strip()
	if line.startswith("import "):
	# For 'import package' or 'import package as alias'
	parts = line.split(" ")
	if len(parts) > 1:
	package_name = parts[1].split(".")[0]
	if package_name == "random":
	package_name = "random2"
	if package_name == "sklearn":
	package_name = "scikit-learn"
	package_names.add(package_name)
	elif line.startswith("from "):
	parts = line.split(" ")
	if len(parts) > 1:
	package_name = parts[1].split(".")[0]
	if package_name == "sklearn":
	package_name = "scikit-learn"
	package_names.add(package_name)
	return sorted(package_names)

This approach had several issues:

1. Hardcoded mappings: Only random and sklearn were mapped to their PyPI names. Other common mismatches (e.g., PIL → pillow, cv2 → opencv-python, yaml → pyyaml) were not handled.
2. No stdlib filtering: Standard library modules like os, sys, json, etc. were being added to the package list unnecessarily.
3. Fragile parsing: String splitting doesn't handle all valid Python import syntaxes:

• import a, b, c (multiple imports)
• from x import (a, b, c) (parenthesized imports)
• Multi-line imports
• Comments on import lines

4. Not extensible: Adding new package mappings required modifying the compiler code directly.

Solution

This PR introduces a new backend/kale/common/imports.py module that uses Python's ast module to properly parse imports:

Key Components

• STDLIB_MODULES: A comprehensive set of Python 3.12+ standard library modules to exclude from package requirements.
• PACKAGE_NAME_MAP: A centralized registry mapping import names to PyPI package names:

PACKAGE_NAME_MAP = {
    "sklearn": "scikit-learn",
    "cv2": "opencv-python",
    "PIL": "pillow",
    "yaml": "pyyaml",
    # ... and more
}

• ImportInfo: A dataclass providing structured information about each import (module, names, alias, line number).
• parse_imports_ast(): Parses all imports from source code using AST, handling all valid Python import syntaxes.
• get_packages_to_install(): Returns the set of PyPI packages needed for given source code, filtering stdlib and applying name mappings.

Benefits

• Correct parsing: Handles all valid Python import syntax
• Extensible: New package mappings can be added to PACKAGE_NAME_MAP
• Maintainable: Centralized configuration instead of scattered conditionals
• Tested: 70 unit tests covering all import forms and edge cases

Changes

• New module: backend/kale/common/imports.py
• Refactored: backend/kale/compiler.py - Use new module instead of string matching
• Tests: backend/kale/tests/unit_tests/test_imports.py - 70 comprehensive tests

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from stefanofioravanzo. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[FAUST-BENCHOU]

so many lint error
uv run ruff check backend --fix may fix them?