Skip to content

THRIFT-5537: Drop support for end-of-life python versions (including python 2)#2

Open
greggdonovan wants to merge 49 commits intomasterfrom
python_drop_eol
Open

THRIFT-5537: Drop support for end-of-life python versions (including python 2)#2
greggdonovan wants to merge 49 commits intomasterfrom
python_drop_eol

Conversation

@greggdonovan
Copy link
Owner

@greggdonovan greggdonovan commented Jan 31, 2026

  • If a ticket exists: Does your pull request title follow the pattern "THRIFT-NNNN: describe my issue"?

  • Did you squash your changes to a single commit? (not required, but preferred)
    Not yet - but will do after review. It's a large PR so I left separate commits to make it a bit easier to grok.

  • Did you do your best to avoid breaking changes? If one was needed, did you label the Jira ticket with "Breaking-Change"?

greggdonovan and others added 30 commits January 31, 2026 16:05
The system bison (2.3) doesn't support the %code directive.
Add homebrew's bison to GITHUB_PATH so it persists across all steps.

Co-Authored-By: Claude <noreply@anthropic.com>
greggdonovan and others added 12 commits January 31, 2026 16:05
This commit adds type checking validation for thrift-generated Python code
using Astral's ty type checker to ensure our Python 3.10+ type hints are
correct and complete.

New test infrastructure:
- type_check_test.thrift: Comprehensive IDL covering all thrift features
  (enums, typedefs, structs, unions, exceptions, services, constants)
- test_type_check.py: Test runner that auto-installs ty via uv, generates
  code, runs ty check, and validates imports/behavior
- py.typed marker generation in t_py_generator.cc for PEP 561 compliance

The test validates:
- ty check passes with zero errors on all generated code
- py.typed marker exists in generated packages
- All generated types are importable
- Enums are IntEnum subclasses with correct values
- Structs can be instantiated with type-correct arguments
- Exceptions inherit from TException

Also includes Python library modernization:
- Simplified SSL/TLS handling for Python 3.10+
- Removed Python 2 compatibility code
- Updated type annotations throughout

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Configure ty to find the thrift library by passing --extra-search-path
pointing to the build directory. This fixes the unresolved-import errors
in CI where ty couldn't locate the thrift module.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The enum, type_hints, and no_utf8strings options were removed from the
Python generator as they are now the default behavior for Python 3.10+.
This commit updates the test infrastructure to remove:

- gen-py-enum, gen-py-type_hints, gen-py-no_utf8strings test directories
- Corresponding generation rules in generate.cmake and Makefile.am
- References in RunClientServer.py genpydirs default

Also fixes test_ciphers SSL test to skip NULL cipher tests on Windows
where the SSL library handles invalid cipher specifications differently.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Make Iface class extend Protocol for proper abstract method support
- Use ellipsis (...) instead of pass for interface stub methods
- Add generated code directory to ty search paths for relative imports
- Add installed package location detection for CI environments

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
For frozen structs/exceptions, type checkers need to see attributes
declared at class level to understand they exist when set via
super().__setattr__() in __init__.

This fixes ty errors like:
- unresolved-attribute: Object of type `Self@__hash__` has no attribute `message`
- invalid-parameter-default: Default value of type `None` is not assignable

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
All __init__ parameters and instance attributes now consistently use
`| None` type hints since:
1. All parameters default to None for backward compatibility
2. Validation of required fields happens at runtime in validate()

This fixes ty errors:
- invalid-parameter-default: Default value of type `None` is not assignable to annotated parameter type `str`
- invalid-assignment: Cannot assign `str | None` to `str`

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
greggdonovan and others added 3 commits January 31, 2026 16:18
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Replace pip install uv with the official GitHub Action which is faster
(prebuilt binaries) and properly maintained by the uv team.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The NULL cipher mismatch test was failing on some platforms because
ssl_version only sets the minimum TLS version, allowing TLS 1.3 to
negotiate. TLS 1.3 has its own cipher suites that aren't affected by
set_ciphers('NULL'), causing the connection to succeed when it should
fail.

Fix by using ssl_context with both minimum_version and maximum_version
set to TLS 1.2, ensuring TLS 1.3 cannot bypass the cipher check.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@greggdonovan greggdonovan changed the title Python drop eol THRIFT-5537: Drop support for end-of-life python versions (including python 2) Jan 31, 2026
greggdonovan and others added 2 commits January 31, 2026 17:39
The test_normalconnection test was flaky on slower CI runners because
it only waited 2 seconds for the server thread to exit after calling
close_server(). Increased the timeout to 10 seconds to allow more time
for server shutdown on slower systems.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The test was calling stop() and close() in quick succession, but
close() destroys the socket pair that wake_up() uses to signal the
server to exit from select(). This race condition caused the server
thread to hang on macOS CI runners.

Fix by calling stop() first, waiting for the server thread to exit,
and only then calling close() to release resources.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Repository owner deleted a comment from greptile-apps bot Feb 1, 2026
@greggdonovan
Copy link
Owner Author

@greptile

@greptile-apps
Copy link

greptile-apps bot commented Feb 1, 2026

Greptile Overview

Greptile Summary

This PR successfully removes Python 2 support and modernizes the Thrift Python library to require Python 3.10+. The changes span 93 files with comprehensive updates across build configuration, code generation, runtime library, and tests.

Key Changes:

  • Updated minimum Python version to 3.10+ in setup.py, configure.ac, and documentation
  • Removed deprecated generator options (py:old_style, py:new_style, py:utf8strings, py:enum) - all now default behavior
  • Enforced IntEnum for all generated enum types with graceful unknown value handling
  • Added comprehensive type hints to generated code and created py.typed marker for PEP 561
  • Simplified SSL/TLS implementation: enforced TLS 1.2+ minimum, removed legacy Python 2.6/2.7 compatibility code
  • Removed backports.ssl_match_hostname dependency - now using native OpenSSL hostname verification
  • Cleaned up C extension module to remove Python 2 C API code (PyInt_AsLongPyLong_AsLong, etc.)
  • Unified build system: removed separate py3 build target, consolidated all Python testing under py
  • Added comprehensive type checking tests using Astral's ty type checker
  • Fixed test stability issues (server shutdown sequences, timeouts, socket cleanup)
  • Removed 280+ known test failures that were specific to the now-removed py3 target

Testing & Quality:

  • Added new test files: test_type_check.py, test_sslcontext_hostname.py, type_check_test.thrift
  • Enhanced SSL tests with TLS version validation, deprecated protocol rejection, and expired certificate tests
  • Improved server test cleanup to prevent resource leaks
  • Multiple commits show iterative refinement and test stabilization

Breaking Changes:
This is explicitly a breaking change (THRIFT-5537) that drops support for Python < 3.10. Users on older Python versions must remain on previous Thrift releases.

Confidence Score: 5/5

  • This PR is safe to merge - it's a well-executed modernization with comprehensive test coverage
  • The changes are systematic and thorough across all affected areas. The PR includes 47 commits showing iterative development and test fixes. New type checking infrastructure validates generated code quality. Test failures were resolved through multiple refinement commits. The scope is appropriate for dropping Python 2 support.
  • No files require special attention - the changes are consistent and well-tested

Important Files Changed

Filename Overview
lib/py/setup.py Updated Python version requirements to 3.10+, removed Python 2 backport dependencies
configure.ac Removed py3 separate build target, unified on Python 3.10+ requirement
lib/py/src/transport/TSSLSocket.py Enforced TLS 1.2+ minimum, removed Python 2.6/2.7 compatibility code, modernized SSL context handling
lib/py/src/transport/sslcompat.py Simplified to TLS version enforcement utilities, removed legacy hostname validation
compiler/cpp/src/thrift/generate/t_py_generator.cc Removed py2/py3 generator options, enforced IntEnum, added type hints, improved enum handling
lib/py/src/ext/module.cpp Removed Python 2 C API code, simplified to Python 3 only
lib/py/test/test_sslsocket.py Added comprehensive TLS tests, enforced TLS 1.2+, improved test stability
lib/py/test/thrift_TNonblockingServer.py Fixed test shutdown sequence, added timeouts, improved cleanup ordering

Sequence Diagram

sequenceDiagram
    participant Dev as Developer
    participant Build as Build System
    participant Compiler as Thrift Compiler
    participant PyLib as Python Library
    participant Tests as Test Suite
    
    Note over Dev,Tests: Python 2 Removal Process
    
    Dev->>Build: Update configure.ac (Python 3.10+)
    Dev->>Build: Update setup.py (python_requires='>=3.10')
    
    Dev->>Compiler: Remove py:old_style, py:new_style options
    Dev->>Compiler: Enforce IntEnum for all enums
    Dev->>Compiler: Add type hints to generated code
    
    Dev->>PyLib: Remove Python 2 C API code
    Dev->>PyLib: Simplify SSL/TLS (enforce TLS 1.2+)
    Dev->>PyLib: Remove backports.ssl_match_hostname
    Dev->>PyLib: Use modern SSLContext APIs
    
    Dev->>Tests: Add comprehensive type checking tests
    Dev->>Tests: Update SSL tests for TLS 1.2+
    Dev->>Tests: Fix server shutdown sequences
    Dev->>Tests: Remove 280+ py3-specific test entries
    
    Build->>Compiler: Generate Python code
    Compiler-->>Build: Modern Python 3.10+ code with type hints
    
    Build->>Tests: Run test suite
    Tests->>PyLib: Validate TLS 1.2+ enforcement
    Tests->>PyLib: Validate type hints with ty
    Tests-->>Build: All tests pass
    
    Build-->>Dev: Python 2 successfully removed
Loading

Copy link

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

8 files reviewed, no comments

Edit Code Review Agent Settings | Greptile

greggdonovan and others added 2 commits February 5, 2026 22:26
- Fix TSocket.py: use b'' instead of '' for empty bytes on ECONNRESET
- Fix TJSONProtocol.py: correct 'senf' typo to 'self' in 4 properties
- Fix t_py_generator.cc: update Python keywords (remove exec/print,
  add async/await) for Python 3.10+
- Fix TSSLSocket.py: enable hostname verification for CERT_OPTIONAL
- Fix debian/rules: remove Python 2 pyversions, use python3 directly
- Fix Vagrantfile: upgrade Ubuntu 14.04 to 22.04, JDK 8 to 17
- Fix util.py: raise ImportError instead of returning None
- Fix TNonblockingServer.py: move poll registration to prepare(),
  use elif for mutually exclusive connection states

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The code generator now allows __traceback__, __context__, __cause__,
and __suppress_context__ on immutable exceptions (not just slots).
Update the test to verify settability instead of checking slots.
Also includes timeout improvements for test stability.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant