First pass of draft post on typing conformance of ty and pyrefly

content/future-python-type-checkers.md

@@ -0,0 +1,122 @@
++++
+title = "Ty versus Pyrefly versus Zuban: Reviewing the future of Python type checkers (Draft)"
+date = 2025-08-29
+description = "A comparison of three new Rust-based Python type checkers: Astral's ty, Meta's pyrefly, and David Halter's zuban"
+
+[taxonomies]
+tags = ["rust", "python", "type-checkers", "typing"]
++++
+
+# Table of Contents
+
+- [Introduction](#introduction)
+    - [`ty` from Astral](#ty-from-astral)
+    - [`pyrefly` from Meta](#pyrefly-from-meta)
+    - [`zuban` from David Halter](#zuban-from-david-halter)
+- [Typing Conformance Suite Analysis](#typing-conformance-suite-analysis)
+    - [Summary](#summary)
+    - [Review of progress](#review-of-progress)
+- [Other resources to learn more](#other-resources-to-learn-more)
+- [Footnotes](#footnotes)
+
+# Introduction
+
+The Python type checking landscape is experiencing a particularly active phase of innovation in 2025. This year has witnessed the emergence of not one, not two, but **three** new Python type checking tools, each backed by significant pedigree and resources. While these tools have slightly different goals and philosophies, they share a common foundation: all are built in Rust with performance as a core design principle.
+
+This continued shift toward Rust-based implementations represents a significant evolution in the Python tooling ecosystem, promising faster type checking, better IDE integration, and improved developer experience for large codebases.
+
+
+## `ty` from Astral
+
+**Repository:** <https://github.com/astral-sh/ty/>  
+**Development Repository:** <https://github.com/astral-sh/ruff/> 
+
+**Key Highlights:**
+- Strong focus on gradual guarantee principles
+- Planned tight integration with Astral's existing linting tool `ruff`, with the aim to support type based linting rules.[^1]
+- Backing from a team combining Python core developers and very experienced Rust tooling developers
+- Incremental computation at the function level leveraging the [salsa](https://github.com/salsa-rs/salsa) library used by [rust-analyzer](https://github.com/rust-lang/rust-analyzer/). This could be particularly important on the LSP that `ty` provides to give very quick/responsive feedback loops in code editors.
+
+**Philosophy:** Astral's approach emphasizes reliability and gradual adoption, making it easier for teams to incrementally add type checking to existing codebases without overwhelming developers with false positives.
+
+## `pyrefly` from Meta
+
+**Repository:** <https://github.com/facebook/pyrefly>  
+
+**Key Highlights:**
+- Successor to Meta's existing [pyre](https://github.com/facebook/pyre-check) type checker, designed to eventually replace it
+- Enhanced type inference capabilities by default
+- Potentially higher upfront adoption cost due to aggressive inference which might flag issues with correct code
+- Backed by Meta's substantial engineering resources and real-world usage at scale
+
+**Philosophy:** Meta prioritizes powerful inference and catching more potential issues out of the box, even if this means a steeper learning curve for teams new to type checking.
+
+## `zuban` from David Halter
+
+**Homepage:** <https://zubanls.com/>  
+**Documentation:** <https://docs.zubanls.com/en/latest/>
+
+**Key Highlights:**
+- Created by the author of the popular Python LSP tool `jedi`
+- Aims for high-degree of compatibility with `mypy` to make adoption in large existing codebases seamless.
+- Not FOSS[^2], will require a license for codebases above 1.5 MB (~50,000 lines of code)[^3].
+- Currently maintained by a single author seems a potential risk to long-term sustainability as Python typing does not stand still.
+
+**Philosophy:** Zuban aims to provide the smoothest possible migration path from existing type checkers, particularly `mypy`, making it attractive for organizations with substantial existing typed codebases.
+
+
+# Typing Conformance Suite Analysis
+
+The Python Typing Council maintains a [Conformance test suite](https://github.com/python/typing/tree/main/conformance) which validates the behavior of static type checkers against the expectations defined in the [Python typing specification](https://typing.python.org/en/latest/spec/index.html).
+
+`ty` and `pyrefly` have not yet been added to the conformance suite, so it's harder to establish a baseline for their progress on this front.
+
+To help resolve this gap I have [expanded the current test harness to support both of them](https://github.com/sinon/typing/pull/1)[^4], this also adds some additional debugging information to the html report to show the split between false negatives and false positives that the suite has detected in each test case.
+
+I've also included a local build of `ty` for two main reasons:
+
+* I've wanted to investigate contributing to `ty` and/or `ruff` for a while, so this was good impetus to get things set up.
+* `ty` releases are cut relatively infrequently and I am impatient.
+
+## Summary
+
+__Generated 29/08/2025__
+
+> NOTE
+>
+> The following section is slightly unfair, all of these tools are in alpha and only one of these tools (Zuban) has opted into the Conformance suite.
+>
+> That being said even though `ty` is lagging on this metric at the moment it is still the type checker that I am most excited to use long-term because of the quality of the tooling Astral has built so far.
+
+|                   Type Checker                  | Total Test Case Passes | Total Test Case Partial | Total False Positives | Total False Negatives | False Negatives |
+|:-----------------------------------------------:|:----------------------:|:-----------------------:|:---------------------:|:---------------------:|-----------------|
+|                   zuban 0.0.20                  | 97                     | 42                      | 152                   | 89                    | 92              |
+|     ty 0.0.1-alpha.19 (e9cb838b3 2025-08-19)    | 20                     | 119                     | 371                   | 603                   | 603             |
+| Local:ty ruff/0.12.11+27 (0bf5d2a20 2025-08-29) | 20                     | 119                     | 370                   | 590                   | 590             |
+|                  pyrefly 0.30.0                 | 81                     | 58                      | 100                   | 187                   | 187             |
+
+## Review of progress
+
+`zuban` has a lead, having full `Pass` on ~69% of test cases, compared with ~15% for `ty` and ~58% for `pyrefly`. Which makes sense as though it's released in a similar time period to `ty` and `pyrefly` it has been in active development in private for several years.
+
+The thing that surprised me more was how much progress `pyrefly` has made when compared to `ty`. This can maybe be partially explained from a point raised in [Edward Li's blog excellent post on the Typing Summit at PyCon 2025][1] which mentions that the `pyrefly` team devoted a lot of up front time into solving some of the hard problems, such as generics.
+
+
+# Other resources to learn more
+
+* [Edward Li's excellent blog post comparing `ty` and `pyrefly`][1] which also contains the videos recorded at the PyCon typing summit which both `ty` and `pyrefly` gave presentations at.
+* [Happy Path Programming - 114 ty: Fast Python Type Checking with Carl Meyer](https://www.youtube.com/watch?v=V1OmqEYoSz4)
+* [Happy Path Programming - 115 More Python Type Checking! Pyrefly with Aaron Pollack & Steven Troxler](https://www.youtube.com/watch?v=huHF0Rv8L14)
+* [Talk Python - ty: Astral's New Type Checker (Formerly Red-Knot)](https://talkpython.fm/episodes/show/506/ty-astrals-new-type-checker-formerly-red-knot)
+
+
+# Footnotes
+
+[^1]: Which can be demonstrated in the [open issues](https://github.com/astral-sh/ruff/issues?q=is%3Aissue%20state%3Aopen%20label%3Atype-inference ) on ruff tagged with `type-inference` which are bugs or new features that can only be resolved with `ruff` having access to deeper type inference data that `ty` can supply. 
+[^2]: David has indicated a plan to make [source available in the future](https://github.com/python/typing/pull/2067#issuecomment-3177937964) when adding Zuban to the Python typing conformance suite.
+[^3]: Full pricing information at: <https://zubanls.com/pricing/>
+[^4]: This is just for this blog post, no plans to seek merging this.
+
+
+<!-- Reference links --->
+[1]: https://blog.edward-li.com/tech/comparing-pyrefly-vs-ty/

content/pages/_index.md

@@ -1,2 +1,3 @@
 +++
+render = false
 +++