Skip to content

Add JSON type support for SQLAlchemy dialect#620

Merged
laughingman7743 merged 1 commit intomasterfrom
feature/json-type-support
Nov 8, 2025
Merged

Add JSON type support for SQLAlchemy dialect#620
laughingman7743 merged 1 commit intomasterfrom
feature/json-type-support

Conversation

@laughingman7743
Copy link
Member

Summary

This PR implements JSON type support for the PyAthena SQLAlchemy dialect to resolve Issue #619.

Changes

  • Add visit_JSON method to AthenaTypeCompiler in pyathena/sqlalchemy/compiler.py
  • Update ischema_names mapping in pyathena/sqlalchemy/base.py to use types.JSON instead of types.String
  • Add unit test in tests/pyathena/sqlalchemy/test_compiler.py to verify JSON type compilation
  • Add integration test in tests/pyathena/sqlalchemy/test_base.py to test JSON type with CAST operations
  • Update documentation in docs/sqlalchemy.rst:
    • Add JSON Type Support section with usage examples and limitations
    • Reorganize documentation structure (move Query Execution Callback section before Complex Data Types)

Key Features

  • Automatic JSON parsing: JSON strings are automatically converted to Python dictionaries
  • Supports nested JSON structures with arrays and objects
  • Compatible with all PyAthena SQLAlchemy dialects (rest, pandas, arrow)

Athena JSON Type Limitations

  • JSON objects are fully supported - Objects with key-value pairs work correctly
  • Top-level JSON arrays are NOT supported - Direct CAST of arrays like [1, 2, 3] will fail with InvalidRequestException
  • Arrays within objects are supported - JSON objects can contain arrays as property values
  • ℹ️ DML only - JSON type is supported for SELECT queries but not in CREATE TABLE statements

Test Results

All tests pass successfully:

make chk  # All checks passed
uv run pytest tests/pyathena/sqlalchemy/test_base.py::TestBase::test_json_type_with_cast -v  # PASSED
uv run pytest tests/pyathena/sqlalchemy/test_compiler.py::TestCompiler::test_visit_json -v  # PASSED

Example Usage

from sqlalchemy import select, literal_column
from sqlalchemy.sql import type_coerce
from sqlalchemy.types import JSON

# Query with explicit type coercion
result = connection.execute(
    select(
        type_coerce(
            literal_column('CAST(\'{"name": "test", "value": 123}\' AS JSON)'),
            JSON
        ).label("json_col")
    )
).fetchone()

# Result is automatically parsed as a dictionary
print(result.json_col)  # {"name": "test", "value": 123}

Fixes #619

🤖 Generated with Claude Code

Implement JSON type support to resolve GitHub Issue #619.
This enables proper handling of JSON columns when reflecting
Iceberg tables and querying JSON data with SQLAlchemy.

Changes:
- Add visit_JSON method to AthenaTypeCompiler
- Update ischema_names mapping from String to JSON type
- Add unit test for JSON type compilation
- Add integration test for JSON type with CAST operations
- Update documentation with JSON Type Support section
- Reorganize documentation structure (move Query Execution Callback section)

Limitations:
- Athena only supports JSON objects, not top-level arrays
- JSON type is for DML operations (SELECT), not DDL (CREATE TABLE)

Fixes #619

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@laughingman7743 laughingman7743 marked this pull request as ready for review November 8, 2025 14:14
@laughingman7743 laughingman7743 merged commit 0d2671c into master Nov 8, 2025
5 checks passed
@laughingman7743 laughingman7743 deleted the feature/json-type-support branch November 8, 2025 14:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSON type support

1 participant