Skip to content

Comments

Add async SQLAlchemy dialects for native asyncio support#679

Merged
laughingman7743 merged 15 commits intomasterfrom
feature/673-async-sqlalchemy-dialect
Feb 21, 2026
Merged

Add async SQLAlchemy dialects for native asyncio support#679
laughingman7743 merged 15 commits intomasterfrom
feature/673-async-sqlalchemy-dialect

Conversation

@laughingman7743
Copy link
Member

@laughingman7743 laughingman7743 commented Feb 21, 2026

Summary

Closes #673.

Add async SQLAlchemy dialects enabling create_async_engine usage with PyAthena for frameworks like FastAPI.

Connection string Dialect Cursor
awsathena+aiorest:// AthenaAioRestDialect AioCursor
awsathena+aiopandas:// AthenaAioPandasDialect AioPandasCursor
awsathena+aioarrow:// AthenaAioArrowDialect AioArrowCursor
awsathena+aiopolars:// AthenaAioPolarsDialect AioPolarsCursor
awsathena+aios3fs:// AthenaAioS3FSDialect AioS3FSCursor

Changes

  • Core adapter layer (pyathena/aio/sqlalchemy/base.py):
    • AsyncAdapt_pyathena_cursor — wraps async cursors with sync DBAPI interface, eagerly buffers results in deque during execute() (same pattern as asyncpg/aiosqlite/asyncmy)
    • AsyncAdapt_pyathena_connection — wraps AioConnection as AdaptedConnection
    • AsyncAdapt_pyathena_dbapi — fake DBAPI module with connect() and exception hierarchy
    • AthenaAioDialect — base async dialect (is_async=True, AsyncAdaptedQueuePool)
  • Variant dialects: aiorest, aiopandas, aioarrow, aiopolars, aios3fs — each sets cursor_class and extracts cursor-specific URL parameters
  • Entry points registered in pyproject.toml + aiosqlalchemy optional dependency (sqlalchemy[asyncio]>=2.0.0)
  • Tests: basic query parametrized across all 5 async drivers, Unicode, table reflection, schema inspection, column metadata
  • Documentation: async usage, connection strings, dialect table, and run_sync() reflection examples integrated into docs/sqlalchemy.md
  • Dependency: greenlet upgraded 3.1.1 → 3.3.2 (fixes Python 3.14 build failure)

Key design decisions

  • Cursor-class-agnostic adapter: The DBAPI adapter wraps whatever async cursor AioConnection.cursor() returns, so variant dialects only need to set cursor_class in create_connect_args()
  • Eager result buffering: SQLAlchemy calls fetch methods outside the greenlet context, so execute() eagerly fetches all rows into a deque buffer — this is the standard pattern used by asyncpg, aiosqlite, and asyncmy
  • Reflection compatibility: get_driver_connection() returns the adapted connection itself (not the raw AioConnection), so reflection methods work through the adapted cursor's sync wrappers
  • Naming convention: Adapter classes follow SQLAlchemy's AsyncAdapt_{driver}_{role} naming (e.g. AsyncAdapt_asyncpg_dbapi)

Related

Test plan

  • make fmt && make chk passes (lint + mypy)
  • Async SQLAlchemy tests pass locally (11/11 green)
  • CI passes on Python 3.10–3.14
  • Per-driver parametrized test_basic_query for all 5 async drivers
  • Schema reflection works via run_sync() in async context

🤖 Generated with Claude Code

laughingman7743 and others added 15 commits February 21, 2026 17:52
Enable `create_async_engine` usage by adding 5 async dialects that
mirror the existing sync ones, using SQLAlchemy's AdaptedConnection
and greenlet-based await_only() bridge pattern.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… extra

- Move async dialect files from pyathena/sqlalchemy/async_*.py to
  pyathena/aio/sqlalchemy/ to follow existing async code structure
- Add aiosqlalchemy optional dependency (sqlalchemy[asyncio]>=2.0.0)
  which pulls in greenlet required by await_only()

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add representative async tests in tests/pyathena/aio/sqlalchemy/:
  basic query, reflection, schema inspection, dialect properties
- Add ASYNC_SQLALCHEMY_CONNECTION_STRING to tests/__init__.py
- Register async dialects in tests/sqlalchemy/__init__.py
- Support --dburi async in tests/sqlalchemy/conftest.py for running
  the SQLAlchemy standard test suite with async dialects
- Add make test-sqla-async target

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add make test-sqla-async to tox [testenv] commands for CI
- Change dev group sqlalchemy dep to sqlalchemy[asyncio] to pull in
  greenlet required by async dialect tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match the naming convention of the existing create_engine helper.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These tests only verified static class attributes (is_async, driver,
supports_statement_cache) which have no logic to test.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parametrize the existing test_basic_query to run across all dialect
drivers (rest, pandas, arrow, polars, s3fs for sync; aiorest, aiopandas,
aioarrow, aiopolars, aios3fs for async) to verify each driver works
with basic queries through SQLAlchemy.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename AsyncAdaptPyathenaCursor, AsyncAdaptPyathenaConnection, and
AsyncAdaptPyathenaDbapi to AsyncAdapt_pyathena_cursor,
AsyncAdapt_pyathena_connection, and AsyncAdapt_pyathena_dbapi to match
SQLAlchemy's internal async adapter naming convention
(e.g. AsyncAdapt_asyncpg_dbapi).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Override get_pool_class() in AthenaAioDialect to return
AsyncAdaptedQueuePool, matching SQLAlchemy's asyncpg pattern.
Without this, create_async_engine fails with "QueuePool cannot be
used with asyncio engine".

Also upgrade greenlet 3.1.1 -> 3.3.2 to fix Python 3.14 build failure.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SQLAlchemy calls fetch methods (fetchone/fetchall/fetchmany) outside
the greenlet context after execute() returns. Following asyncpg's
pattern, eagerly buffer all results during execute() so that fetch
methods return from the in-memory buffer without needing await_only().

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace list with collections.deque for O(1) popleft in fetchone(),
matching asyncpg's pattern and PyAthena's own result_set implementation.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unused await_only_ attribute from AsyncAdapt_pyathena_connection
- Fix AthenaAioDialect docstring to describe base class role generically
- Replace unused engine variables with _ in async tests
- Add async SQLAlchemy documentation integrated into existing doc structure

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@laughingman7743 laughingman7743 marked this pull request as ready for review February 21, 2026 12:52
@laughingman7743 laughingman7743 merged commit d876865 into master Feb 21, 2026
5 checks passed
@laughingman7743 laughingman7743 deleted the feature/673-async-sqlalchemy-dialect branch February 21, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add async SQLAlchemy dialect using native asyncio cursors

1 participant