Add async SQLAlchemy dialects for native asyncio support#679
Merged
laughingman7743 merged 15 commits intomasterfrom Feb 21, 2026
Merged
Add async SQLAlchemy dialects for native asyncio support#679laughingman7743 merged 15 commits intomasterfrom
laughingman7743 merged 15 commits intomasterfrom
Conversation
Enable `create_async_engine` usage by adding 5 async dialects that mirror the existing sync ones, using SQLAlchemy's AdaptedConnection and greenlet-based await_only() bridge pattern. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… extra - Move async dialect files from pyathena/sqlalchemy/async_*.py to pyathena/aio/sqlalchemy/ to follow existing async code structure - Add aiosqlalchemy optional dependency (sqlalchemy[asyncio]>=2.0.0) which pulls in greenlet required by await_only() Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add representative async tests in tests/pyathena/aio/sqlalchemy/: basic query, reflection, schema inspection, dialect properties - Add ASYNC_SQLALCHEMY_CONNECTION_STRING to tests/__init__.py - Register async dialects in tests/sqlalchemy/__init__.py - Support --dburi async in tests/sqlalchemy/conftest.py for running the SQLAlchemy standard test suite with async dialects - Add make test-sqla-async target Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add make test-sqla-async to tox [testenv] commands for CI - Change dev group sqlalchemy dep to sqlalchemy[asyncio] to pull in greenlet required by async dialect tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Match the naming convention of the existing create_engine helper. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These tests only verified static class attributes (is_async, driver, supports_statement_cache) which have no logic to test. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Parametrize the existing test_basic_query to run across all dialect drivers (rest, pandas, arrow, polars, s3fs for sync; aiorest, aiopandas, aioarrow, aiopolars, aios3fs for async) to verify each driver works with basic queries through SQLAlchemy. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rename AsyncAdaptPyathenaCursor, AsyncAdaptPyathenaConnection, and AsyncAdaptPyathenaDbapi to AsyncAdapt_pyathena_cursor, AsyncAdapt_pyathena_connection, and AsyncAdapt_pyathena_dbapi to match SQLAlchemy's internal async adapter naming convention (e.g. AsyncAdapt_asyncpg_dbapi). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Override get_pool_class() in AthenaAioDialect to return AsyncAdaptedQueuePool, matching SQLAlchemy's asyncpg pattern. Without this, create_async_engine fails with "QueuePool cannot be used with asyncio engine". Also upgrade greenlet 3.1.1 -> 3.3.2 to fix Python 3.14 build failure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
SQLAlchemy calls fetch methods (fetchone/fetchall/fetchmany) outside the greenlet context after execute() returns. Following asyncpg's pattern, eagerly buffer all results during execute() so that fetch methods return from the in-memory buffer without needing await_only(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace list with collections.deque for O(1) popleft in fetchone(), matching asyncpg's pattern and PyAthena's own result_set implementation. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove unused await_only_ attribute from AsyncAdapt_pyathena_connection - Fix AthenaAioDialect docstring to describe base class role generically - Replace unused engine variables with _ in async tests - Add async SQLAlchemy documentation integrated into existing doc structure Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #673.
Add async SQLAlchemy dialects enabling
create_async_engineusage with PyAthena for frameworks like FastAPI.awsathena+aiorest://AthenaAioRestDialectAioCursorawsathena+aiopandas://AthenaAioPandasDialectAioPandasCursorawsathena+aioarrow://AthenaAioArrowDialectAioArrowCursorawsathena+aiopolars://AthenaAioPolarsDialectAioPolarsCursorawsathena+aios3fs://AthenaAioS3FSDialectAioS3FSCursorChanges
pyathena/aio/sqlalchemy/base.py):AsyncAdapt_pyathena_cursor— wraps async cursors with sync DBAPI interface, eagerly buffers results indequeduringexecute()(same pattern as asyncpg/aiosqlite/asyncmy)AsyncAdapt_pyathena_connection— wrapsAioConnectionasAdaptedConnectionAsyncAdapt_pyathena_dbapi— fake DBAPI module withconnect()and exception hierarchyAthenaAioDialect— base async dialect (is_async=True,AsyncAdaptedQueuePool)aiorest,aiopandas,aioarrow,aiopolars,aios3fs— each setscursor_classand extracts cursor-specific URL parameterspyproject.toml+aiosqlalchemyoptional dependency (sqlalchemy[asyncio]>=2.0.0)run_sync()reflection examples integrated intodocs/sqlalchemy.mdKey design decisions
AioConnection.cursor()returns, so variant dialects only need to setcursor_classincreate_connect_args()execute()eagerly fetches all rows into adequebuffer — this is the standard pattern used by asyncpg, aiosqlite, and asyncmyget_driver_connection()returns the adapted connection itself (not the rawAioConnection), so reflection methods work through the adapted cursor's sync wrappersAsyncAdapt_{driver}_{role}naming (e.g.AsyncAdapt_asyncpg_dbapi)Related
Test plan
make fmt && make chkpasses (lint + mypy)test_basic_queryfor all 5 async driversrun_sync()in async context🤖 Generated with Claude Code