Improve log poll #103

luoyuxia · 2025-12-20T13:14:35Z

Purpose

Linked issue: close #94

Brief change log

Tests

API and Format

Documentation

Copilot

Pull request overview

This PR improves the log polling mechanism by introducing asynchronous fetch request handling with a buffering system. The changes enable pipelined fetching where new fetch requests can be sent while the client processes previously fetched data, improving throughput and reducing latency.

Key Changes

Introduced LogFetchBuffer to manage pending and completed fetches with async notification support
Refactored LogFetcher to spawn async tasks for fetch requests, enabling non-blocking parallel fetching
Changed LogRecordsBatches from borrowed to owned data (using Bytes) to support cross-task data sharing
Added is_from_remote flag to ReadContext to handle projection differences between local and remote log reads

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
crates/fluss/src/client/table/log_fetch_buffer.rs	New file implementing buffering system for log fetches with traits for pending/completed fetches
crates/fluss/src/client/table/scanner.rs	Refactored poll() to use timeout-based retry logic; send_fetches() spawns async tasks for parallel fetching
crates/fluss/src/client/table/remote_log.rs	Modified RemoteLogDownloadFuture to support completion callbacks; RemotePendingFetch implements PendingFetch trait
crates/fluss/src/record/arrow.rs	Removed lifetimes from LogRecordsBatches/LogRecordBatch; added remote read support in ReadContext
crates/fluss/src/client/credentials.rs	CredentialsCache now owns RpcClient and Metadata for simplified async usage
crates/fluss/tests/integration/table_remote_scan.rs	Added projection test and increased timeout to accommodate async fetching delays
crates/fluss/tests/integration/table.rs	Increased poll timeouts for integration tests
crates/fluss/src/client/table/mod.rs	Added log_fetch_buffer module
crates/fluss/Cargo.toml	Added scopeguard dependency for cleanup guards

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

crates/fluss/src/client/table/log_fetch_buffer.rs