Support consume records once some of the remote logs are downloaded #89
Description
Search before asking
- I searched in the issues and found nothing similar.
Motivation
Currently, remote reads download all files first before consuming them, right? One potential optimization is to make it a streaming process, where downloaded files can be consumed immediately.
Our internal index building scenario typically involves consuming files from a day ago or even a few hours ago. I'm concerned about efficiency issues if we use the current model.
Solution
No response
Anything else?
No response
Willingness to contribute
- I'm willing to submit a PR!