Skip to content

[QDP] Double-buffered pinned I/O pipeline and faster Parquet decode #751

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
400Ping wants to merge 14 commits into
base: dev-qdp
Choose a base branch
from
Open

[QDP] Double-buffered pinned I/O pipeline and faster Parquet decode #751

400Ping wants to merge 14 commits into from

Conversation

@400Ping
Copy link

@400Ping 400Ping commented Dec 22, 2025 •
edited
Loading

Purpose of PR

  • Add an pinned host buffer pool and wire it into the dual-stream pipeline so each chunk uses double-buffered pinned staging before H2D copies (reduces malloc/free and GPU idle).

Related Issues or PRs

Closes #703

Changes Made

  • Bug fix
  • New feature
  • Refactoring
  • Documentation
  • Test
  • CI/CD pipeline
  • Other

Breaking Changes

  • Yes
  • No

Checklist

  • Added or updated unit tests for all changes
  • Added or updated documentation for all changes
  • Successfully built and ran all unit tests or manual tests locally
  • PR title follows "MAHOUT-XXX: Brief Description" format (if related to an issue)
  • Code follows ASF guidelines

@400Ping 400Ping marked this pull request as draft December 22, 2025 13:39
@400Ping 400Ping marked this pull request as ready for review December 22, 2025 13:51
@400Ping
Copy link
Author

400Ping commented Dec 22, 2025

cc @guan404ming @rich7420 @ryankert01

@400Ping 400Ping changed the title [QDP] Double-buffered async I/O for read_parquet_batch [QDP] Pinned host buffer + dual-stream event pipeline to overlap copy and compute Dec 22, 2025
@rich7420
Copy link
Contributor

rich7420 commented Dec 23, 2025 •
edited
Loading

Thanks @400Ping for the patch!

  1. What's the reason you define a ffi function again?
  2. some tests failed locally due to tensor shape problem or you should fix test for excepted output.

@400Ping
Copy link
Author

400Ping commented Dec 23, 2025

Thanks @400Ping for the patch!

  1. What's the reason you define a ffi function again?
  2. some tests failed locally due to tensor shape problem or you should fix test for excepted output.

My bad just fixed it.

@400Ping 400Ping marked this pull request as draft December 24, 2025 07:54
@400Ping 400Ping marked this pull request as ready for review December 25, 2025 10:18
rich7420
rich7420 reviewed Dec 25, 2025
View reviewed changes
Copy link
Contributor

@rich7420 rich7420 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@400Ping thanks for the patch!
left some comments

@rich7420 rich7420 marked this pull request as draft December 25, 2025 13:24
@400Ping 400Ping changed the title [QDP] Pinned host buffer + dual-stream event pipeline to overlap copy and compute [QDP] Double-buffered pinned I/O pipeline and faster Parquet decode Dec 25, 2025
@400Ping 400Ping marked this pull request as ready for review December 25, 2025 22:51
rich7420
rich7420 reviewed Dec 26, 2025
View reviewed changes
@rich7420
Copy link
Contributor

rich7420 commented Dec 26, 2025

I think maybe we could add some unit tests for this.

@ryankert01
Copy link
Contributor

ryankert01 commented Dec 27, 2025

We have 2 improvement in this PR. Based on the benchmark result, I'm speculating if there's one of them are not contributing to the speed improvement. What's your experience?

400Ping added 11 commits December 29, 2025 21:13
@400Ping
Double-buffered async I/O for read_parquet_batch
bcc9970 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
update
eceabf0 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
fix python binding error
ba9b021 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
update
d4a33bb 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
update
d5075ac 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
update
198e9d2 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
update
5eb1ce0 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
update
28292ac 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
fix build error
735ed08 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
Revert "fix build error"
7cd8489 
This reverts commit 3556b5a.
@400Ping
fix build errors
b411dcf 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
Copy link
Author

400Ping commented Dec 29, 2025

We have 2 improvement in this PR. Based on the benchmark result, I'm speculating if there's one of them are not contributing to the speed improvement. What's your experience?

I think both have improvements, for the second one is what @rich7420 and @guan404ming suggested to change a different decompression technique to improve its performance. But I think overall it is because of the first one improving the speed improvements.

@400Ping
update unit test and boundary check
288cf73 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
Copy link
Author

400Ping commented Dec 29, 2025

Just tested, the second one doesn't improve much performance, going to remove it.

@400Ping
remove improvement 2
8d9ebf4 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping
fix qdp-core error
6334829 
Signed-off-by: 400Ping <fourhundredping@gmail.com>
@400Ping 400Ping requested a review from rich7420 December 29, 2025 14:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rich7420 rich7420 Awaiting requested review from rich7420

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@400Ping @rich7420 @ryankert01