-
Notifications
You must be signed in to change notification settings - Fork 27
refactor task bridge to handle async tasks concurrently, preventing potential duplicate submission #600
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
JannikSt
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! This looks a lot more solid. Same for this change on the task bridge I would actually run this on a real GPU now with a proper workload for testing before merging this.
Also what do you think of using FuturesUnordered instead of the JoinSet that requires tokio unstable?
|
@JannikSt going to copy paste my response here: I'm using |
JannikSt
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
minor comment - otherwise good to go!
.github/workflows/checks.yml
Outdated
| CARGO_TERM_COLOR: always | ||
| CARGO_INCREMENTAL: 0 | ||
| RUSTFLAGS: "-C debuginfo=0" | ||
| RUSTFLAGS: "-C debuginfo=0 --cfg tokio_unstable" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this is no longer required right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no, will remove!
handle_file_validationwhich includes the on-chain call tosubmitWork()) was being submitted multiple times for the same work (identified by hash). this was likely also happening for file upload as well. a potential cause of this is because the call tohandle_file_validationwas beingtokio::spawned, so it was possible for multiple submission tasks for the same hash to be startedTaskBridgesuch that all futures run concurrently inside oneselect!{..}statement instead of being spawnedhandle_file_validationinside a tokioJoinMap, which stores the futures with the file hash as the key. thus duplicatehandle_file_validationcalls cannot run at the same time.closes #596