Skip to content

Conversation

@noot
Copy link
Contributor

@noot noot commented Jun 25, 2025

  • previously, it appears that file validation (via handle_file_validation which includes the on-chain call to submitWork()) was being submitted multiple times for the same work (identified by hash). this was likely also happening for file upload as well. a potential cause of this is because the call to handle_file_validation was being tokio::spawned, so it was possible for multiple submission tasks for the same hash to be started
  • refactor TaskBridge such that all futures run concurrently inside one select!{..} statement instead of being spawned
  • store all handle_file_validation inside a tokio JoinMap, which stores the futures with the file hash as the key. thus duplicate handle_file_validation calls cannot run at the same time.
  • this assumes that the issue is not due to the task bridge receiving the same file on the socket twice, in which case, the on-chain submission would still happen multiple times and only the first would pass.

closes #596

@noot noot requested a review from JannikSt June 25, 2025 21:01
Copy link
Member

@JannikSt JannikSt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! This looks a lot more solid. Same for this change on the task bridge I would actually run this on a real GPU now with a proper workload for testing before merging this.
Also what do you think of using FuturesUnordered instead of the JoinSet that requires tokio unstable?

@noot
Copy link
Contributor Author

noot commented Jul 1, 2025

@JannikSt going to copy paste my response here: I'm using JoinMap which is already using tokio unstable. I can change the JoinSets to FuturesUnordered<JoinHandle> which is mostly the same as JoinSet, and the JoinMap to a hash map + FuturesUnordered<JoinHandle>, but the ergonomics are not as good imo. is your issue with using tokio unstable?

@noot noot requested a review from JannikSt July 1, 2025 18:14
Copy link
Member

@JannikSt JannikSt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor comment - otherwise good to go!

CARGO_TERM_COLOR: always
CARGO_INCREMENTAL: 0
RUSTFLAGS: "-C debuginfo=0"
RUSTFLAGS: "-C debuginfo=0 --cfg tokio_unstable"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is no longer required right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, will remove!

@noot noot merged commit 2a19236 into develop Jul 3, 2025
1 check passed
@noot noot deleted the noot/refactor-work-submission branch July 3, 2025 17:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Worker submission shows error (Work already submitted)

3 participants