Skip to content

Commit 32d4889

Browse files
committed
fix: clarify comments on worker behavior during dataset resumption
1 parent 41db56b commit 32d4889

File tree

1 file changed

+4
-2
lines changed

1 file changed

+4
-2
lines changed

tests/streaming/test_parallel.py

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -916,8 +916,10 @@ def test_parallel_dataset_partial_iteration_resume(tmp_path_factory, length, res
916916
batches_2.append(batch)
917917
if i == break_at:
918918
break
919-
# for some reason the workers are swapping their samples compared to the previous epoch when not resuming
920-
# so we update expected_2 and batches_2 accordingly
919+
# When creating a fresh DataLoader after simulating a crash (resume=False case with num_workers=2),
920+
# worker index assignment may differ from the previous session. This is expected PyTorch behavior:
921+
# each worker processes a subset of indices, but the order workers deliver batches can vary.
922+
# We adjust expected values to match the actual (deterministic but different) worker ordering.
921923
expected_2 = [expected_2[i + 1] if i % 2 == 0 else expected_2[i - 1] for i in range(len(expected_2))]
922924
batches_2 = [batches_2[i + 1] if i % 2 == 0 else batches_2[i - 1] for i in range(len(batches_2))]
923925
expected_4 = [

0 commit comments

Comments
 (0)