Process synchronization in the Accuracy Reward Method

[saitejatangudu]

Hi team,

I noticed that the accuracy reward calculation seems to involve responses from different images within a batch, which are generated in parallel across multiple processes.

However, it looks like the reward computation itself is also happening in parallel without synchronization across processes. This might lead to inconsistencies in the accuracy reward, since it depends on cross-process data.

Is this behavior intentional? Or should there be a mechanism to synchronize the responses before computing the reward in a multi-process setting?

Thanks in advance for the clarification!

[TianheWu]

Hi team,

I noticed that the accuracy reward calculation seems to involve responses from different images within a batch, which are generated in parallel across multiple processes.

However, it looks like the reward computation itself is also happening in parallel without synchronization across processes. This might lead to inconsistencies in the accuracy reward, since it depends on cross-process data.

Is this behavior intentional? Or should there be a mechanism to synchronize the responses before computing the reward in a multi-process setting?

Thanks in advance for the clarification!

Hi, thanks for your attention to our work. Actually, I have not get the idea of "the reward computation itself is also happening in parallel without synchronization across processes". Could you explain with more details?

[whycantfindaname]

Hi there, I believe I understand the promblem and here is my explanation:

1. The phrase "the reward computation itself is also happening in parallel without synchronization across processes" refers to a scenario in distributed training where each process independently calculates rewards based on its local data shard, with no explicit coordination to share or align data across processes. This can raise concerns when reward calculation depends on global batch information (e.g., comparisons between all samples in a batch), as individual processes may lack access to cross-process data, potentially leading to inconsistent reward values across processes.
2. Hugging Face TRL addresses this by leveraging Accelerate's gather and broadcast utilities for inter-process data synchronization. gather aggregates local tensors or objects (e.g., rewards, generated responses) from all processes into the main process, enabling global data consolidation. broadcast then synchronizes the aggregated results back to all processes, ensuring consistent data across the distributed setup.
   This is the code in vllm_grpo_trainer.py to achieve this,

completion_ids = broadcast_object_list(completion_ids, from_process=0) # line 570

rewards_per_func = gather(rewards_per_func) # line 695
  # Sum the rewards from all reward functions
  rewards = rewards_per_func.sum(dim=1)

3. In Ray, aggregating rollout completions across workers relies on its distributed task scheduling and object store. Each worker, acting as a remote task, generates rollout results and stores them via ray.put(). The main process then collects all results using ray.get() to aggregate completions from all workers. For dynamic coordination, Ray Actors can maintain global state, with remote methods facilitating incremental result aggregation across workers.