ArXiv: 2603.02604

🎯 Pitch

Agents trained from different starting points don't need to work alone—sharing their successful reasoning steps can cut training costs in half while boosting accuracy over 3%. The catch is that random collaboration from mismatched model strengths can degrade rather than improve results, so the method introduces a principled way to weight advice based on whether an agent's confidence actually predicts when it is correct.

1. Executive Summary

This paper introduces Heterogeneous Agent Collaborative Reinforcement Learning (HACRL), a new paradigm that enables multiple independently-deployed LLM agents to mutually improve during training by sharing verified rollouts, rather than relying solely on costly on-policy sampling in isolation. The authors formalize three escalating types of agent heterogeneity—heterogeneous state (different optimization checkpoints from the same model), heterogeneous size (same family, different parameter counts), and heterogeneous model (architecturally distinct models with incompatible tokenizers)—and propose Heterogeneous Agent Collaborative Policy Optimization (HACPO), an algorithm built on GSPO that incorporates four tailored mechanisms to handle capability discrepancies and policy distribution shifts: Agent-Capability-Aware Advantage Estimation (a capability-ratio-reweighted baseline that calibrates advantage estimates across agents of differing strength), Model Capabilities Discrepancy Coefficient (a gradient-modulating factor that amplifies signals from stronger agents while attenuating those from weaker ones), Exponential Importance Sampling (a non-gradient exponential reweighting of cross-agent importance ratios to conservatively limit distributional divergence), and Stepwise Clipping (an asymmetric, per-minibatch-tightening clipping scheme that prevents cross-agent rollouts from dominating late-batch updates). Across seven mathematical reasoning benchmarks using Qwen3 and Llama3.2 models, HACPO outperforms GSPO by an average of 3.3% while using only half the rollout cost, establishing that heterogeneous agents can achieve bidirectional mutual benefit through principled rollout reuse—but only when the collaborating agents satisfy a positive competence alignment condition where their confidence correlates with actual response quality, as formalized in the theoretical gradient consistency proof.

2. Context and Motivation

The Core Problem: Wasteful Isolation in LLM Post-Training

The fundamental inefficiency this paper tackles is deceptively simple yet pervasive: when multiple LLM agents are separately optimized for the same task using Reinforcement Learning with Verifiable Rewards (RLVR), they each independently generate and then discard expensive rollouts that could benefit other agents. In current practice, given a set of agents that all need to solve mathematical reasoning problems, each agent collects its own on-policy training trajectories—sampling responses, computing verifiable rewards (e.g., checking whether the final answer matches a ground-truth solution), and updating its policy. The rollouts from Agent A are invisible to Agent B, even though both are optimizing toward the same objective on the same task distribution.

The paper frames this as a sample utilization crisis in Section 1:

"For essentially the same objective, they repeatedly generate trajectories and yield verifiable rewards, while these costly intermediate results are only utilized for self-training."

The cost of on-policy sampling dominates RLVR training budgets. In group-based policy optimization algorithms like GRPO and GSPO, each training step requires generating GG responses per prompt (typically G=8G = 8 or 1616) and then verifying all of them against ground-truth answers or unit tests. In a multi-agent setting with nn agents training independently, the total sampling cost is n×Gn \times G responses per prompt—and crucially, each agent sees only its own GG responses. The paper's core question, stated explicitly in Section 1, is:

"Can an agent improve both effectiveness and efficiency by leveraging rollouts generated by other agents, rather than relying solely on its own on-policy rollouts?"

This matters because LLM post-training (the stage after pretraining and supervised fine-tuning, where models are aligned using RL) is increasingly the bottleneck in production LLM pipelines. Unlike pretraining, which amortizes its enormous cost over billions of inference tokens, RLVR training produces models that may be used for relatively few inference tokens—making the per-step sampling cost a dominant factor. If agents could share rollouts, an nn-agent system would effectively multiply each agent's available training data by up to n×n\times, without any additional sampling.

The Broader Context: Heterogeneous LLM Ecosystems

The paper identifies a crucial real-world trend that makes this problem urgent and non-trivial. Modern LLM deployment environments are inherently heterogeneous along multiple dimensions (Section 1):

  • Parameter state: Different teams fine-tune the same base model from different checkpoints or with different hyperparameters, producing agents that differ only in their optimization state but share identical architectures.
  • Model size: Organizations deploy models at different scales (e.g., 1.7B, 4B, 8B parameters) from the same model family for different cost-latency regimes—small models for real-time applications, larger models for batch processing.
  • Architecture and vendor: Different model families (Qwen vs. Llama, for instance) have incompatible tokenizers, attention mechanisms, and pretraining corpora, yet they may be deployed side-by-side in the same ecosystem—each model independently solving overlapping sets of tasks.

The paper's heterogeneity taxonomy (Definitions 2.1–2.3) formalizes these three escalating degrees of difference: heterogeneous state (same architecture and size, different optimization checkpoints), heterogeneous size (same family, different parameter counts), and heterogeneous model (different architectures, tokenizers, training objectives). This taxonomy isn't purely academic—it captures real deployment scenarios. A company might run Qwen3-4B for cost-sensitive customer-facing applications and Qwen3-8B for internal evaluation, both being fine-tuned on the same math dataset. Under current practice, these models' RL training runs are completely siloed.

The critical insight is that these heterogeneous agents are solving the same tasks with a shared verifiable reward function. Whether a response comes from a 1.7B model or an 8B model, its correctness can be checked by the same mathematical answer verifier. This shared reward structure is what makes cross-agent rollout reuse theoretically possible—there's a common ground truth to evaluate against.

Where Existing Paradigms Fall Short

The paper positions HACRL against three established paradigms that could, in principle, address multi-agent learning but each has fundamental limitations for the heterogeneous independent-execution setting:

1. Single-Agent RLVR (GRPO, GSPO) cannot exploit cross-agent information. Standard group-based policy optimization algorithms compute advantages relative to a group of responses sampled from the same agent's policy. In GRPO (Shao et al., 2024), the advantage for the ii-th response from agent kk is:

At,i(k)=R(yt,i(k))−1G∑i=1GR(yt,i(k))σt(k)A_{t,i}^{(k)} = \frac{R(y_{t,i}^{(k)}) - \frac{1}{G}\sum_{i=1}^{G} R(y_{t,i}^{(k)})}{\sigma_t^{(k)}}

where the baseline is the mean reward of agent kk's own GG responses, and σt(k)\sigma_t^{(k)} is their standard deviation. This formulation knows nothing about other agents' responses. If another agent is substantially stronger and produces higher-quality responses, those signals are wasted. More subtly, the group-relative baseline can become miscalibrated: an agent that happens to sample a particularly good batch will have an inflated baseline, making even strong responses look mediocre (their advantage is reduced), while an agent that samples poorly on a given batch will have a depressed baseline, making weak responses look deceptively good. This batch-to-batch variance in the baseline is a known issue (Yang et al., 2026a), and it's exacerbated in multi-agent settings where different agents have systematically different capability levels.

GSPO (Zheng et al., 2025) improves on GRPO by replacing token-level importance sampling with sequence-level importance sampling—taking the geometric mean of per-token probability ratios rather than the product—which stabilizes training for Mixture-of-Experts models where different experts activate for different tokens. But GSPO is still fundamentally single-agent: each model learns only from its own rollouts. The paper explicitly builds HACPO on top of GSPO's sequence-level importance sampling (Section 3.3, Equation 11) but extends it to the cross-agent setting, where the importance ratio involves two different policies:

st,i(k,j)=(πθt(k)(yt,i(j))πθold(j)(yt,i(j)))1∣yt,i(j)∣s_{t,i}^{(k,j)} = \left(\frac{\pi_{\theta_t}^{(k)}(y_{t,i}^{(j)})}{\pi_{\theta_{\text{old}}}^{(j)}(y_{t,i}^{(j)})}\right)^{\frac{1}{|y_{t,i}^{(j)}|}}

The numerator is the probability of agent jj's response under agent kk's current policy—a cross-policy evaluation that doesn't arise in single-agent GSPO.

2. LLM-based Multi-Agent Reinforcement Learning (MARL) is a different problem. MARL approaches like MARFT (Liao et al., 2025a), MAPoRL (Park et al., 2025), and ReMA (Wan et al., 2025) design systems where multiple agents coordinate at inference time to solve tasks jointly. They might assign different roles (e.g., one agent proposes solutions, another critiques them), engage in multi-turn debates (Du et al., 2023), or iteratively refine each other's outputs. These methods are fundamentally about building a coupled multi-agent system.

HACRL, in contrast, targets a setting where agents must execute independently at inference time—only a single agent is deployed, and it must produce the final answer without interacting with other agents. The collaboration happens purely during training. As the paper puts it (Section 1):

"In many practical scenarios, only a single agent is deployed at inference time; however, we still desire that this agent benefits from knowledge acquired from other agents during training."

This distinction matters operationally: MARL systems have higher inference latency (multiple agents must communicate) and require all agents to be available simultaneously. HACRL-trained agents have the same inference cost and latency as single-agent-trained models, since they don't call other agents at test time. The paper uses Figure 1 (left panel) to visually contrast this: MARL agents form an interconnected inference graph, while HACRL agents share data during training but execute independently.

There is also a deeper capability limitation. Approaches like COPY (Ma et al., 2024) that train homogeneous copies of the same model to critique and refine each other's outputs are constrained by the model's own ceiling:

"Homogeneous models struggle to transcend their intrinsic performance ceilings" (Appendix B.2)

If both agents are identical copies of the same model, agent B's feedback to agent A cannot contain knowledge that agent A doesn't already possess—it can only help with variance reduction, not with injecting genuinely novel capabilities. Heterogeneous agents, by contrast, can contribute complementary knowledge: a model from a different family might have different inductive biases, and a larger model might have capabilities the smaller model cannot discover through self-exploration.

3. Knowledge Distillation (KD) is unidirectional and typically homogeneous. Traditional distillation (Hinton et al., 2015) follows a fixed teacher-to-student hierarchy: a large, high-capacity model (teacher) transfers knowledge to a smaller model (student) through soft labels or output distribution matching. The flow of information is one-way. Even more recent on-policy distillation methods (Agarwal et al., 2024b) that use the student's own generated trajectories to bridge the distribution gap still maintain the teacher-student asymmetry.

The paper argues that heterogeneous multi-agent settings demand bidirectional mutual learning—not a fixed teacher-student hierarchy (Section 1):

"HACRL instead enables bidirectional mutual learning among heterogeneous agents, where each agent simultaneously acts as both a knowledge provider and a learner."

Why does bidirectionality matter? Consider a 4B model and a 1.7B model training together. The 4B model is generally stronger, so it can teach the 1.7B model correct reasoning patterns. But the 1.7B model might also generate unique correct solutions that the 4B model's sampling distribution doesn't cover—different reasoning paths, different intermediate steps—that provide genuinely novel positive training signals. Furthermore, the 1.7B model generates a much larger volume of informative errors: incorrect solutions that fail in characteristic ways, which provide negative training signals (what not to do) that can sharpen the 4B model's policy. The 4B model's own rollouts might be biased toward its specific reasoning style, and the 1.7B model's rollouts serve as an exploratory complement.

This is a concrete instantiation of the exploration-exploitation tradeoff in reinforcement learning. Stronger agents tend to exploit their known good strategies; weaker agents, with less optimized policies, explore more broadly (albeit with lower average quality). This exploration yields training data diversity that the stronger agent cannot generate on its own. The paper's core design principle of capability-aware scaling (Section 3.2) explicitly operationalizes this: the stronger agent learns aggressively from the weaker agent's rare correct solutions (high learning rate on those signals) while conservatively incorporating the weaker agent's more common errors (attenuated gradient).

The paper's Figure 1 (right panels) illustrates the conceptual distinction: KD shows a directed arrow from teacher to student; HACRL shows bidirectional arrows among all agents, with each agent both sending and receiving rollouts.

The Naive Approach and Why It Fails

To motivate the complexity of HACPO's four-component design, the paper demonstrates what happens when you simply pool all agents' rollouts and apply standard GSPO without any special handling (the "Naive" baseline in Table 1). The results are devastating: across all three heterogeneity settings, the Naive baseline substantially underperforms even single-agent GSPO with the same total compute budget.

For example, in the Qwen3-4B + Qwen3-4B-Instruct setting (Table 1, top section), the 4B model achieves 68.4% average accuracy with single-agent GSPO but drops to 58.3% with Naive rollout sharing—a 10.1 percentage point degradation. The 4B-Instruct model drops from 79.9% (GSPO) to 69.1% (Naive)—a 10.8 point degradation. This is not a small inefficiency; it's catastrophic, with the collaborative approach being worse than no collaboration at all.

Why does naive sharing fail? Three interconnected reasons, which the paper's four mechanisms are designed to address:

Reason 1: Miscalibrated advantage baselines. When a 1.7B model and a 4B model share rollouts, the pooled group's mean reward is systematically lower than the 4B model's own mean (because the 1.7B model's responses are generally worse) and systematically higher than the 1.7B model's own mean (because the 4B model pulls the average up). The group-relative advantage from Equation 5, applied naively to all responses, would systematically underestimate the 4B model's advantages (its good answers look less good relative to an artificially depressed baseline) and overestimate the 1.7B model's advantages (its mediocre answers look better relative to an artificially elevated baseline). The paper calls this out explicitly in Section 3.1:

"Naively averaging rewards across all agents disregards inter-model capability differences and often results in miscalibrated advantage estimates."

Reason 2: Massive distribution shift in importance sampling. The sequence-level importance sampling ratio in Equation 11 involves the probability of agent jj's response under agent kk's current policy. When the two agents are from different model families with different tokenizers (Definition 2.3), or even just different-sized models from the same family (Definition 2.2), these probability ratios can be far from 1.0. The paper's analysis in Appendix C quantifies this: in the Qwen3-1.7B-Base and 4B-Base setting, the self-generated importance ratio shomos^{\text{homo}} averages 1.00002 with a range of only 0.00060, while the cross-agent ratio shetes^{\text{hete}} averages 0.89550 with a range of 0.07417—over 120 times larger variance. These extreme ratios, if used directly in gradient updates, would cause cross-agent responses to dominate or destabilize the training signal.

Reason 3: Capability asymmetry without gradient modulation. Even if you fix the baseline calibration (Reason 1) and importance sampling (Reason 2), there's a subtler problem: when agent kk updates on agent jj's responses, the quality of those responses matters. Learning from a stronger agent's correct solutions should be encouraged; learning from a weaker agent's errors should be conservative (the errors might be noise rather than informative negative examples). A naive approach treats all cross-agent responses equally regardless of the generating agent's capability, leading to either overly aggressive mimicry of weaker agents or insufficient exploitation of stronger ones.

The paper's ablation studies provide direct evidence for each mechanism's necessity. Table 2 shows that removing the Agent-Capability-Aware Advantage Estimator drops the 1.7B model's average from 49.3% to 46.5% and the 4B model's from 60.1% to 58.6%. Table 3 shows that removing the Model Capabilities Discrepancy Coefficient causes similar degradation (1.7B drops to 46.2%, 4B to 60.0%). Figure 4 demonstrates that removing stepwise clipping causes severe training instability (erratic reward curves), while removing the stepwise schedule (using a static clipping bound) leads to suboptimal convergence.

How the Paper Positions Itself

The paper makes three key positioning moves that situate it within the broader landscape:

First, it frames HACRL as a new problem setting, not just a new algorithm. The formalization in Definition 2.5—decomposing each agent's objective into a homogeneous term (self-generated rollouts) and a heterogeneous term (cross-agent rollouts)—is arguably the paper's primary conceptual contribution. This decomposition serves as a unifying framework for any algorithm that wants to enable cross-agent rollout reuse, not just HACPO. Future work could plug different baseline estimators, different importance weighting schemes, or different clipping strategies into the same Jhomo+Jhete\mathcal{J}_{\text{homo}} + \mathcal{J}_{\text{hete}} structure.

Second, it draws a deliberate contrast with MARL that redefines the scope of "multi-agent." Most MARL work is about joint task execution—agents must coordinate at test time. HACRL is about joint training with independent execution—a setting that, the paper argues, captures a far broader class of real-world deployment scenarios where inference-time coordination is impractical (due to latency, cost, or availability constraints) but training-time data sharing is feasible. This is not a minor distinction; it fundamentally changes the algorithm design space because there is no inference-time coordination to optimize for—only training-time knowledge transfer.

Third, it connects GSPO's sequence-level importance sampling to the cross-agent setting as a natural extension. The paper explicitly cites GSPO's success with Mixture-of-Experts models (where different expert networks handle different tokens, causing token-level importance ratios to explode) as the inspiration for handling cross-agent distribution shift:

"This success inspires a broader consideration of measuring the deviation between a sample from other models and the current policy distribution." (Appendix B.1)

GSPO showed that replacing per-token probability ratios with their geometric mean (the 1/∣y∣1/|y| exponent in Equation 11) stabilizes training when the sampling and training policies differ. HACPO repurposes this mechanism for a fundamentally harder setting: in GSPO, the two policies differ because the training policy has been updated (intra-agent drift over optimization steps); in HACPO, the two policies differ because they're different models entirely (inter-agent architectural and capability divergence). The exponential reweighting (Equation 12) and asymmetric clipping (Equation 13) are novel additions that GSPO never needed, because intra-agent drift is bounded in ways that inter-agent divergence is not.

Fourth, the theoretical analysis serves a specific rhetorical purpose. Theorem 4.1 (unbiased advantage estimation) proves that the capability-weighted mixed baseline doesn't introduce systematic bias—it's not that HACPO accepts bias in exchange for more data, but rather that it achieves additional data without bias, under the assumption that capability ratios are properly estimated. Theorem 4.3 (gradient alignment) proves that learning from cross-agent rollouts doesn't pull the optimization in a harmful direction, provided the collaborating agents satisfy a positive competence alignment condition (Assumption D.12 in the appendix): the other agent's confidence (its probability ratio relative to the learner) must be positively correlated with actual response quality. These theorems transform the paper from "we tried some heuristics and they worked" to "we can prove that this works under clearly stated conditions," which matters for adoption in sensitive applications where training stability guarantees are important.

Summary of the Motivation Chain

The paper's motivation unfolds as a tight chain of reasoning:

  1. Observation: RLVR training is bottlenecked by expensive on-policy sampling, and multiple agents training on the same task each independently discard potentially useful rollouts.
  2. Opportunity: These agents share a task and a verifiable reward function, making cross-agent rollout reuse theoretically possible.
  3. Challenge: Agents are heterogeneous (different states, sizes, or architectures), and naive rollout sharing catastrophically degrades performance due to miscalibrated baselines, extreme distribution shifts, and unmodulated capability asymmetry.
  4. Existing gaps: Single-agent RLVR can't share, MARL requires inference-time coordination, and KD is unidirectional—none address the heterogeneous collaborative independent-execution setting.
  5. Solution space: GSPO's sequence-level importance sampling provides a foundation, but needs four additional mechanisms (capability-aware baselines, gradient modulation, exponential reweighting, stepwise clipping) to handle the severity of inter-agent distribution shift.
  6. Theoretical validation: Under reasonable assumptions (capability ratios are independent of current-batch rewards, collaborating agents show positive competence alignment), the approach is provably unbiased and gradient-consistent.

3. Technical Approach

3.1 Reader Orientation

This paper proposes a training-time coordination algorithm called HACPO that lets multiple independently-deployed LLM agents share their generated problem-solving attempts (rollouts) during RL training, so each agent learns from both its own successes and mistakes and those of other agents. The system solves the problem of wasted computation in isolated RLVR training—where each agent discards rollouts that could benefit others—by introducing four mechanisms that carefully correct for the fact that different agents have different capabilities and produce responses from different probability distributions, ensuring that shared data helps rather than harms each learner.

3.2 Big-Picture Architecture (Diagram in Words)

The HACPO system has five major components:

  1. Multiple heterogeneous LLM agents ($\pi_{\theta_1}, \pi_{\theta_2}, ..., \pi_{\theta_n}$) — the policies being trained. These can differ in parameter state, model size, or architecture (Definitions 2.1–2.3). Each agent operates independently at inference time but shares its training rollouts with all others.

  2. A shared verifiable reward function ($R(\cdot)$) — an automatic correctness checker (e.g., matching the final answer string against a ground-truth answer for math problems) that assigns the same score to a given response regardless of which agent generated it. This shared reward is what makes cross-agent learning possible.

  3. The rollouts pool — for each training prompt, each agent independently samples $G$ responses from its current policy. All $n \times G$ responses across all agents are pooled into a joint dataset $\mathcal{Y}(q)$ with corresponding rewards $\mathcal{R}(q)$.

  4. Four corrective mechanisms that transform the raw pooled data into safe training signals for each agent:

    • Agent-Capability-Aware Advantage Estimation (Section 3.1) — computes a customized baseline for each agent by reweighting other agents' rewards according to relative capability.
    • Model Capabilities Discrepancy Coefficient (Section 3.2) — rescales the advantage of cross-agent responses based on the generating agent's strength relative to the learning agent.
    • Exponential Importance Sampling (Section 3.3) — applies conservative reweighting to cross-agent probability ratios to limit distribution shift.
    • Stepwise Clipping (Section 3.4) — uses asymmetric, per-minibatch-tightening bounds on importance ratios to prevent cross-agent responses from dominating late-stage batch updates.

  5. The HACPO objective (Appendix E, Equation 53) — the final loss function for each agent, which sums two terms: $\mathcal{J}_{\text{homo}}$ (loss on self-generated rollouts, using standard GSPO) and $\mathcal{J}_{\text{hete}}$ (loss on rollouts from all other agents, with the four corrective mechanisms applied).

Information flows as follows: for each training step, all agents sample $G$ responses per prompt → rewards are computed via the shared verifier → each agent's per-batch mean reward is tracked over a sliding window of $K=5$ steps to estimate capability ratios → a joint advantage baseline is computed per agent by capability-weighted averaging of all agents' rewards → each agent's policy is updated via mini-batch gradient steps using its own responses (standard GSPO) plus all other agents' responses (with exponential reweighting, capability-gradient modulation, and stepwise-clipped importance sampling).

3.3 Roadmap for the Deep Dive

  • First, the decomposed HACRL objective (Definition 2.5 and Appendix E) — the mathematical structure that separates self-generated from cross-agent loss terms, since this is the framework into which all four mechanisms plug.
  • Second, the Agent-Capability-Aware Advantage Estimator (Section 3.1) — how HACPO computes advantage baselines that account for differing agent strength, because all downstream gradient computations depend on correctly calibrated advantages.
  • Third, the Model Capabilities Discrepancy Coefficient (Section 3.2) — how the same capability ratios used for baseline calibration are repurposed as gradient modulation factors, since this dual use is a central design insight.
  • Fourth, the Exponential Importance Sampling mechanism (Section 3.3) — how cross-agent probability ratios are conservatively reweighted, because this is the primary defense against distribution shift.
  • Fifth, the Stepwise Clipping mechanism (Section 3.4) — the asymmetric, temporally-tightening clipping bounds that prevent cross-agent responses from dominating updates, because this addresses the unique temporal dynamics of cross-agent importance ratios.
  • Sixth, the full HACPO objective and training loop (Appendix E, Algorithm 1) — how everything composes into a single loss function and an end-to-end procedure.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an algorithm design paper whose core idea is that heterogeneous agents can mutually benefit from shared rollouts during RLVR training, but only if four specific mechanisms are applied to handle inter-agent capability discrepancies and policy distribution shifts.

The Decomposed HACRL Objective

The paper formalizes the collaborative training problem through a decomposition of each agent's objective into a homogeneous term (learning from self-generated rollouts) and a heterogeneous term (learning from rollouts generated by all other agents). This decomposition, defined in Equation 4 of Section 2.2, is:

J(k)=Jhomo(k)(Yk(q),Rk(q))+Jhete(k)({Yj(q),Rj(q)}j≠k)J^{(k)} = J_{\text{homo}}^{(k)}\left(Y_k(q), \mathcal{R}_k(q)\right) + J_{\text{hete}}^{(k)}\left(\{Y_j(q), \mathcal{R}_j(q)\}_{j \neq k}\right)

where for a given query $q$, $Y_k(q)$ is the set of $G$ responses sampled from agent $k$'s policy $\pi_{\theta_k}$, $\mathcal{R}_k(q)$ is the set of corresponding verifiable rewards, and the notation $\{Y_j, \mathcal{R}_j\}_{j \neq k}$ denotes the combined sets of responses and rewards from all other agents $j \neq k$.

What it represents: For each agent $k$, the total optimization objective is a sum of two components: one computed exclusively from agent $k$'s own sampled responses (the homogeneous term), and one computed from all responses sampled by every other agent in the system (the heterogeneous term). Both terms share the same structure—they reward responses with high verifiable scores and penalize those with low scores—but the heterogeneous term operates on data that agent $k$ did not generate itself.

Why this decomposition: This structure cleanly separates the standard RLVR training signal (which any single-agent algorithm like GRPO or GSPO would compute) from the novel cross-agent learning signal. It makes the heterogeneous term an explicit, independently-tunable additive component, which is critical because it allows the paper's four corrective mechanisms to be applied only to the cross-agent data while leaving self-generated data processing unchanged. Without this decomposition, any mechanism applied to handle inter-agent distribution shift would also affect the well-behaved self-generated rollouts, potentially degrading the standard training signal. The decomposition also makes the training procedure incremental: if the heterogeneous term is removed (set to zero), HACPO reduces exactly to single-agent GSPO, providing a clean ablation baseline.

The concrete instantiation of this abstract objective into an actual loss function for two agents is given in Appendix E, Equations 47–53. For agent 1, the homogeneous loss is:

Jhomo(1)=1G∑i=1G[min⁡(st,i(1,1),clip(st,i(1,1)))⋅At,i(1)]\mathcal{J}_{\text{homo}}^{(1)} = \frac{1}{G} \sum_{i=1}^{G} \left[ \min\left( s_{t,i}^{(1,1)}, \text{clip}(s_{t,i}^{(1,1)}) \right) \cdot A_{t,i}^{(1)} \right]

where $s_{t,i}^{(1,1)}$ is the sequence-level importance sampling ratio for agent 1's own responses (probability of response $i$ under current policy vs. old policy, raised to $1/|y|$), $\text{clip}(\cdot)$ is standard symmetric clipping to $[1 - \epsilon_l, 1 + \epsilon_h]$ (with $\epsilon_l = 0.0003$ and $\epsilon_h = 0.0004$ per Appendix A), and $A_{t,i}^{(1)}$ is the advantage computed by the Agent-Capability-Aware Estimator. The $\min$ operator implements the standard PPO-style pessimistic clipping: if the importance ratio is too high or too low, the clipped version is used, preventing the policy from changing too dramatically in a single update.

The heterogeneous loss for agent 1 (learning from agent 2's rollouts) is:

Jhete(1)=1G∑i=1G[clip(st,i(1,2))⋅sg(st,i(1,2))α⋅ωt(2,1)⋅At,i(1)]\mathcal{J}_{\text{hete}}^{(1)} = \frac{1}{G} \sum_{i=1}^{G} \left[ \text{clip}\left( s_{t,i}^{(1,2)} \right) \cdot \text{sg}\left( s_{t,i}^{(1,2)} \right)^\alpha \cdot \omega_t^{(2,1)} \cdot A_{t,i}^{(1)} \right]

where $s_{t,i}^{(1,2)}$ is the cross-agent importance ratio (probability of agent 2's response under agent 1's current policy vs. agent 2's old policy), $\text{sg}[\cdot]$ is the stop-gradient operator preventing the exponential term from contributing to backpropagation, $\alpha$ controls the degree of conservativeness (typically 1.0 or 3.0 depending on the agent pair, per Table 5), $\omega_t^{(2,1)}$ is the capability ratio from agent 2 to agent 1, and $\text{clip}(\cdot)$ here uses the asymmetric stepwise clipping scheme of Equation 14 rather than standard symmetric clipping. The total loss is $\mathcal{J} = \mathcal{J}_{\text{homo}} + \mathcal{J}_{\text{hete}}$.

Key design insight: The heterogeneous loss does NOT use the standard $\min$ operator that appears in the homogeneous loss. Instead, it uses a direct multiplication of the clipped importance ratio, the exponential reweighting factor, the capability coefficient, and the advantage. This is because cross-agent importance ratios exhibit fundamentally different behavior from self-agent ratios (they don't monotonically drift upward across mini-batches; they fluctuate irregularly), making the standard PPO pessimistic clipping objective inappropriate. The exponential reweighting and capability modulation serve as alternative mechanisms to control the update magnitude.

The total sample efficiency gain comes from the multiplicative effect: in an $n$-agent system, each rollout is reused $n$ times (once by its generating agent and $n-1$ times by all other agents). The paper's experiments use $n=2$, meaning each rollout appears in one homogeneous loss term and one heterogeneous loss term, effectively doubling the training data per unit of sampling cost.

Agent-Capability-Aware Advantage Estimation

The first corrective mechanism addresses a fundamental problem: when a weak agent and a strong agent share rollouts, the pooled group's average reward is not a good baseline for either agent individually. Concretely, consider a 1.7B model that achieves 30% accuracy and a 4B model that achieves 60% accuracy on the same math problems. If their $G$ responses each are pooled and the mean reward is computed, that mean falls somewhere between 30% and 60%—systematically overestimating the 1.7B model's expected performance (inflating its baseline, making its own mediocre responses look worse than they are) and systematically underestimating the 4B model's expected performance (deflating its baseline, making its own good responses look better than they should).

The standard single-agent group-relative advantage from Equation 5 is:

At,i(k)(single)=R(yt,i(k))−1G∑i=1GR(yt,i(k))σt(k)A_{t,i}^{(k)}(\text{single}) = \frac{R(y_{t,i}^{(k)}) - \frac{1}{G}\sum_{i=1}^{G} R(y_{t,i}^{(k)})}{\sigma_t^{(k)}}

where the baseline is the mean reward of agent $k$'s own $G$ responses and $\sigma_t^{(k)}$ is their standard deviation. In a multi-agent setting, this formulation ignores all other agents' data—it wastes information—but it at least ensures the baseline is calibrated to agent $k$'s own expected reward. Naively extending this to use a pooled mean from all agents would break calibration.

HACPO's solution is the Agent-Capability-Aware Advantage Estimator defined in Equations 6–9. The advantage for the $i$-th response of agent $k$ becomes:

At,i(k)=R(yt,i(k))−μ^t(k)σt,jointA_{t,i}^{(k)} = \frac{R(y_{t,i}^{(k)}) - \hat{\mu}_t^{(k)}}{\sigma_{t,\text{joint}}}

where $\hat{\mu}_t^{(k)}$ is a capability-adjusted baseline specific to agent $k$, and $\sigma_{t,\text{joint}}$ is the standard deviation computed over all agents' rewards in the current batch.

What $\hat{\mu}_t^{(k)}$ computes: A weighted average of rewards across all agents, where the weight applied to agent $j$'s rewards when computing agent $k$'s baseline is the capability ratio $\omega_t^{(k,j)}$. Specifically:

μ^t(k)=1nG∑j=1n∑i=1Gωt(k,j)⋅R(yt,i(j))\hat{\mu}_t^{(k)} = \frac{1}{nG} \sum_{j=1}^{n} \sum_{i=1}^{G} \omega_t^{(k,j)} \cdot R(y_{t,i}^{(j)})

What this does operationally: For each agent $k$ being updated, the system takes every reward from every agent's responses, multiplies each reward by a correction factor $\omega_t^{(k,j)}$ that depends on the relative strength of the generating agent $j$ versus the learning agent $k$, and averages the results. If agent $k$ is stronger than agent $j$ (so $\omega_t^{(k,j)} < 1$), agent $j$'s rewards are downweighted—their low scores don't artificially deflate the strong agent's baseline. If agent $k$ is weaker than agent $j$ (so $\omega_t^{(k,j)} > 1$), agent $j$'s rewards are upweighted—their high scores don't artificially inflate the weak agent's baseline beyond its actual expected performance.

The capability ratio itself is defined as:

ωt(k,j)=P^t(k)P^t(j)\omega_t^{(k,j)} = \frac{\hat{P}_t^{(k)}}{\hat{P}_t^{(j)}}

where $\hat{P}_t^{(k)}$ is a smoothed estimate of agent $k$'s recent performance, computed by averaging per-batch mean rewards over a sliding window of the most recent $K$ steps:

P^t(k)=1K∑τ=t−K+1tPτ(k),Pτ(k)=1G∑i=1GR(yτ,i(k))\hat{P}_t^{(k)} = \frac{1}{K} \sum_{\tau = t-K+1}^{t} P_\tau^{(k)}, \quad P_\tau^{(k)} = \frac{1}{G} \sum_{i=1}^{G} R(y_{\tau,i}^{(k)})

The window size $K = 5$ is used in all experiments (Appendix A). Each $P_\tau^{(k)}$ is simply the average reward of agent $k$'s $G$ responses in batch $\tau$—a single scalar per agent per training step.

Why this specific form: The capability ratio $\omega_t^{(k,j)}$ is designed to be a multiplicative correction that transforms rewards from agent $j$'s distribution into the scale of agent $k$'s expected rewards. If agent $k$ is twice as capable as agent $j$ (on average, $\hat{P}^{(k)} \approx 2\hat{P}^{(j)}$), then $\omega_t^{(k,j)} \approx 2$, meaning agent $j$'s rewards are doubled before being averaged into agent $k$'s baseline—correcting for the fact that $j$'s raw rewards are systematically lower than what $k$ would expect from its own policy. The temporal smoothing over $K=5$ batches is critical because per-batch mean rewards are noisy (they depend on which specific prompts were sampled and the stochasticity of the generation process). Using only the current batch's mean would introduce high-variance corrections that could destabilize training. The smoothing provides a stable, slowly-evolving estimate of each agent's true capability while still adapting to genuine capability improvements over the course of training (as all agents improve, their $\hat{P}$ values gradually increase).

Why the joint standard deviation $\sigma_{t,\text{joint}}$ instead of per-agent $\sigma_t^{(k)}$: The standard deviation normalizes advantages to a consistent scale across training steps. Using only agent $k$'s own $G$ responses to compute $\sigma_t^{(k)}$ would give a noisy estimate (only $G$ data points, typically 8). Using the pooled standard deviation across all $nG$ responses provides a much more stable normalization factor, reducing variance in the advantage estimates. This is especially important in the early stages of training when individual agents' response quality can fluctuate dramatically from batch to batch.

Theoretical guarantee: Theorem 4.1 (proved in Appendix D.1) establishes that, under Assumption D.1 (the capability ratio $\omega_t^{(k,j)}$ is statistically independent of the specific reward realizations in the current batch), the expected value of the capability-adjusted baseline $\hat{\mu}_t^{(k)}$ equals the expected reward of agent $k$ under its own policy:

E[μ^t(k)]=Ey∼πθk[R(y)]\mathbb{E}[\hat{\mu}_t^{(k)}] = \mathbb{E}_{y \sim \pi_{\theta_k}}[R(y)]

Corollary 4.2 then establishes that the resulting advantages are zero-mean in expectation: $\mathbb{E}[A_{t,i}^{(k)}] = 0$, which is the defining property of a properly calibrated advantage estimator. This means that the cross-agent data doesn't introduce systematic bias into the learning signal—it only reduces variance by providing more data points for baseline estimation.

Practical justification of Assumption D.1: The appendix acknowledges (Remark D.2) that the sliding window estimator $\hat{P}_t^{(k)}$ includes the current batch's rewards in its computation, so the independence assumption is not strictly satisfied in practice. However, as the window size $K$ increases, the contribution of the current batch diminishes as $\mathcal{O}(1/K)$, making the assumption asymptotically valid. The choice of $K=5$ represents a practical tradeoff: large enough that the current batch's influence is small, small enough that the estimator responds to genuine capability changes during training. If $K$ were too large (e.g., 100), the capability estimate would lag behind the agent's actual improving performance, causing the baseline correction to be systematically wrong in the later stages of training.

Ablation evidence (Table 2): Removing this mechanism and falling back to standard single-agent advantage estimation (where each agent computes its baseline from only its own $G$ responses) causes the 1.7B model's average benchmark accuracy to drop from 49.3% to 46.5% (a 2.8 percentage point degradation) and the 4B model's from 60.1% to 58.6% (a 1.5 point degradation). The degradation is more severe for the weaker agent because its own $G$ responses provide a much noisier estimate of its expected reward (its pass@1 is lower, so its reward variance is higher).

Model Capabilities Discrepancy Coefficient

The second corrective mechanism addresses a subtly different problem from baseline calibration: given that an agent will learn from another agent's responses, how aggressively should it incorporate those responses? The capability ratio $\omega_t^{(k,j)}$, already computed for baseline calibration, is repurposed here as a gradient modulation factor.

The mechanism is defined in Equation 10 of Section 3.2. When agent $k$ is updated using a response $y_{t,i}^{(j)}$ generated by agent $j$ (where $j \neq k$), the effective advantage used in the update is:

A~t,i(k)=ωt(j,k)⋅At,i(j)\tilde{A}_{t,i}^{(k)} = \omega_t^{(j,k)} \cdot A_{t,i}^{(j)}

where $\omega_t^{(j,k)} = \hat{P}_t^{(j)} / \hat{P}_t^{(k)}$ is the capability ratio of agent $j$ relative to agent $k$. For self-generated responses ($y_{t,i}^{(k)}$), no modulation is applied: $\tilde{A}_{t,i}^{(k)} = A_{t,i}^{(k)}$.

What this does operationally: Before using a cross-agent response to update agent $k$, the system multiplies that response's advantage by a scaling factor. If agent $j$ is stronger than agent $k$ ($\hat{P}^{(j)} > \hat{P}^{(k)}$), then $\omega_t^{(j,k)} > 1$, so the advantage is amplified—agent $k$ learns more aggressively from the stronger agent's correct solutions and more aggressively avoids its errors. If agent $j$ is weaker than agent $k$ ($\hat{P}^{(j)} < \hat{P}^{(k)}$), then $\omega_t^{(j,k)} < 1$, so the advantage is attenuated—agent $k$ learns conservatively from the weaker agent's responses, not wanting to be pulled toward its lower-quality policy.

Why this matters: Without this modulation, all cross-agent responses would be treated equally regardless of the generating agent's quality. A strong agent learning from a weak agent would give equal weight to the weak agent's frequent errors (which might be noise rather than informative training signals) and its rare correct solutions (which might represent genuinely novel reasoning paths the strong agent hasn't discovered). The capability coefficient allows the strong agent to be selective: it downweights the abundant but low-quality negative examples from the weak agent (which could otherwise dominate the gradient and pull the strong agent's policy downward) while preserving the signal from the sparse but valuable positive examples.

Why this is not redundant with the baseline correction: The baseline correction in Section 3.1 adjusts where zero is—it ensures that the expected advantage is zero for each agent. The capability coefficient adjusts how far from zero a given response's advantage pulls the policy—it's a learning-rate-like scaling factor. These are orthogonal corrections: the first addresses calibration (is this response above or below my expected performance?), the second addresses update magnitude (if it's above, how strongly should I move toward it?).

Remark 3.1 in the paper explicitly distinguishes these two roles: (i) in baseline calibration, $\omega_t^{(k,j)}$ rescales rewards to align statistics across agents; (ii) in gradient modulation, $\omega_t^{(j,k)}$ amplifies or attenuates the effective advantage. Note the inverted indices: the baseline correction multiplies agent $j$'s reward by $\omega_t^{(k,j)}$ when computing agent $k$'s baseline (making weak agent rewards look stronger to match the strong agent's scale); the gradient modulation multiplies agent $j$'s advantage by $\omega_t^{(j,k)}$ when updating agent $k$ (attenuating weak agent signals). These are mathematical inverses of each other ($\omega_t^{(k,j)} = 1 / \omega_t^{(j,k)}$) but serve complementary purposes.

Ablation evidence (Table 3): Removing the gradient modulation (setting $\omega_t^{(j,k)} = 1$ for all cross-agent responses in Equation 10, while keeping $\omega_t^{(k,j)}$ for baseline calibration) causes the 1.7B model's average accuracy to drop from 49.3% to 46.2% (a 3.1 percentage point degradation) and the 4B model's from 60.1% to 60.0% (negligible change). The asymmetric impact—the weaker agent suffers much more from removing this mechanism—makes sense: the 1.7B model benefits substantially from amplified gradients on the 4B model's strong responses, while the 4B model is already strong enough that the attenuation of the 1.7B model's weak signals doesn't dramatically change its learning trajectory.

Exponential Importance Sampling

The third corrective mechanism addresses distribution shift in the importance sampling ratios. When agent $k$ uses a response generated by agent $j$, the standard importance sampling correction (from Equation 11) computes the ratio of that response's probability under agent $k$'s current policy versus agent $j$'s old (sampling) policy:

st,i(k,j)=(πθt(k)(yt,i(j))πθold(j)(yt,i(j)))1∣yt,i(j)∣s_{t,i}^{(k,j)} = \left( \frac{\pi_{\theta_t}^{(k)}(y_{t,i}^{(j)})}{\pi_{\theta_{\text{old}}}^{(j)}(y_{t,i}^{(j)})} \right)^{\frac{1}{|y_{t,i}^{(j)}|}}

where $\pi_{\theta_t}^{(k)}$ is agent $k$'s current policy (the policy being updated), $\pi_{\theta_{\text{old}}}^{(j)}$ is agent $j$'s sampling policy (frozen at the start of the training step), and $|y_{t,i}^{(j)}|$ is the length of the response in tokens. The exponent $1/|y|$ converts the product of per-token probability ratios into a geometric mean, following GSPO's sequence-level normalization.

In the single-agent setting, this ratio stays close to 1.0 because the current policy and the old policy differ only by the parameter updates within the current training step—the drift is small. Appendix C quantifies this for self-generated responses in the Qwen3-1.7B/4B setup: $s^{\text{homo}}$ averages 1.00002 with a min-max range of only 0.00060.

In the cross-agent setting, the ratio can be far from 1.0 because the two policies are different models entirely, not just different checkpoints of the same model. For the same Qwen3-1.7B/4B setup, $s^{\text{hete}}$ averages 0.89550 with a range of 0.07417—over 120 times larger variance. These extreme ratios mean that some cross-agent responses would receive disproportionately large or small importance weights, destabilizing training.

HACPO's solution is Exponential Importance Sampling, defined in Equation 12 of Section 3.3. The raw cross-agent ratio $s_{t,i}^{(k,j)}$ is transformed into:

s~t,i(k,j)=st,i(k,j)⋅(sg[st,i(k,j)])α,for k≠j and st,i(k,j)<1.0\tilde{s}_{t,i}^{(k,j)} = s_{t,i}^{(k,j)} \cdot \left( \text{sg}[s_{t,i}^{(k,j)}] \right)^\alpha, \quad \text{for } k \neq j \text{ and } s_{t,i}^{(k,j)} < 1.0

where $\text{sg}[\cdot]$ is the stop-gradient operator (the enclosed expression contributes to the forward computation but not to gradient backpropagation), and $\alpha \geq 0$ is a hyperparameter controlling the degree of conservativeness.

What this does operationally: When agent $k$ computes the loss on a cross-agent response, the effective importance weight is the raw ratio multiplied by the raw ratio raised to the power $\alpha$, but only the first factor contributes to the gradient (because the second factor is wrapped in stop-gradient). This means the gradient sees $s_{t,i}^{(k,j)} \cdot (\text{constant})^\alpha$—the gradient magnitude is proportional to the raw ratio, but the overall scaling is dampened by a factor of $s^\alpha$ in the forward pass. The constraint $s_{t,i}^{(k,j)} < 1.0$ means this reweighting is only applied when the cross-agent ratio is less than 1.0 (i.e., when agent $k$ assigns lower probability to the response than agent $j$ did—the response is "far" from agent $k$'s distribution).

Why this form: The paper argues that in heterogeneous settings, cross-agent policy discrepancies can be much larger than on-policy drift, making direct use of the raw ratio overly aggressive. The exponential reweighting "biases agent $k$ toward learning from agents whose output distributions are more aligned with its own, while reducing the impact of large cross-agent distribution shifts" (Section 3.3). When $\alpha = 0$, the mechanism has no effect—raw importance ratios are used unchanged. When $\alpha = 1.0$, the effective weight is $s^2$ for ratios below 1.0. When $\alpha = 3.0$, the effective weight is $s^4$ for ratios below 1.0—a much more aggressive dampening.

The stop-gradient is critical: if both factors contributed to the gradient, the effective gradient would scale with $(\alpha + 1) \cdot s^\alpha$, which for large $\alpha$ could create very large gradients for responses with ratios close to 1.0 (where $s^\alpha$ remains near 1.0 but the gradient scaling factor $\alpha + 1$ is large). The stop-gradient decouples the forward scaling (for loss magnitude) from the gradient scaling (which stays proportional to $s$ regardless of $\alpha$).

Why only for $s < 1.0$: Ratios below 1.0 correspond to responses that agent $k$ considers less likely than agent $j$ did—these are the responses most likely to cause distribution shift problems because they fall outside agent $k$'s typical output distribution. Ratios above 1.0 correspond to responses that agent $k$ considers more likely than agent $j$ did—these are responses that are well-aligned with agent $k$'s policy, and the standard importance sampling correction is appropriate. However, Section 3.4 notes that $s > 1$ for cross-agent responses is "highly rare" and would be actively harmful if it occurred, because it would mean agent $k$ is amplifying the importance of another agent's responses beyond what standard importance sampling would dictate.

Hyperparameter sensitivity (Table 4): The paper sweeps $\alpha \in \{0.0, 1.0, 2.0, 3.0\}$ for two agent pairs. For Qwen3-1.7B/4B, $\alpha = 1.0$ gives the best 4B accuracy (79.2% on MATH-500), with $\alpha = 0.0$ substantially worse (75.6%). For Qwen3-4B/8B, all $\alpha$ values perform similarly (77.0–77.8% for 4B, 76.4–77.8% for 8B). The paper concludes that "the optimal $\alpha$ is model combination dependent, necessitating a balance between stable convergence and maximal information extraction" (Section 5.2). Table 5 shows that $\alpha = 3.0$ was used for the Qwen3-4B + 4B-Instruct and Qwen3-4B + 8B pairs, while $\alpha = 1.0$ was used for most other combinations.

Stepwise Clipping

The fourth corrective mechanism addresses the temporal dynamics of cross-agent importance ratios during a training step. In standard PPO/GSPO training, each training step consists of $k$ mini-batch gradient updates (using subsets of the $G$ responses). As the policy changes across these mini-batch updates, the self-generated importance ratio $s_{t,i}^{(k,k)}$ gradually drifts away from 1.0 (because the current policy diverges from the frozen old policy). This drift is monotonically increasing—each update moves the policy further from the sampling distribution.

For cross-agent responses, this monotonic drift does NOT hold. As the paper states in Section 3.4:

"Within a single training step, $s_{t,i}^{(k,j)}$ fluctuates irregularly as the number of parameter updates increases, in contrast to the self-agent ratio, which typically decays smoothly."

Why cross-agent ratios don't drift monotonically: The self-agent ratio compares the current policy against the old policy—as the current policy moves away from the old policy, the ratio consistently diverges from 1.0. The cross-agent ratio compares the current policy ($\pi_{\theta_t}^{(k)}$) against a different model's old policy ($\pi_{\theta_{\text{old}}}^{(j)}$). As agent $k$ updates, its policy might move toward or away from agent $j$'s distribution depending on the specific gradient direction in each mini-batch. It might first move closer (ratio increases toward 1.0), then move away (ratio decreases), then move closer again—there's no guarantee of monotonicity.

The problem this creates: In later mini-batches of a training step, the self-generated importance ratios tend to be farther from 1.0, so the standard PPO clipping mechanism increasingly constrains the self-generated updates. But cross-agent importance ratios might be near 1.0 in a late mini-batch (if the current policy happens to align with agent $j$'s distribution after several updates), causing cross-agent responses to suddenly dominate the gradient in that mini-batch. This creates instability: the relative influence of cross-agent vs. self-generated data fluctuates unpredictably across mini-batches within a single training step.

HACPO's solution is Stepwise Clipping, defined in Equations 13–14 of Section 3.4. The mechanism has two components:

Component 1: Asymmetric clipping bounds. Unlike standard PPO clipping, which symmetrically bounds ratios to $[1 - \epsilon_{\text{low}}, 1 + \epsilon_{\text{high}}]$, cross-agent ratios are clipped to:

st,i(k,j)∈[1.0−δ,1.0],k≠js_{t,i}^{(k,j)} \in [1.0 - \delta, 1.0], \quad k \neq j

where $\delta$ is a hyperparameter (typically 0.8, per Table 5; meaning ratios below 0.2 are clipped to 0.2).

Why asymmetric: The paper argues that $s_{t,i}^{(k,j)} > 1$ is undesirable in heterogeneous settings because it means agent $k$ assigns higher likelihood to agent $j$'s response than agent $j$ itself did. Such amplification "may guide cross-agent rollouts to dominate the gradient updates of the current agent, thereby introducing severe distributional bias" (Section 3.4). By capping the upper bound at 1.0, the clipping ensures that cross-agent responses can only be downweighted relative to their standard importance sampling weight, never upweighted. If $s_{t,i}^{(k,j)} < 1 - \delta$, the ratio is clipped to the lower bound (typically 0.2), preventing extremely small ratios from zeroing out the learning signal entirely.

The value $\delta = 0.8$ is notably large compared to standard PPO clipping bounds ($\epsilon \approx 0.2$ in GRPO, or $\epsilon_{\text{low}} = 0.0003$ and $\epsilon_{\text{high}} = 0.0004$ in GSPO for self-generated data). This reflects the fact that cross-agent policy divergence can be much larger than intra-agent drift—ratios of 0.3 or 0.5 are common, whereas self-agent ratios of 0.9997 would already trigger clipping in GSPO.

Component 2: Stepwise schedule. Within a training step with $k$ mini-batch updates (indexed from 0), the lower clipping bound tightens with each successive update:

clip(st,i(k,j))=clip(st,i(k,j),1−δ+k⋅δstep,1.0)\text{clip}(s_{t,i}^{(k,j)}) = \text{clip}\left( s_{t,i}^{(k,j)}, 1 - \delta + k \cdot \delta_{\text{step}}, 1.0 \right)

where $\delta_{\text{step}}$ is a per-update tightening factor (typically 0.025 or 0.01, per Table 5).

What this does operationally: In the first mini-batch ($k = 0$), the lower bound is $1 - \delta = 0.2$ (for $\delta = 0.8$). In the second mini-batch ($k = 1$), with $\delta_{\text{step}} = 0.025$, the bound tightens to $1 - 0.8 + 1 \cdot 0.025 = 0.225$. In the eighth mini-batch ($k = 7$), the bound is $1 - 0.8 + 7 \cdot 0.025 = 0.375$. The lower bound rises linearly, meaning cross-agent responses are subject to increasingly strict clipping as training within the step progresses.

Why stepwise tightening: The paper's intuition is that "cross-agent responses appearing in later mini-batches are subject to increasingly stricter clipping bounds. This prevents cross-agent rollouts from dominating late-stage updates within a batch" (Section 3.4). As the policy drifts across mini-batch updates, the reliability of cross-agent importance ratios decreases—the current policy is increasingly far from the state it was in when the cross-agent ratios were computed (at the start of the step). By tightening the clipping, HACPO gradually reduces the influence of cross-agent data as the step progresses, ensuring that self-generated data (with its more reliable, monotonically-drifting ratios) dominates the later updates.

Ablation evidence (Figure 4): The paper tests three variants on the Qwen3-4B/8B combination: (1) no Clip—completely removing the clipping constraint on cross-agent ratios, which causes "severe instability" with erratic training curves; (2) no Stepwise—using a static asymmetric bound $[1 - \delta, 1.0]$ without the per-minibatch tightening, which leads to "suboptimal convergence compared to the full HACPO"; and (3) full HACPO with stepwise clipping, which achieves the best final performance and most stable training. This confirms that both the asymmetric bounds and the stepwise schedule are necessary—static bounds alone don't adequately prevent late-batch cross-agent dominance.

Why not just exclude cross-agent data from later mini-batches: An alternative design would be to use cross-agent data only in early mini-batches and exclude it entirely from later ones. The stepwise clipping achieves a softer version of this: cross-agent data is gradually constrained rather than abruptly removed. This preserves some cross-agent signal throughout the step while ensuring it doesn't dominate. The paper doesn't experiment with the hard-exclusion alternative, so we can't know if it would be better or worse.

The Full HACPO Training Loop

Algorithm 1 in Appendix E provides the end-to-end procedure. The key stages are:

Initialization: $n$ agents are initialized from their respective pretrained or fine-tuned checkpoints (e.g., Qwen3-4B-Base and Qwen3-4B-Instruct). Each agent maintains its own policy parameters $\pi_{\theta_i}$ and its own sliding window of recent batch accuracies for capability estimation.

Per-step procedure (for step $t = 1$ to $N$):

  1. Sample a batch of prompts $\mathcal{D}_{\text{batch}}$ from the training distribution (7.5k high-quality MATH questions).
  2. For each agent $i$, freeze its current policy as $\pi_{\theta_i^{\text{old}}}$ (the sampling policy for this step).
  3. For each agent $i$, sample $G = 8$ responses per prompt from $\pi_{\theta_i^{\text{old}}}$ (total $8 \times |\mathcal{D}_{\text{batch}}|$ responses per agent). Compute verifiable rewards for all responses.
  4. Compute each agent's per-batch mean reward $P_t^{(i)}$ and update its smoothed capability estimate $\hat{P}_t^{(i)}$ using the sliding window of $K = 5$ steps.
  5. For each agent $i$:
    • Compute the joint advantage baseline $\hat{\mu}_t^{(i)}$ using Equation 7 with capability ratios $\omega_t^{(i,j)}$ for all agents $j$.
    • Compute the joint standard deviation $\sigma_{t,\text{joint}}$ across all agents' rewards.
    • Compute advantages $A_{t}^{(i)}$ for all responses (both self-generated and cross-agent) using Equation 6.
    • For $k$ mini-batch updates (within this training step):
      • Compute the homogeneous loss $\mathcal{J}_{\text{homo}}$ on the mini-batch subset of self-generated responses, using standard GSPO with symmetric clipping to $[1 - 0.0003, 1 + 0.0004]$.
      • Compute the heterogeneous loss $\mathcal{J}_{\text{hete}}$ on the mini-batch subset of all other agents' responses, with:
        • Exponential importance sampling reweighting (Equation 12) with $\alpha$ from Table 5.
        • Capability coefficient gradient modulation (Equation 10).
        • Stepwise asymmetric clipping (Equation 14) with bounds $[1 - \delta + k \cdot \delta_{\text{step}}, 1.0]$.
      • Sum the losses and perform a gradient update on $\pi_{\theta_i}$.

Key configuration details (from Appendix A):

  • Learning rate: $1 \times 10^{-6}$.
  • Total batch size: 128 prompts, mini-batch size: 64, with $G = 8$ rollouts per prompt.
  • For the Resource-Equivalent Baseline (GSPO×2): mini-batch size 32 and $G = 16$ rollouts per prompt, ensuring double updates per step.
  • Maximum prompt length: 1024 tokens. Maximum response length: 4096 tokens (extended to 8196 for AIME2025 evaluation).
  • Training for one epoch over the 7.5k MATH training questions.
  • Evaluation metrics: best@30 for AIME2025, avg@1 for all other benchmarks.
  • Implemented using the verl framework (Sheng et al., 2024) on 8 GPUs.

Handling tokenizer incompatibility (Section 3.3): For heterogeneous model pairs that satisfy Definition 2.3 (e.g., Qwen3-4B-Base and Llama3.2-3B-Instruct, which have different tokenizers), the response generated by one agent must be retokenized to compute probabilities under the other agent's policy. The procedure is: detokenize agent $j$'s response back into raw text, then retokenize it using agent $k$'s tokenizer. The paper notes that "through sequence-level normalization, the slight length discrepancies arising from re-tokenization become negligible" (Section 3.3). This is because the geometric mean normalization ($1/|y|$ exponent) makes the importance ratio robust to small token count differences—the probability ratio is averaged over the response length rather than being a product that would explode or vanish with length mismatches.

Why the Resource-Equivalent Baseline (GSPO×2) is important: A skeptical reader might argue that HACPO's gains simply come from using more data (each agent sees rollouts from two agents instead of one). The GSPO×2 baseline controls for this: it doubles the number of rollouts per prompt ($G = 16$ instead of 8) and doubles the number of mini-batch updates per step (by halving the mini-batch size to 32), matching HACPO's total data throughput and compute budget. Critically, GSPO×2 uses only self-generated rollouts from a single agent—it has twice as much data but no cross-agent diversity. The fact that HACPO consistently outperforms GSPO×2 (Table 1, across all settings) demonstrates that cross-agent diversity provides complementary value beyond mere data volume—the heterogeneous rollouts contain genuinely novel information that additional self-generated samples cannot replicate.

4. Key Insights and Innovations

Innovation 1: Reframing Multi-Agent LLM Training as "Collaborative Optimization with Independent Execution"

The paper's most fundamental conceptual move is not a specific algorithm, but rather a reframing of the problem space that opens up an entirely new research direction. Prior work on multi-agent systems with LLMs falls into two camps. The dominant paradigm—LLM-based Multi-Agent Reinforcement Learning (MARL), as in MARFT (Liao et al., 2025a), MAPoRL (Park et al., 2025), and ReMA (Wan et al., 2025)—designs systems where agents coordinate at inference time to jointly solve tasks, typically through debate, role specialization, or iterative refinement. The other paradigm—Knowledge Distillation (Hinton et al., 2015; Agarwal et al., 2024b)—transfers knowledge unidirectionally from a fixed teacher to a fixed student.

HACRL proposes something fundamentally different: agents train together but deploy independently. The collaboration is purely during training—agents share their verified rollouts—but at inference time, a single agent produces the final answer without communicating with any other agent. This changes the design space in a non-trivial way. In MARL, the algorithm designer must optimize for inference-time coordination: which agent speaks when? How should agents resolve disagreements? In HACRL, the designer must optimize for training-time knowledge transfer while ensuring the resulting single-agent policy is self-sufficient.

The significance of this reframing extends beyond the paper's immediate results. It captures a far broader class of real-world deployment scenarios where inference-time coordination is impractical—due to latency constraints (multiple agents communicating would add seconds to response time), cost (running multiple models per query multiplies inference cost), or availability (different agents may be deployed by different teams at different times). By separating the concerns of training collaboration from inference execution, HACRL makes multi-agent training compatible with single-agent deployment. The paper validates this reframing with the strong negative result from the Naive baseline in Table 1: simply pooling rollouts and applying standard RL algorithms degrades performance by ~10 percentage points across all settings. This negative result isn't just an ablation—it's a proof that the naive approach to this reframed problem is wrong, establishing that the problem is genuinely hard and requires the tailored mechanisms HACPO provides.

Figure 1's visual contrast between MARL (interconnected inference graph) and HACRL (isolated inference, shared training data) encodes this conceptual shift in a single diagram. Prior work used similar figures to illustrate system architecture; HACRL uses it to illustrate a problem boundary redefinition.

Innovation 2: Capability Ratios as a Unifying Mechanism for Heterogeneous Collaboration

The paper's second conceptual contribution is the identification of capability ratios ($\omega_t^{(k,j)}$) as the single sufficient statistic that enables heterogeneous agents to safely share training data. Prior work on multi-agent learning typically assumes homogeneity (all agents have identical architectures and similar capabilities) or handles heterogeneity through architectural mechanisms like shared parameter spaces. HACRL introduces a different approach: measure relative capability and use it to correct for differences, rather than trying to eliminate the differences.

The capability ratio appears in two mathematically complementary roles that together form a complete correction scheme. In the baseline calibration (Section 3.1), $\omega_t^{(k,j)}$ rescales rewards from agent $j$ so that agent $k$'s baseline is properly calibrated to its own expected performance—preventing the systematic overestimation or underestimation that naive pooling would cause. In gradient modulation (Section 3.2), the inverse ratio $\omega_t^{(j,k)}$ scales the effective advantage of cross-agent responses, amplifying signals from stronger agents and attenuating signals from weaker ones. Remark 3.1 in the paper explicitly notes this dual use, but the deeper insight is that these two roles are mathematical inverses serving complementary purposes: one fixes where the baseline is, the other fixes how far from baseline each response pulls the policy.

What makes this conceptually distinctive is that it treats heterogeneity not as a nuisance to be eliminated but as structural information to be exploited. The capability ratio doesn't try to make agents look identical; it translates between their respective performance scales. A weaker agent's reward of 0.3 and a stronger agent's reward of 0.6 might both represent "average" performance for their respective capabilities, and the capability ratio encodes this translation. The paper's theoretical analysis (Theorem 4.1) proves that this translation is unbiased under mild assumptions—the expected value of the capability-adjusted baseline equals the agent's own expected reward.

This is a fundamental shift from the dominant approach in heterogeneous-agent reinforcement learning, where the typical strategies are either to homogenize agents (share parameters, use a common critic) or to manually design agent-specific reward structures. HACRL's approach is more principled: learn the translation automatically from recent performance statistics, and apply it uniformly across both baseline estimation and gradient modulation. The sliding window ($K=5$) provides a practical approximation to the theoretical ideal, trading off responsiveness (small $K$) against statistical independence (large $K$). This is not an ad-hoc engineering choice; it's a principled resolution of the exploration-exploitation tension in capability estimation.

The ablation evidence (Tables 2 and 3) confirms that both roles of the capability ratio are independently necessary: removing the baseline calibration degrades the weaker agent more (its baseline becomes miscalibrated), while removing the gradient modulation degrades the weaker agent more (it loses amplified signals from the stronger agent). The asymmetry of these degradation patterns validates that the two roles address distinct problems.

Innovation 3: The Discovery That Naive Rollout Sharing Is Worse Than No Sharing—and the Theoretical Diagnosis of Why

The paper's third contribution is a diagnostic finding that is arguably more important than the positive results: naive rollout sharing across heterogeneous agents catastrophically degrades performance, and the paper provides a theoretical framework for understanding exactly why. The Naive baseline in Table 1 shows drops of 10+ percentage points compared to single-agent GSPO across all three heterogeneity settings. This is not a marginal inefficiency; it's a finding that the intuitive idea of "more training data is better" reverses sign when the data comes from a different distribution without appropriate correction.

Prior work had demonstrated that LLMs struggle to learn from off-policy data—the entire motivation for on-policy RL algorithms like PPO is to avoid distribution shift—but the scale of the degradation here is striking. The field's default assumption, implicit in the enthusiasm for data sharing and collaborative training, was that combining rollouts would at worst provide no benefit. The paper shows it can be actively harmful, and worse, that the harmful effects are systematic rather than stochastic: the weak agent's baseline gets inflated (its mediocre responses look bad relative to the strong agent's good responses), and the strong agent's baseline gets deflated (its good responses look even better relative to the weak agent's poor responses). Both effects lead to miscalibrated training signals that push policies in wrong directions.

The Appendix C analysis of importance ratio distributions provides the quantitative diagnosis. Self-agent ratios cluster tightly around 1.0 (range 0.00060), while cross-agent ratios have over 120× larger variance (range 0.07417) and systematically deviate from 1.0 (mean 0.89550). This isn't just noise—it's a structural property of heterogeneous policies that means standard importance sampling corrections designed for small intra-agent drift are fundamentally inadequate for the inter-agent case.

The Stepwise Clipping mechanism's motivation in Section 3.4 contains a second diagnostic finding: cross-agent importance ratios don't drift monotonically across mini-batch updates. This is a non-obvious empirical observation with significant implications. Standard PPO clipping assumes monotonic drift (the current policy steadily diverges from the old policy across updates), which justifies using a fixed clipping bound. Cross-agent ratios violate this assumption because updates might move the policy toward or away from another agent's distribution in unpredictable ways. This diagnostic finding justifies the entire stepwise clipping design—it's not an arbitrary engineering choice but a response to a specific, empirically observed failure mode.

These diagnostic findings have implications beyond HACPO. Any future work on multi-agent rollout sharing will need to contend with baseline miscalibration and non-monotonic importance ratio dynamics. The paper effectively provides a checklist of failure modes that any collaborative training algorithm must address, even if the specific corrective mechanisms differ.

Innovation 4: Proving That Cross-Agent Learning Preserves Gradient Direction Under a Minimal Competence Assumption

The paper's fourth contribution is a theoretical result that establishes boundary conditions for when cross-agent learning is beneficial. Theorem 4.3 (developed in Appendix D.3) proves that the heterogeneous objective's gradient is positively aligned with the homogeneous objective's gradient—meaning cross-agent responses provide a directionally consistent learning signal—under a specific condition formalized as Assumption D.12: the collaborating agent's confidence must be positively correlated with actual response quality.

This assumption, which the paper calls "Positive Competence Alignment," is deceptively simple but carries substantial weight. It essentially requires that when agent $j$ assigns high probability to a response (relative to agent $k$'s assessment), that response tends to actually be good. If this holds, then the importance-weighted gradient from cross-agent data points in the same direction as the gradient from self-generated data—it accelerates learning without pulling the policy off-course. If it fails—if agent $j$ is confidently wrong—then cross-agent learning will be harmful.

This is not an obvious theoretical result. The gradient of the heterogeneous objective (Proposition D.6) involves importance-weighted log-probability gradients from a different model's distribution, which could in principle point in any direction. The proof shows that under the competence alignment assumption and the specific reweighting mechanisms of HACPO (exponential importance sampling, capability modulation, clipping), the resulting gradient decomposes into a positively weighted version of the homogeneous gradient direction plus a covariance term that is constrained to be sufficiently bounded.

The significance lies in transforming HACRL from a heuristic "let's share data and hope it helps" into a principled framework with testable conditions. The competence alignment assumption is not automatically satisfied—it depends on the specific collaborating agents. If two models are adversarially misaligned (one model is confidently wrong on problems where the other is uncertain), HACPO's theoretical guarantees break down. This provides a principled explanation for when cross-agent collaboration will succeed versus fail, and suggests that agent selection matters—not all heterogeneous pairs will benefit equally from collaboration.

The practical manifestation of this theory appears in the asymmetric benefits across agent pairs. In the Qwen3-4B + 4B-Instruct setting (Table 1), the weaker 4B model gains substantially more from collaboration than the stronger 4B-Instruct model (from 68.4% to 75.5% average, a 7.1 point gain, versus 79.9% to 81.3%, a 1.4 point gain). This asymmetry is predicted by the theory: the stronger model's responses are positively competence-aligned (its confidence correlates with correctness), so the weaker model benefits substantially; the weaker model's responses are less competence-aligned (it may be confidently wrong), so the stronger model benefits less. The theory doesn't just explain that collaboration helps; it explains who benefits how much and why.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. Training uses 7.5k high-quality math questions from the MATH dataset (Hendrycks et al., 2021). Evaluation spans seven benchmarks: MATH-500, MATH, GSM8K (Cobbe et al., 2021), AIME2025, AMC23 (Cairns, 1916), Minerva (Lewkowycz et al., 2022), and Olympiad (He et al., 2024). The paper adopts the specific MATH split from Lightman et al. (2022): 12,000 training questions and 500 test questions, though only the training portion is used for RLVR optimization.

  • Base models. The experiments use Qwen3 (Yang et al., 2025a) and Llama3.2 (Grattafiori et al., 2024) model families. Specific configurations include: Qwen3-1.7B-Base, Qwen3-4B-Base, Qwen3-8B-Base (base pretrained models); Qwen3-4B and Qwen3-4B-Instruct (further fine-tuned variants); and Llama3.2-1B-Instruct and Llama3.2-3B-Instruct. The paper states that Qwen3-(1.7B/4B/8B)-Base denotes raw pretrained models, while Qwen3-(1.7B/4B/8B) refers to distilled variants obtained through strong model distillation from their corresponding base models, and Qwen3-4B-Instruct is further fine-tuned for instruction following (Appendix A). This diversity enables testing across all three heterogeneity types: heterogeneous state (4B vs. 4B-Instruct—same architecture, different fine-tuning stages), heterogeneous size (1.7B-Base vs. 4B-Base, 4B-Base vs. 8B-Base—same family, different parameter counts), and heterogeneous model (Qwen3-4B-Base vs. Llama3.2-3B-Instruct, Qwen3-1.7B-Base vs. Llama3.2-1B-Instruct—different architectures, tokenizers, and pretraining corpora).

  • Metrics. The primary metric is accuracy (%) on each benchmark, measured as the fraction of problems for which the model's final answer matches the ground-truth solution. For AIME2025 specifically, the paper reports best@30 (the highest accuracy achieved by selecting the best among 30 candidate solutions per problem). For all other benchmarks, avg@1 is used (the accuracy of a single greedy or sampled response per problem). The average across all seven benchmarks is reported as a summary statistic. The paper uses the grading function released by Lightman et al. (2022) for answer verification.

  • Baselines. The paper compares against four distinct baselines, each designed to isolate a specific alternative explanation for HACPO's performance:

    • GRPO (Shao et al., 2024): Standard group-relative policy optimization applied independently to each agent. No cross-agent data sharing. Same rollout cost as HACPO per agent (G=8 responses per prompt) but only self-generated data. Uses clipping bounds $\epsilon_{\text{low}} = 0.2$ and $\epsilon_{\text{high}} = 0.28$ following the DAPO trick (Yu et al., 2025).
    • GSPO (Zheng et al., 2025): Group Sequence Policy Optimization with sequence-level importance sampling, also applied independently. Same rollout cost as HACPO (G=8). Uses clipping bounds $\epsilon_{\text{low}} = 0.0003$ and $\epsilon_{\text{high}} = 0.0004$ as in the original GSPO paper. This serves as the most direct single-agent comparison point.
    • GSPO×2: A resource-equivalent baseline where a single agent is trained with GSPO using double the rollouts per prompt (G=16) and double the mini-batch updates per step (achieved by halving mini-batch size to 32). This controls for the possibility that HACPO's gains come merely from increased data volume rather than cross-agent diversity. Crucially, GSPO×2 uses only self-generated rollouts—it has twice as much data per agent but no heterogeneity.
    • Naive: A two-agent setting where rollouts are pooled across agents and standard GSPO is applied without any of HACPO's corrective mechanisms (no capability-aware baseline, no gradient modulation, no exponential importance sampling, no stepwise clipping). This validates the necessity of HACPO's algorithmic innovations by demonstrating that naive sharing is worse than no sharing at all. Same rollout and policy update costs as HACPO.

  • Generation budget / compute accounting. The paper measures compute in terms of rollouts per prompt per agent. In HACPO with two agents, each agent generates G=8 responses per prompt, for a total of 16 responses across both agents. Each agent then trains on all 16 responses: its own 8 (homogeneous term) plus the other agent's 8 (heterogeneous term). The standard single-agent baselines (GRPO, GSPO) use G=8 responses per prompt—same per-agent sampling cost as HACPO but with only self-generated training data. The GSPO×2 baseline uses G=16 responses per prompt—double the per-agent sampling cost but still self-generated only. Policy updates are matched: HACPO and GSPO both perform updates on all available responses (16 for HACPO, 8 for GSPO, 16 for GSPO×2). The total batch size is 128 prompts, with mini-batch size 64 for HACPO and GSPO, and mini-batch size 32 for GSPO×2 (to maintain double updates per step while processing double the rollouts). All experiments use the verl framework (Sheng et al., 2024) on 8 GPUs, train for one epoch over the 7.5k MATH training questions, with learning rate $1 \times 10^{-6}$, maximum prompt length 1024 tokens, and maximum response length 4096 tokens (extended to 8196 for AIME2025 evaluation).

  • Statistical protocol. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any results. Main experimental results are from a single training run per configuration (no mention of multiple seeds or repeated trials). The evaluation benchmarks have varying sizes: MATH-500 (500 problems), MATH (5,000 test problems), GSM8K (1,319 problems), AIME2025 (30 problems), AMC23 (40 problems), Minerva (272 problems), and Olympiad (675 problems). The average across benchmarks weights all seven equally regardless of their size, meaning AIME2025 (30 problems) and AMC23 (40 problems) contribute as much to the average as MATH (5,000 problems). For the main experimental results, the paper reports best@30 on AIME2025 while using avg@1 for all other benchmarks—a potential inconsistency in metric treatment across benchmarks.

Main Quantitative Results

Heterogeneous State: Qwen3-4B and Qwen3-4B-Instruct

Table 1 (top section) presents results for the weakest form of heterogeneity—two models from the same architecture and size family that differ only in their fine-tuning stage (base distilled model vs. instruction-tuned variant). The headline finding: HACPO improves the weaker 4B model from 68.4% (GSPO) to 75.5% average accuracy (+7.1 percentage points) and the stronger 4B-Instruct model from 79.9% to 81.3% (+1.4 points).

Breaking this down by benchmark (Table 1, top section):

  • 4B model: HACPO achieves substantial improvements on the hardest benchmarks: AIME2025 jumps from 48.5% (GSPO) to 62.2% (+13.7 points), AMC23 from 67.5% to 85.0% (+17.5 points), and Olympiad from 56.4% to 64.3% (+7.9 points). On easier benchmarks, gains are more modest but still consistent: MATH-500 from 85.4% to 91.0% (+5.6 points), MATH from 87.0% to 90.5% (+3.5 points). GSM8K shows a negligible gain (92.5% to 93.3%, +0.8 points), likely due to ceiling effects.
  • 4B-Instruct model: The stronger model shows modest but broad improvements: AIME2025 from 72.0% to 75.7% (+3.7 points), AMC23 from 90.0% to 95.0% (+5.0 points), Olympiad from 72.6% to 73.2% (+0.6 points). GSM8K, MATH-500, and MATH show small improvements or remain flat (93.9% to 94.6%, 93.8% to 94.8%, 94.0% to 94.3% respectively), again consistent with ceiling effects.

The comparison to the GSPO×2 baseline is revealing: the 4B model's average with GSPO×2 is 69.1% versus 75.5% with HACPO—a 6.4 point gap. Since GSPO×2 uses twice the self-generated data (same total compute as HACPO), this demonstrates that the cross-agent diversity from the Instruct model provides value beyond mere data volume. For the 4B-Instruct model, GSPO×2 achieves 79.9% average—identical to standard GSPO and 1.4 points below HACPO—suggesting that for already-strong models, additional self-generated data provides diminishing returns while cross-agent data from a weaker but differently-distributed model offers complementary signals.

The Naive baseline confirms the necessity of HACPO's mechanisms: the 4B model drops to 58.3% average (10.1 points below GSPO), and the 4B-Instruct model drops to 69.1% (10.8 points below GSPO). This 10+ point degradation from naive sharing is one of the paper's most important results—it is worse than no collaboration at all. The effect is particularly severe on harder benchmarks: on AIME2025, the 4B model drops from 48.5% (GSPO) to 37.8% (Naive); on Olympiad, the 4B-Instruct model drops from 72.6% to 55.2%.

Figure 3(a) shows the training dynamics. The HACPO curves for both agents consistently lie above the corresponding GSPO curves throughout training, with the gap widening as training progresses. This indicates that HACPO's benefit is not just a one-time boost but compounds over the course of optimization.

Heterogeneous Size: Qwen3-1.7B-Base and Qwen3-4B-Base

Table 1 (middle section) presents results for models from the same family (Qwen3) with different parameter counts: a 1.7B model and a 4B model, both starting from base pretrained checkpoints. The headline finding: HACPO improves the 1.7B model from 46.7% (GSPO) to 49.3% average accuracy (+2.6 points) and the 4B model from 57.8% to 60.1% (+2.3 points). Both models benefit, with the weaker model gaining slightly more.

Breaking this down by benchmark:

  • 1.7B-Base model: The most dramatic improvement is on AIME2025, from 14.8% (GSPO) to 22.5% (+7.7 points)—a 52% relative improvement. AMC23 improves from 45.0% to 45.0% (flat), Minerva from 27.2% to 27.9% (+0.7 points), and Olympiad from 28.7% to 31.4% (+2.7 points). On easier benchmarks, MATH-500 improves from 64.8% to 69.0% (+4.2 points), MATH from 64.1% to 67.4% (+3.3 points), and GSM8K from 82.6% to 82.2% (−0.4 points, a small fluctuation).
  • 4B-Base model: Improvements are concentrated on harder benchmarks: AMC23 from 52.5% to 57.5% (+5.0 points), Olympiad from 46.0% to 46.7% (+0.7 points). MATH-500 improves from 78.2% to 80.8% (+2.6 points) and MATH from 78.7% to 80.1% (+1.4 points). AIME2025 shows a small gain (25.0% to 26.7%, +1.7 points), and GSM8K improves from 87.7% to 90.3% (+2.6 points).

The GSPO×2 comparison tells a nuanced story. For the 1.7B model, GSPO×2 achieves 47.5% average versus HACPO's 49.3%—HACPO wins by 1.8 points. For the 4B model, GSPO×2 achieves 57.5% versus HACPO's 60.1%—HACPO wins by 2.6 points. This suggests that the weaker 1.7B model benefits roughly equally from additional data (whether self-generated or cross-agent), while the stronger 4B model benefits more from the cross-agent diversity than from additional self-generated data. This asymmetry makes intuitive sense: the 4B model's own rollouts are already relatively high-quality, so doubling them provides diminishing returns; the 1.7B model's rollouts are lower quality, so interleaving them with 4B rollouts provides genuinely novel training signals (including correct solutions the 4B wouldn't generate on its own).

The Naive baseline again confirms catastrophic degradation: the 1.7B model drops to 42.5% average (4.2 points below GSPO), and the 4B model drops to 52.6% (5.2 points below GSPO). The degradation is less severe than in the heterogeneous state setting (10+ points vs. 4–5 points), which may reflect the fact that these are both base models with similar training objectives, making their output distributions more compatible even without corrective mechanisms. Figure 3(b) shows the HACPO training curves consistently above GSPO for both models, with smoother trajectories than Figure 3(a), suggesting that same-family models experience less distribution shift than models from different fine-tuning stages.

A notable pattern: the 1.7B model actually shows a decline on GSM8K (82.6% with GSPO to 82.2% with HACPO) while improving on harder benchmarks. This is unusual—typically, improvements on hard tasks come with at least stable performance on easy ones—and without error bars, it's impossible to determine whether this is a real effect or sampling noise from a single training run.

Heterogeneous Model: Qwen3-4B-Base and Llama3.2-3B-Instruct

Table 1 (bottom section) presents the most challenging setting: two models from entirely different families with incompatible tokenizers, architectures, and pretraining objectives. This is the ultimate stress test for HACPO's distribution-shift handling mechanisms. The headline finding: HACPO improves the Qwen3-4B model from 57.8% (GSPO) to 59.7% average accuracy (+1.9 points) and the Llama3.2-3B model from 35.1% to 39.0% (+3.9 points). Both models benefit, with the weaker Llama model gaining more.

Breaking this down by benchmark:

  • Qwen3-4B-Base: GSM8K shows the largest improvement, from 87.7% (GSPO) to 92.1% (+4.4 points)—notable because GSM8K is an easier benchmark where ceiling effects might limit gains. AMC23 improves from 52.5% to 60.0% (+7.5 points). MATH-500 drops slightly from 78.2% to 78.6% (+0.4 points), and MATH drops from 78.7% to 78.3% (−0.4 points). AIME2025 improves from 25.0% to 26.8% (+1.8 points). Olympiad drops from 46.0% to 44.2% (−1.8 points). The inconsistent direction of small changes on individual benchmarks (some up, some down) suggests these may be within the noise range of single-run evaluations.
  • Llama3.2-3B-Instruct: MATH-500 shows the largest improvement, from 51.2% (GSPO) to 56.6% (+5.4 points)—a 10.5% relative improvement. MATH improves from 50.1% to 54.8% (+4.7 points), and AMC23 from 22.5% to 35.0% (+12.5 points). Olympiad improves from 17.0% to 20.8% (+3.8 points). GSM8K from 81.2% to 82.6% (+1.4 points). AIME2025 shows zero gain (5.4% for both GSPO and HACPO), and Minerva dips slightly from 18.4% to 17.6% (−0.8 points).

The GSPO×2 comparison for the Llama3.2-3B model is particularly telling: GSPO×2 achieves only 33.4% average, which is worse than standard GSPO at 35.1%. This is a striking finding: doubling the Llama model's self-generated data actually degrades performance, while sharing heterogeneous rollouts from a completely different model family (Qwen) improves it by 3.9 points. This suggests that the Llama3.2-3B model suffers from limited self-generated exploration diversity—its own policy distribution is narrow, and doubling the samples doesn't expand it meaningfully—while Qwen's rollouts provide genuinely novel trajectories that the Llama model would never discover through self-sampling.

The Naive baseline again confirms degradation: Qwen3-4B drops to 52.6% average (5.2 points below GSPO), and Llama3.2-3B drops to 29.4% (5.7 points below GSPO). The magnitude of degradation is comparable to the heterogeneous size setting, suggesting that model family differences alone don't make naive sharing dramatically worse—the primary failure modes are common across all heterogeneity types.

Figure 3(c) shows the training curves. Both HACPO curves lie above their GSPO counterparts, but the gap is smaller than in Figures 3(a) and 3(b), consistent with the smaller average improvements in this setting. The curves also appear somewhat noisier, which may reflect the additional challenge of tokenizer incompatibility requiring retokenization at each step.

Additional Results: Qwen3-4B-Base and Qwen3-8B-Base

Table 7 (Appendix F) extends the heterogeneous size analysis to a larger scale gap: 4B vs. 8B, both Qwen3-Base models. The headline finding: HACPO improves the 4B model from 57.8% (GSPO) to 61.4% average accuracy (+3.6 points) and the 8B model from 60.6% to 63.0% (+2.4 points). Notably, the weaker model gains more than the stronger one.

The 4B model's improvements are concentrated on harder benchmarks: AIME2025 from 25.0% to 27.5% (+2.5 points), AMC23 from 52.5% to 60.0% (+7.5 points), Olympiad from 46.0% to 46.3% (+0.3 points). MATH-500 improves from 78.2% to 81.0% (+2.8 points), and MATH from 78.7% to 80.3% (+1.6 points). The 8B model shows improvements on AIME2025 (22.5% to 32.3%, +9.8 points—a 44% relative improvement) and smaller gains elsewhere: AMC23 from 60.0% to 62.5% (+2.5 points), MATH-500 from 79.4% to 82.8% (+3.4 points).

The GSPO×2 baseline for the 8B model achieves only 59.5%, which is worse than standard GSPO at 60.6%—mirroring the Llama3.2-3B finding where doubled self-generated data degrades a relatively strong model's performance. HACPO at 63.0% substantially exceeds both. This reinforces the paper's central argument: heterogeneous cross-agent data provides qualitatively different value from additional homogeneous self-generated data, and for models near their self-exploration ceiling, this difference can be the difference between improvement and degradation.

Llama3.2-1B-Instruct and Llama3.2-3B-Instruct

Table 7 (Appendix F) reports results for same-family heterogeneous-size models from the Llama family. HACPO improves the 1B model from 21.8% (GSPO) to 23.3% average (+1.5 points) and the 3B model from 35.1% to 37.0% (+1.9 points). These gains are smaller in absolute terms than the Qwen heterogeneous-size results but represent comparable relative improvements (6.9% and 5.4% relative, respectively).

The 1B model's improvement is driven by AMC23 (12.5% to 20.0%, +7.5 points) and modest gains on MATH-500 (35.0% to 35.0%, flat), MATH (34.6% to 35.2%, +0.6 points), and AIME2025 (2.1% to 2.2%, +0.1 points). The 3B model shows improvements on MATH-500 (51.2% to 52.2%, +1.0 point), AMC23 (22.5% to 27.5%, +5.0 points), AIME2025 (5.4% to 6.7%, +1.3 points), and Olympiad (17.0% to 18.8%, +1.8 points).

The GSPO×2 baseline for the 1B model (22.7%) falls between GSPO (21.8%) and HACPO (23.3%), suggesting that both additional self-data and cross-agent data help. For the 3B model, GSPO×2 (33.4%) is worse than GSPO (35.1%), reinforcing the pattern that models near their self-exploration limit benefit more from heterogeneous data than from doubled self-data.

Qwen3-1.7B-Base and Llama3.2-1B-Instruct

Table 7 (Appendix F) tests heterogeneous model collaboration between similarly-sized but architecturally different models (1.7B Qwen vs. 1B Llama). HACPO improves the Qwen3-1.7B from 46.7% (GSPO) to 49.6% average (+2.9 points) and the Llama3.2-1B from 21.8% to 22.8% (+1.0 point). The asymmetric benefit—the stronger model gains more—is opposite to the pattern in other heterogeneous model settings, though the Llama model's smaller gain may partly reflect its very low baseline performance (near-floor effects on hard benchmarks).

For the Qwen model, improvements are concentrated on AIME2025 (14.8% to 22.0%, +7.2 points—a 49% relative gain) and Minerva (27.2% to 30.5%, +3.3 points), with smaller gains on MATH-500 (64.8% to 67.6%, +2.8 points) and MATH (64.1% to 66.1%, +2.0 points). GSM8K improves from 82.6% to 83.8% (+1.2 points). For the Llama model, the improvement is driven by MATH (34.6% to 36.8%, +2.2 points), while GSM8K drops from 52.3% to 53.3% (+1.0 point), and AMC23 remains flat at 12.5%.

The training curves in Figure 3 provide dynamic context for these final results. Across all three settings, HACPO curves diverge from GSPO curves early in training and maintain or widen the gap throughout. This suggests that the benefit of cross-agent data is not just a better initialization or a one-time boost, but a sustained advantage that compounds as training progresses. The curves are notably smoother in the heterogeneous size setting (Figure 3b) than in the heterogeneous state setting (Figure 3a), which may indicate that base-to-base collaboration is more stable than base-to-instruct collaboration—perhaps because instruct-tuned models have more peaked output distributions that create larger importance ratio discrepancies.

Ablation Studies and Robustness Checks

Agent-Capability-Aware Advantage Estimation (Table 2): Removing this module and falling back to standard GSPO advantage estimation (each agent computes baseline from only its own G=8 responses) degrades the 1.7B model from 49.3% to 46.5% average (−2.8 points) and the 4B model from 60.1% to 58.6% (−1.5 points) on the Qwen3-1.7B/4B-Base pair. The degradation is asymmetric—the weaker model suffers more—consistent with the theoretical argument that weaker models benefit more from the reduced-variance joint baseline (the 1.7B model's own 8 responses per prompt give a noisier baseline estimate than the pooled 16 responses across both agents). The 1.7B model's performance on AIME2025 drops dramatically from 22.5% to 12.6% (−9.9 points), suggesting the joint baseline is particularly important for hard problems where the weak model's self-generated rewards have high variance (most responses score 0, making the self-only baseline estimate highly unstable).

Model Capabilities Discrepancy Coefficient (Table 3): Removing the gradient modulation factor $\omega_t^{(j,k)}$ from Equation 10 (setting it to 1.0 for all cross-agent responses while keeping capability ratios for baseline calibration) reduces the 1.7B model from 49.3% to 46.2% average (−3.1 points) and leaves the 4B model essentially unchanged (60.1% to 60.0%). The 1.7B model's AIME2025 drops from 22.5% to 10.5% (−12.0 points)—an even larger drop than removing the advantage estimator. This is a striking finding: for the weaker agent, the gradient amplification from the stronger agent's responses is the single most important mechanism. Without it, the 1.7B model fails to capitalize on the 4B model's high-quality trajectories, particularly on hard problems where the 4B model's correct solutions are the only source of positive training signal. The 4B model's robustness to removing this mechanism reflects the reverse dynamic: the 1.7B model's responses are lower quality, so attenuating their gradients (what the coefficient does) or not (the ablation) doesn't dramatically change the 4B model's update direction.

Exponential Importance Sampling—alpha sensitivity (Table 4): The paper sweeps $\alpha \in \{0.0, 1.0, 2.0, 3.0\}$ on two model pairs, reporting MATH-500 accuracy as the metric.

For Qwen3-1.7B/4B-Base: The 1.7B model peaks at $\alpha = 3.0$ (66.8%) followed by $\alpha = 1.0$ (66.4%), with $\alpha = 0.0$ giving the worst result (63.0%)—a 3.8 percentage point gap between best and worst. The 4B model peaks at $\alpha = 1.0$ (79.2%), with $\alpha = 0.0$ again worst (75.6%)—a 3.6 point gap. The pattern suggests that some conservativeness ($\alpha > 0$) is universally beneficial for cross-agent learning, but the optimal degree depends on the agent pair.

For Qwen3-4B/8B-Base: Both models show minimal sensitivity to $\alpha$. The 4B model varies between 77.0% ($\alpha = 2.0$) and 77.6% ($\alpha = 1.0$ and $\alpha = 3.0$)—only a 0.6 point range. The 8B model varies between 76.4% ($\alpha = 0.0$) and 77.8% ($\alpha = 3.0$)—a 1.4 point range. This suggests that when both models are from the same family and have similar training objectives (both base models), the distribution shift is small enough that the exponential reweighting is less critical. The mechanism matters most when the collaborating agents have substantially different output distributions, such as base vs. instruct models or models from different families.

The paper's Table 5 reveals the $\alpha$ values used in the main experiments: $\alpha = 3.0$ for Qwen3-4B + 4B-Instruct and Qwen3-4B-Base + 8B-Base; $\alpha = 1.0$ for all other combinations. The choice of $\alpha = 3.0$ for the instruct variant makes sense given the instruct model's more concentrated output distribution (it's been fine-tuned to produce specific response formats), which likely creates larger importance ratio discrepancies with the base model.

Stepwise Clipping (Figure 4): The ablation compares three variants on the Qwen3-4B/8B-Base pair: (1) Full HACPO with stepwise clipping ($\delta = 0.8$, $\delta_{\text{step}} = 0.025$), (2) no Clip (asymmetric bounds removed entirely, cross-agent ratios used raw), and (3) no Stepwise (static asymmetric bound $[0.2, 1.0]$ without per-minibatch tightening). The results are presented as training curves in Figure 4:

  • no Clip causes severe instability: the reward curves oscillate wildly for both the 4B and 8B models, with the 4B model showing particularly erratic behavior including sharp downward spikes. This confirms that unbounded cross-agent importance ratios can catastrophically destabilize training, as predicted by the paper's distribution-shift analysis.
  • no Stepwise converges but underperforms full HACPO: both models' reward curves are below the full HACPO curves, with the gap widening in later stages of training. This validates that the stepwise tightening schedule provides value beyond the asymmetric bounds alone—it prevents cross-agent responses from dominating late-batch updates when the policy has drifted furthest from its state at the start of the step.
  • Full HACPO achieves the highest and most stable reward curves for both models, confirming that both the asymmetric bounds and the stepwise schedule are necessary.

The paper notes that the full HACPO curves for this ablation were run for more than one epoch to observe long-term stability effects, whereas main experiments use one training epoch.

Additional ablations and configurations (Appendix A, Table 5): The paper reports per-experiment hyperparameter choices: $\delta$ (clipping lower bound) varies from 0.8 to 0.9, and $\delta_{\text{step}}$ varies from 0.01 to 0.025. The most common configuration is $\alpha = 1.0$, $\delta = 0.8$, $\delta_{\text{step}} = 0.025$, used for 3 of 6 model combinations. The Qwen3-4B + 4B-Instruct combination uses $\alpha = 3.0$, $\delta = 0.8$, $\delta_{\text{step}} = 0.01$—a smaller per-step tightening rate, possibly because the instruct model's policy is more stable across mini-batch updates (instruction-tuned models may have more peaked distributions that change less with each gradient step). The Llama3.2-1B-Instruct + 3B-Instruct combination uses $\delta = 0.9$ (a tight lower bound of 0.1, meaning cross-agent ratios below 0.1 are clipped) with $\delta_{\text{step}} = 0.01$—the most conservative clipping configuration, possibly reflecting larger inter-model distribution gaps in the Llama family.

Tokenizer incompatibility handling: The heterogeneous model experiments (Qwen + Llama pairs) require retokenization at each step, which the paper states is handled by detokenizing to text and retokenizing with the target model's tokenizer. No ablation is provided that isolates the effect of retokenization on importance ratio accuracy or overall performance, which would be informative for understanding whether the modest gains in these settings (1.9–3.9 points) are limited by retokenization noise or by fundamental limits of cross-family knowledge transfer.

Critical Assessment

Claim: HACPO improves performance by an average of 3.3% over GSPO

This claim is supported by the reported numbers in Table 1, but the averaging across the three main experimental settings requires careful examination. From the Table 1 average columns:

  • Heterogeneous state: HACPO for 4B is 75.5% vs. GSPO 68.4% (+7.1), for 4B-Instruct is 81.3% vs. 79.9% (+1.4). Average improvement for this setting: +4.25 points.
  • Heterogeneous size: HACPO for 1.7B is 49.3% vs. GSPO 46.7% (+2.6), for 4B is 60.1% vs. 57.8% (+2.3). Average improvement: +2.45 points.
  • Heterogeneous model: HACPO for Qwen3-4B is 59.7% vs. GSPO 57.8% (+1.9), for Llama3.2-3B is 39.0% vs. 35.1% (+3.9). Average improvement: +2.9 points.

The overall average across all six agents is roughly (7.1 + 1.4 + 2.6 + 2.3 + 1.9 + 3.9) / 6 ≈ 3.2 percentage points, which is consistent with the reported 3.3%. However, this masks enormous heterogeneity: the weakest agents in each pairing (4B in heterogeneous state, 1.7B in heterogeneous size, Llama3.2-3B in heterogeneous model) gain substantially more (1.9–7.1 points) than the stronger agents (1.4–2.3 points). The 4B-Instruct model's 1.4-point gain is the smallest, and given the absence of error bars, it's unclear whether this exceeds statistical noise. The claim should be qualified as "HACPO improves weaker agents substantially (2–7 points) and stronger agents modestly (1–2 points), with an unweighted average of approximately 3.3%."

Claim: HACPO uses only half the rollout cost

The paper states that HACPO "outperforms GSPO by an average of 3.3% while using only half the rollout cost." This comparison needs unpacking. HACPO with two agents uses G=8 rollouts per prompt per agent, for a total of 16 rollouts across the system. Single-agent GSPO uses G=8 rollouts—same per-agent cost. The "half the rollout cost" claim cannot be about per-agent cost (they're equal). It must be about total system cost per unit of performance improvement: to achieve the same average accuracy, GSPO would need more rollouts per agent (as demonstrated by the GSPO×2 baseline, which uses G=16 per prompt but still underperforms HACPO). The paper's phrasing is ambiguous and potentially misleading—it should specify "half the rollout cost compared to the resource-equivalent GSPO×2 baseline" or frame it as "matching or exceeding the performance of GSPO with double the rollouts." A fairer summary would be: HACPO achieves superior performance to GSPO×2 (which uses twice the sampling budget per agent) while spending the same per-agent sampling budget as standard GSPO.

Claim: HACPO enables bidirectional mutual learning between heterogeneous agents

The results consistently support bidirectional benefit: in every experimental setting, all agents improve under HACPO compared to their single-agent GSPO baselines. However, the asymmetry of benefits reveals an important nuance. In the heterogeneous state setting, the weaker 4B model gains 7.1 points while the stronger 4B-Instruct model gains only 1.4 points—the learning is bidirectional but heavily asymmetric. The paper briefly acknowledges this in Section 5.1, attributing the stronger model's smaller gains to the weaker model providing complementary exploration signals, but doesn't quantify how much of the benefit comes from correct solutions vs. informative errors from the weaker model. An ablation that only shares correct responses (or only shares incorrect responses) between agents would clarify the mechanism, but this experiment is not reported.

Claim: All four mechanisms are necessary

The ablation studies in Tables 2–3 and Figure 4 provide evidence for this claim, but with an important caveat: each ablation removes one mechanism while keeping the other three. The paper does not report ablations that remove combinations of mechanisms (e.g., removing both the advantage estimator AND the capability coefficient), so we cannot assess whether mechanisms are partially redundant or whether their benefits are additive vs. multiplicative. The Figure 4 ablation for stepwise clipping also tests more than one epoch of training (unlike the main experiments), making it unclear whether the stepwise clipping benefit is specific to longer training or general.

Genuine weaknesses in the experimental design

Single training run per configuration. The paper reports no error bars, standard deviations, confidence intervals, or multiple seeds for any result. This is a significant limitation, particularly given the small sizes of several benchmarks (AIME2025: 30 problems, AMC23: 40 problems). Small benchmark sizes amplify the variance of single-run evaluations, and the reported gains of 1–2 percentage points on some benchmarks may fall within the range of run-to-run variation. For a paper making algorithmic claims, the absence of any statistical rigor is a notable weakness.

The 3.3% average claim aggregates across settings and benchmarks with no variance information. The average is an unweighted mean across seven benchmarks of vastly different sizes (from 30 to 5,000 problems) and across three heterogeneity settings. A weighted average or per-setting breakdown would be more informative. Without variance estimates, we cannot assess whether the 3.3% is a robust central tendency or an artifact of averaging noisy measurements.

No baselines from other collaborative learning paradigms. The paper argues that HACRL is fundamentally different from MARL and KD, but it doesn't compare against adapter-based methods where a GRPO-trained model is used as a teacher for distillation. A comparison against standard knowledge distillation from a stronger to a weaker agent (even if unidirectional) would contextualize HACPO's bidirectional gains—how much of the weak agent's improvement comes from having a stronger collaborator (which KD would also provide) versus from the bidirectional mechanism? Similarly, the paper doesn't compare against an ensemble-based approach where multiple independently-trained models are simply ensembled at inference time.

The GSPO×2 baseline doesn't fully control for data diversity. GSPO×2 doubles rollouts from the same policy distribution. A stronger baseline would be "GSPO with doubled rollout budget but sampled at higher temperature to increase diversity"—this would test whether HACPO's benefit comes from heterogeneous architectures specifically or simply from increased response diversity, regardless of source. The paper doesn't run this experiment.

Missing ablations on capability estimation. The sliding window size K=5 is used in all experiments, but no sensitivity analysis is provided. Would K=1 (using only the current batch, violating the independence assumption) cause measurable degradation? Would K=20 (slower adaptation) hurt because capability estimates lag behind actual improvements? This ablation would directly test the theoretical tradeoff discussed in Remark D.2.

Hyperparameter sensitivity not fully characterized. Table 4 shows alpha sensitivity for two model pairs using only MATH-500, not the full benchmark suite. The paper's Table 5 reveals different hyperparameters for different settings, but doesn't explain how these were chosen (grid search? manual tuning? single configuration tried?). Without this information, it's impossible to assess whether HACPO's gains require careful per-setting hyperparameter tuning—a practical limitation for deployment.

No latency or wall-clock analysis. HACPO requires computing cross-agent importance ratios, which involves forward passes through each agent's model for every other agent's responses. For two agents with G=8 responses each, this means 16 forward passes per agent per prompt (8 for self-generated responses + 8 for cross-agent responses). Single-agent GSPO requires 8 forward passes. The paper measures compute in "rollouts" but doesn't report actual training time or GPU-hours, making the "half the rollout cost" claim difficult to translate into practical resource savings. If cross-agent forward passes are computationally equivalent to self-generated forward passes, HACPO's per-agent compute is actually double standard GSPO's—the efficiency gain comes from better use of the sampling budget, not from reduced total computation.

Claims that would benefit from additional experiments

Generalization to non-math domains. All experiments are on mathematical reasoning. The paper's mechanisms (capability ratios, importance sampling corrections) are domain-agnostic, but whether the benefits generalize to code generation, factual QA, or open-ended generation is untested. Math has a particularly clean verifiable reward signal (exact answer matching), which may amplify the benefit of cross-agent data sharing—if rewards are noisy or subjective, the miscalibration problems that HACPO fixes might be dwarfed by reward noise.

Scaling to more than two agents. All experiments use exactly two agents. The paper's formalization supports n agents, and the sample efficiency argument becomes stronger with more agents (each rollout is reused n times). Whether the four mechanisms scale gracefully—whether the capability ratio estimation remains stable with many agents at different capability levels, whether importance ratio variance compounds with more agents—is completely unexplored.

Longer training. Main experiments use one training epoch. The stepwise clipping ablation (Figure 4) was run longer than one epoch, but no main results are reported for multi-epoch training. Whether HACPO's benefits saturate, continue to compound, or reverse with extended training is unknown. The paper's theoretical results don't predict a time-dependence of benefits, but empirical saturation is common in RL training.

Interaction between the four mechanisms. The ablation studies test each mechanism in isolation. Do mechanisms interact synergistically? If you remove the advantage estimator, does the stepwise clipping become more or less important? A factorial ablation design would characterize these interactions but is not reported.

6. Limitations and Trade-offs

No Statistical Rigor: Single-Run Evaluations Without Variance Estimates

The assumption or constraint. The paper reports all experimental results—both main comparisons (Table 1, Figure 3) and ablations (Tables 2–4, Figure 4)—from single training runs with no error bars, standard deviations, confidence intervals, or multiple-seed replication. Nowhere in the main text or appendices does the paper mention running experiments with different random seeds, nor does it report any measure of statistical dispersion for any metric.

The consequence. Without variance estimates, it is impossible to determine whether the reported improvements are statistically reliable or within the range of run-to-run noise. This is particularly acute for three reasons. First, several evaluation benchmarks are extremely small: AIME2025 has only 30 problems, and AMC23 has only 40 problems. A difference of 2–3 correct answers on AIME2025 translates to 6.7–10.0 percentage points—comparable in magnitude to many of HACPO's reported gains (e.g., the 4B model improves from 48.5% to 62.2% on AIME2025, a 13.7-point gain that could arise from getting 4 more problems correct out of 30). Second, the headline "3.3% average improvement" aggregates across seven benchmarks of vastly different size (from 30 to 5,000 problems) and across six agents in three settings with no weighting, making it sensitive to outliers on small benchmarks. Third, the paper reports some counterintuitive patterns—for example, the 1.7B model's GSM8K accuracy dropping from 82.6% (GSPO) to 82.2% (HACPO) while AIME2025 jumps from 14.8% to 22.5% (Table 1, middle section). Without variance information, we cannot distinguish genuine tradeoffs (the model improving on hard problems at the expense of easy ones) from sampling noise.

What evidence exists in the paper. The paper provides no statistical evidence whatsoever. The training curves in Figure 3 are single-run traces. The ablation studies in Tables 2–4 are single-run point estimates. The stepwise clipping ablation in Figure 4 shows training dynamics for a single run per variant. This is a methodological gap that the paper does not acknowledge, let alone address.

Mitigation status. Not addressed. The paper contains no discussion of statistical significance, no recommendation for multiple seeds, and no acknowledgment that single-run evaluations on benchmarks as small as 30–40 problems may produce unreliable point estimates. For a paper making algorithmic claims intended for practical deployment, this is a consequential omission.

Difficulty Estimation for Capability Ratios Is Fully Oracle-Dependent and Exam-Specific

The assumption or constraint. The capability ratio $\omega_t^{(k,j)}$—the central mechanism enabling all four of HACPO's corrective components—is computed from per-batch mean rewards $P_t^{(k)}$ (Equation 9), which in turn require a verifiable reward function that can evaluate every response as correct or incorrect. The paper's experiments exclusively use mathematical reasoning benchmarks (MATH, GSM8K, AIME2025, AMC23, Minerva, Olympiad), all of which provide ground-truth answers that enable exact-match verification.

The paper does not discuss, test, or even acknowledge that this verification requirement is domain-specific. Section 2.2 defines the shared reward function $R(\cdot)$ as applying to every response, and Appendix A describes answer verification using the grading function from Lightman et al. (2022)—a procedure that checks whether a model's final answer string matches a known ground-truth solution. This is only possible for tasks with closed-form, verifiable answers.

The consequence. HACPO's capability ratio estimation—and therefore the entire collaborative training framework—cannot be directly applied to open-ended generation tasks where correctness is ambiguous, multi-dimensional, or subjective. This includes dialogue, creative writing, summarization (where quality is continuous and multi-faceted), code generation for tasks without unit tests, and instruction-following where "correctness" depends on human judgment rather than exact matching. In these domains, the per-batch mean reward would need to come from a learned reward model or human feedback, introducing noise and potential bias into the capability ratio estimates. If the reward model systematically overestimates one agent's outputs relative to another's (e.g., due to distribution shift in the reward model's training data), the capability ratios would be miscalibrated, and Theorem 4.1's unbiasedness guarantee would break down because the "reward" being averaged no longer reflects true response quality in an unbiased way.

Even within mathematical reasoning, the requirement for exact-match verification imposes practical constraints. The paper uses the MATH dataset's ground-truth answers during training. In many real-world scenarios, organizations may want to collaboratively train models on proprietary problem sets where ground-truth answers are not available in the same clean format (e.g., problems scraped from forums, internally authored questions without formalized answer keys). Building the verification infrastructure is itself a non-trivial engineering cost that the paper's "half the rollout cost" efficiency claim does not account for.

What evidence exists in the paper. The paper provides no evidence on this limitation because it is entirely unaddressed. All experiments use exact-match verification on math benchmarks. There is no experiment with learned reward models, no test on code generation with unit tests (which would be a natural extension within the verifiable-reward paradigm), and no discussion of how capability ratio estimation would change under noisy or learned reward signals.

Mitigation status. Not addressed. The paper implicitly assumes access to a perfect verifiable reward function throughout—it is baked into the problem formalization (Definition 2.5) without qualification about what kinds of tasks admit such functions. Appendix B.1 references RLVR as a broader paradigm but does not discuss reward noise. This is a domain-generality limitation that significantly constrains the deployment scope of HACPO as presented.

Difficulty Estimation Cost Is Unaccounted for in the Efficiency Claim

The assumption or constraint. While not explicitly labeled as "difficulty estimation" in the same way as the example paper (which generates 2048 samples per question), HACPO has its own unaccounted cost: the computation of capability ratios $\omega_t^{(k,j)}$ requires tracking per-batch mean rewards $P_t^{(k)}$ over a sliding window, which in turn requires evaluating the verifiable reward on every response from every agent at every training step. This is standard practice in RLVR (rewards are always computed), so it is not an additional cost in the RLVR context. However, there is a subtler cost: computing the cross-agent importance ratios $s_{t,i}^{(k,j)}$ (Equation 11) requires a forward pass of agent $k$'s current policy on agent $j$'s responses. For an $n$-agent system with each agent generating $G$ responses per prompt, this means $n \cdot (n-1) \cdot G$ additional forward passes per training step beyond the $n \cdot G$ forward passes for self-generated responses.

The paper measures compute in "rollouts" (generations) and claims HACPO "uses only half the rollout cost" compared to GSPO×2. But this accounting considers only the sampling cost (generating responses), not the scoring cost (computing cross-agent importance ratios via forward passes). In single-agent GSPO, each training step requires $G$ forward passes for self-generated responses. In two-agent HACPO, each agent requires $G$ forward passes for its own responses plus $G$ forward passes for the other agent's responses—double the per-agent forward-pass cost of standard GSPO, even though the sampling cost is identical.

The consequence. The paper's efficiency framing is incomplete. A practitioner reading "half the rollout cost" might reasonably conclude that HACPO reduces total training compute. In reality, HACPO increases per-agent forward-pass computation by a factor of $n$ (for n agents), while keeping sampling cost constant. Whether the net effect is a compute reduction depends on the relative cost of autoregressive generation (sampling) versus forward passes (scoring). For long-response tasks where generation time dominates, HACPO's efficiency gain is real because sampling cost dwarfs scoring cost. For short-response tasks or when generation is highly optimized (e.g., speculative decoding, batched inference), the additional forward-pass overhead may partially or fully offset the sampling efficiency gain. The paper provides no wall-clock measurements, GPU-hour counts, or FLOPs analysis to quantify this tradeoff.

For heterogeneous model pairs with incompatible tokenizers (Definition 2.3), the cost is even higher because cross-agent importance ratios require detokenization followed by retokenization (Section 3.3). This adds a round-trip through the tokenizer for every cross-agent response, which is computationally cheap per response but adds up across thousands of training steps with thousands of responses per step.

What evidence exists in the paper. The paper provides no evidence on actual compute costs. Appendix A states that experiments were conducted on eight GPUs but reports no training time, no GPU-hours, and no throughput measurements. The "rollout" accounting is purely in terms of generation count, not FLOPs or wall-clock time.

Mitigation status. Not addressed. The paper does not acknowledge this distinction between sampling cost and total compute cost anywhere. The term "half the rollout cost" appears in the abstract and Section 5 without qualification. A more accurate characterization would be "half the sampling budget with additional forward-pass computation that scales linearly with the number of agents."

All Experiments on a Single Domain (Mathematical Reasoning) and a Narrow Set of Model Families

The assumption or constraint. Every experiment in the paper—all main results (Table 1), all additional results (Table 7), all ablations (Tables 2–4, Figure 4), and all training dynamics (Figure 3)—uses exclusively mathematical reasoning benchmarks. The training data is 7.5k MATH questions. The evaluation spans seven benchmarks, all math-focused: MATH-500, MATH, GSM8K, AIME2025, AMC23, Minerva, and Olympiad. The paper states in Section 5 that it "adopts 7.5k high quality math questions from the MATH dataset for training," but never addresses whether the findings are expected to generalize beyond mathematical reasoning.

The model families are limited to Qwen3 (Yang et al., 2025a) and Llama3.2 (Grattafiori et al., 2024). While this covers two distinct architectures with different tokenizers (tested in the heterogeneous model setting), it is still a narrow slice of the LLM landscape. Proprietary models (GPT-4, Claude), encoder-decoder architectures, and models with substantially different pretraining objectives (e.g., code-specialized models) are absent.

The consequence. The paper's claims are, strictly speaking, supported only for mathematical reasoning with Qwen3 and Llama3.2 models. Whether HACPO's mechanisms transfer to other domains depends on several untested factors:

  • Reward signal characteristics. Mathematical reasoning provides sparse, binary, noise-free rewards (answer is either exactly correct or not). Tasks with continuous or noisy rewards (e.g., RLHF with a learned reward model, code generation where partial credit is possible) may produce noisier capability ratio estimates, potentially destabilizing the baseline calibration and gradient modulation mechanisms.
  • Response structure. Math solutions have a characteristic structure (step-by-step reasoning leading to a final answer) that may influence importance ratio behavior. The geometric mean normalization in sequence-level importance sampling ($1/|y|$ exponent) assumes token-level probability ratios are meaningful to average—this may hold differently for free-form dialogue, creative writing, or code with highly variable structure.
  • Capability distribution. The paper's experiments span models from 1B to 8B parameters, all within roughly the same capability tier. If one agent is dramatically stronger (e.g., a 70B model collaborating with a 1B model), the capability ratios $\omega_t^{(k,j)}$ could be extreme (e.g., 0.01 or 100), and the importance ratio distribution could be even more skewed than what Appendix C reports. The paper provides no evidence on how HACPO scales with larger capability gaps.

The paper's findings about asymmetric benefits (weaker agents gain more than stronger ones) might also be domain-specific. In mathematical reasoning, a stronger model's correct solutions provide clean positive training signals for a weaker model. In domains where "correctness" is less well-defined, the stronger model's outputs might not be unambiguously better—they might reflect a different style or set of priorities rather than higher quality per se.

What evidence exists in the paper. The paper provides no cross-domain evidence. There is no experiment on code generation (where verifiable rewards via unit tests would be a natural extension), no experiment on factual QA, and no experiment on any non-reasoning task. This is a scope limitation that the paper does not acknowledge, let alone discuss.

Mitigation status. Not addressed. The paper's abstract, introduction, and conclusion all present HACRL and HACPO as general frameworks without qualifying that the experimental validation is limited to mathematical reasoning. A single sentence acknowledging the domain limitation and suggesting future work on broader evaluation would significantly improve transparency.

The Resource-Equivalent Baseline (GSPO×2) Is a Weak Counterfactual for Cross-Agent Diversity

The assumption or constraint. The paper's primary argument for cross-agent data providing value beyond mere data volume rests on HACPO outperforming GSPO×2—a single agent trained with double the rollouts (G=16 vs. G=8) and double the mini-batch updates (achieved by halving mini-batch size). This baseline controls for data volume but does not control for data diversity from a single source. GSPO×2 doubles samples from the same policy distribution, which may suffer from diminishing returns if the policy's output distribution is narrow.

A stronger baseline would test whether HACPO's benefit comes specifically from heterogeneous architectures versus simply from increased response diversity, regardless of source. An appropriate counterfactual would be "GSPO with doubled rollout budget but sampled at higher decoding temperature" or "GSPO with doubled rollout budget and an explicit diversity-promoting mechanism (e.g., nucleus sampling with a lower top-p threshold)." These baselines would help distinguish whether HACPO's gains arise from the specific mechanism of heterogeneous policy distributions or from the more generic mechanism of increased exploration diversity.

The consequence. The paper cannot rule out the possibility that HACPO's gains could be matched or exceeded by simply running single-agent GSPO at a higher sampling temperature to encourage more diverse self-generated rollouts. If a temperature-tuned GSPO baseline achieved comparable performance to HACPO, the paper's central claim—that heterogeneous cross-agent collaboration provides unique value—would be significantly weakened. The efficiency argument would shift from "heterogeneous collaboration is more efficient than homogeneous scaling" to "heterogeneous collaboration is more efficient than naive homogeneous scaling, but may be matched by diversity-enhanced homogeneous scaling at lower engineering complexity."

This concern is partially supported by the paper's own data. In settings where the collaborating agents are more similar (same-family base models, as in Table 1's middle section), the GSPO×2 baseline performs closer to HACPO (47.5% vs. 49.3% for 1.7B, 57.5% vs. 60.1% for 4B) than in settings where models are more different (base vs. instruct, 68.4% vs. 75.5% for 4B). This pattern suggests that what matters is the diversity gap between self-generated and cross-agent rollouts—and if that gap can be reduced by increasing self-generation diversity, the cross-agent advantage may shrink.

What evidence exists in the paper. The paper provides no diversity-controlled baseline. There is no experiment varying sampling temperature for GSPO×2, no comparison with nucleus sampling or other diversity-promoting decoding strategies, and no measurement of response diversity (e.g., n-gram novelty, semantic embedding dispersion) for either self-generated or cross-agent rollouts. The paper does not acknowledge this as a confound.

Mitigation status. Not addressed. The GSPO×2 baseline is treated as the definitive resource-equivalent comparison, and the possibility that its underperformance reflects insufficient self-generated diversity rather than the inherent value of cross-agent data is not discussed. The paper's conclusion that "cross-agent data provides qualitatively different value from additional homogeneous self-generated data" is asserted based on GSPO×2 comparisons without ruling out alternative explanations for GSPO×2's weaker performance.

No Scaling Beyond Two Agents, One Epoch, or One Capability Tier

The assumption or constraint. The paper's experimental scope is narrow along three critical scaling dimensions:

  • Number of agents: All experiments use exactly two collaborating agents. The formalization in Definition 2.5 and Algorithm 1 supports arbitrary $n$, and the sample efficiency argument becomes stronger with more agents (each rollout is reused $n$ times instead of 2), but this is never tested. Would capability ratio estimation remain stable with 5, 10, or 50 agents at different capability levels? Would the importance ratio variance compound as $n$ grows? Would the stepwise clipping schedule need adjustment?
  • Training duration: Main experiments use exactly one training epoch over the 7.5k MATH training questions. Appendix A states this explicitly. Only the stepwise clipping ablation (Figure 4) runs longer than one epoch. Single-epoch training is common in RLVR (to avoid overfitting to the limited training data), but it means the paper provides no evidence on whether HACPO's benefits saturate, continue to compound, or reverse with extended training. The training curves in Figure 3 show HACPO above GSPO at the end of training, but the curves have not plateaued—it's unclear whether the gap would widen or narrow with additional epochs.
  • Capability range: All models range from 1B to 8B parameters with base MATH-500 accuracies of 17.6% (Llama3.2-1B) to 93.8% (Qwen3-4B-Instruct). The paper provides no evidence on how HACPO behaves with larger capability gaps—e.g., a 0.5B model collaborating with a 70B model—or with models at very different capability tiers (a model that gets 2% accuracy collaborating with one that gets 95%). Extreme capability ratios could cause the weighting coefficients $\omega_t^{(k,j)}$ to approach zero or infinity, potentially breaking the gradient alignment guarantee of Theorem 4.3.

The consequence. These scope limitations mean the paper provides existence proof that HACPO works in a specific regime (2 agents, 1 epoch, moderate capability gaps, Qwen/Llama models, math reasoning) but offers no guidance on how to extrapolate beyond this regime. A practitioner deploying HACPO with 5 agents of varying sizes training over multiple epochs on code generation would be operating entirely outside the validated parameter space. The theoretical results provide some guidance—Theorem 4.3's Assumption D.12 (positive competence alignment) should hold as long as collaborating agents are "competent" rather than adversarial—but do not predict how the empirical benefits scale with $n$, training duration, or capability gap size.

Specific failure modes at larger scales include: (a) the capability ratio sliding window ($K=5$) might need adjustment for slower or faster training dynamics with more agents; (b) the stepwise clipping schedule ($\delta_{\text{step}}$) might need recalibration as the number of mini-batches per step grows with more agents; (c) cross-agent importance ratio distributions might become multi-modal with many agents, making the exponential reweighting (designed for a single cross-agent pair) insufficient.

What evidence exists in the paper. The paper provides no multi-agent scaling experiments, no multi-epoch experiments, and no experiments with models outside the 1B–8B range. These are acknowledged as absences rather than investigated limitations—they are not mentioned anywhere in the paper.

Mitigation status. Not addressed. The paper's title uses "Heterogeneous Agent Collaborative Reinforcement Learning" (singular "Agent" could refer to the paradigm, but the plural is implicit in "collaborative"), and the abstract and introduction frame the approach as general. The discrepancy between the general framing and the narrow experimental validation is not discussed. To the paper's credit, the theoretical analysis (Theorem 4.1, 4.3) is general and does not assume a specific number of agents, which provides some conceptual support for scaling. But the theory assumes conditions (independence of capability ratios from current-batch rewards, positive competence alignment) that become harder to verify and maintain as the system scales.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a new problem setting rather than merely a new algorithm, and this conceptual reframing is its primary contribution to the field. Before HACRL, the landscape of multi-agent LLM training was partitioned into two non-overlapping paradigms: Multi-Agent Reinforcement Learning (MARL), where agents coordinate at inference time to jointly solve tasks, and Knowledge Distillation (KD), where a fixed teacher unidirectionally transfers knowledge to a student. Neither paradigm addresses the scenario that the paper argues is far more common in practice: multiple heterogeneous agents are independently deployed at inference time but could, in principle, share their training rollouts to mutual benefit.

The paper's Figure 1 encodes this reframing in a single diagram—MARL agents form an interconnected inference graph, KD agents form a directed teacher-to-student arrow, and HACRL agents share training data bidirectionally while executing independently. This visual distinction is not cosmetic. It represents a fundamentally different design constraint: in HACRL, the algorithm designer optimizes for training-time knowledge transfer while ensuring the resulting single-agent policy remains self-sufficient at deployment. In MARL, the designer optimizes for inference-time coordination—a different objective with different failure modes. By cleanly separating these concerns, HACRL opens a design space that was previously invisible, where algorithms can be judged by how effectively they translate heterogeneous training data into improved single-agent performance without requiring coordinated deployment.

The magnitude of this shift is mid-level: it is not a paradigm overthrow (single-agent RLVR with GSPO/GRPO remains viable and simpler when only one agent exists), but it is considerably more than an incremental refinement. It redefines what "multi-agent" means in the context of LLM training—shifting the emphasis from joint execution to joint data utilization—and establishes a new axis of comparison (collaborative vs. isolated training efficiency) that prior work had no vocabulary for. The paper's negative result—that naive rollout sharing degrades performance by ~10 percentage points across all settings (Table 1, Naive baseline)—serves as a boundary marker that prevents this new setting from being dismissed as trivial. If naive sharing worked, HACRL would be an obvious engineering trick rather than a research contribution requiring tailored algorithmic mechanisms.

The paper also resolves a latent tension in the RLVR literature that had not been explicitly articulated. Several prior works (GRPO, GSPO, DAPO) had progressively improved the stability and efficiency of group-based policy optimization for single agents. The implicit assumption in these works was that cross-agent data, if available, would be either unusable (due to distribution shift) or trivially beneficial (more data is better). The paper's Naive baseline shows both intuitions are wrong: cross-agent data is usable if properly corrected (HACPO improves all agents), but it is catastrophically harmful if uncorrected (Naive degrades all agents). This resolves a question that the field hadn't yet asked systematically: under what conditions does off-policy data from a different model help versus hurt on-policy RL training? The answer, per this paper, is: it helps only when the data is corrected for capability discrepancies (via agent-specific baselines and gradient modulation) and distribution shift (via exponential reweighting and stepwise clipping). Uncorrected, it's worse than no additional data at all.

The paper's identification of capability ratios as a sufficient statistic for heterogeneous collaboration is a methodological contribution that may influence how future systems think about multi-model training. The capability ratio $\omega_t^{(k,j)}$ plays two mathematically complementary roles—baseline calibration (scaling rewards to a common expected-value scale) and gradient modulation (scaling update magnitudes by relative competence)—and the paper demonstrates that both roles are independently necessary (Tables 2 and 3). This dual-use design pattern—where a single estimated quantity corrects for heterogeneity in two orthogonal ways—is elegant and may recur in other domains where heterogeneous components must be coordinated (federated learning with heterogeneous clients, ensemble training with diverse architectures).

Finally, the theoretical analysis (Theorem 4.3, Appendix D.3) provides a boundary condition that the field previously lacked: cross-agent learning is gradient-aligned (and therefore beneficial) when the collaborating agent satisfies positive competence alignment—its confidence correlates with actual response quality. This transforms the question of "will collaboration help?" from an empirical gamble into a testable hypothesis. If a practitioner can verify that agent B's high-probability responses tend to be correct (and its low-probability responses tend to be incorrect), Theorem 4.3 guarantees that agent A will benefit from B's rollouts under HACPO's corrections. If this condition fails—if agent B is confidently wrong—collaboration will harm. This theoretical result shifts the research agenda from "does collaboration work?" (which is model-pair-dependent) to "how can we ensure positive competence alignment?" (which is a design and selection problem).

One direction that becomes less attractive after this paper: developing ever-more-complex single-agent RLVR algorithms that squeeze marginal gains from self-generated rollouts. The paper shows that a 4B model collaborating with a 1.7B model (Table 1, heterogeneous size) achieves 60.1% average accuracy with HACPO versus 57.8% with GSPO—a 2.3-point gain that no amount of single-agent algorithmic refinement (GRPO vs. GSPO vs. DAPO) could match, because those refinements operate within the inherent ceiling of self-generated data. The largest gains come from breaking the data silo, not from optimizing the single-agent objective. This suggests that the field's marginal dollar of research effort may be better spent on collaborative training infrastructure than on incremental single-agent algorithmic improvements.

Follow-Up Research This Work Enables

1. Cheap and online capability estimation to remove the oracle reward dependency. The current capability ratio estimation (Equations 8–9) requires per-batch mean rewards computed from a verifiable reward function—exact-match grading against ground-truth answers. This is the single largest barrier to deploying HACPO outside of mathematical reasoning and other closed-form verification domains. A concrete follow-up would replace the oracle reward $R(y)$ with a learned outcome reward model (ORM) trained on human preference data or LLM-as-judge annotations, then measure how ORM noise propagates into capability ratio estimates. The key experiment: train an ORM on the MATH training set, use its continuous scores (0–1) in place of binary exact-match rewards for capability estimation, and compare HACPO's performance with ORM-based vs. oracle-based capability ratios. If performance degrades gracefully with ORM noise, HACPO extends to any domain with a reasonable reward model. If it degrades catastrophically (because capability ratio estimation requires low-noise rewards), that would establish a sharp boundary condition for HACRL's applicability and motivate research into noise-robust capability estimators—perhaps using Bayesian filtering (Kalman filters on the sliding window) or learned capability predictors that take only the prompt text and model identifier as input.

2. Agent selection and competence alignment verification before collaborative training. Theorem 4.3's Assumption D.12 (positive competence alignment) is the theoretical linchpin of HACPO's gradient consistency proof, but the paper provides no method for verifying whether a given agent pair satisfies this condition before committing to collaborative training. A natural follow-up would develop a pre-training compatibility check: before running full HACPO, sample ~100 responses from each agent on a held-out set, compute the correlation between agent B's sequence-level probability of its own responses and the actual correctness of those responses, and use this correlation as a compatibility score. The hypothesis: agent pairs with higher pre-training probability-correctness correlation will show larger HACPO gains. The experiment: run HACPO with 10 different agent pairs spanning a range of compatibility scores, and measure whether the compatibility score predicts the magnitude of improvement over GSPO. A strong correlation would provide a practical tool for deciding which agents to pair; a weak correlation would suggest that competence alignment is more nuanced than the simple linear correlation captured by Assumption D.12, motivating deeper theoretical analysis.

3. Scaling HACPO to more than two agents and characterizing the benefit curve. The paper's formalization supports $n$ agents, but all experiments use $n=2$. The sample efficiency argument becomes stronger with more agents—each rollout is reused $n$ times instead of 2—but several mechanisms may degrade: capability ratio estimation requires tracking $n(n-1)/2$ pairwise ratios from a shared pool of per-agent mean rewards, and the cross-agent importance ratio distribution may become multi-modal as $n$ grows. A concrete experiment: run HACPO with $n \in \{2, 3, 4, 5\}$ agents of varying sizes (e.g., Qwen3-1.7B, 4B, 8B, 14B, 32B) on MATH, keeping total sampling budget constant per agent (G=8) and measuring both final accuracy and training stability (variance of reward curves). The key question: does the per-agent benefit follow diminishing returns (each additional agent provides less marginal improvement) or does it compound (more agents create richer cross-agent data that accelerates all learners)? If benefits saturate at $n=3$, practical deployments can be much simpler than the general framework suggests. If benefits continue to scale, HACRL becomes a strong argument for open model ecosystems where many independent teams contribute training rollouts to a shared pool.

4. Combining HACPO with diversity-promoting decoding strategies to isolate the source of cross-agent benefit. The paper cannot distinguish whether HACPO's gains come from heterogeneous architectures specifically or from increased response diversity generally, because the GSPO×2 baseline doubles samples from the same narrow policy distribution. A direct test: compare HACPO (two heterogeneous agents, G=8 each) against a diversity-enhanced GSPO baseline where a single agent generates G=16 responses per prompt using nucleus sampling (top-p=0.9) or high temperature (T=1.5) to maximize self-generated diversity, with total compute matched. If diversity-enhanced GSPO matches HACPO, the paper's central claim—that heterogeneous architectures provide unique collaborative value—would be weakened, and the practical recommendation would shift to "increase your sampling diversity" rather than "find a heterogeneous collaborator." If HACPO still outperforms, that would be strong evidence that architectural heterogeneity provides benefits beyond mere output diversity—perhaps because different model families have genuinely different reasoning strategies (different inductive biases) rather than just different sampling noise.

5. Stress-testing HACPO with adversarial or misaligned collaborators. Theorem 4.3 guarantees gradient alignment only when collaborators satisfy positive competence alignment. What happens when this condition is violated? A carefully designed negative experiment would pair a strong agent with a deliberately miscalibrated agent: take the Qwen3-4B-Base model trained normally, and pair it with a version of itself that has been fine-tuned on a corrupted dataset where correct answers are relabeled as incorrect and vice versa. This "adversarial collaborator" would have high confidence (it's the same architecture) but negative competence alignment (it's confidently wrong). The prediction: HACPO should degrade the strong agent's performance compared to single-agent GSPO, and the degradation should be predictable from the competence alignment measure. If HACPO's mechanisms (stepwise clipping, exponential reweighting) provide some robustness even to adversarial collaborators, that would be a surprising and practically valuable finding (resilience to buggy or poorly-trained collaborating agents). If HACPO catastrophically fails, that establishes a clear security boundary: collaborative training should only occur between verified-compatible agents.

6. Extending HACPO to code generation with unit-test-based verifiable rewards. The paper's experiments are exclusively on mathematical reasoning, but code generation provides an equally natural verifiable reward signal via unit test pass/fail. A replication study on HumanEval (164 problems) and MBPP (974 problems) using the same Qwen and Llama model pairs would test domain generalization. Code generation differs from math reasoning in several ways that stress HACPO's mechanisms: code solutions have more variable structure (some are one-liners, others are multi-function programs), unit test rewards are multi-dimensional (passing all tests vs. passing some), and the "reasoning path" is less standardized than mathematical derivations. The prediction: HACPO should provide similar relative gains on code generation if the benefit comes from general cross-agent knowledge transfer, but may show different patterns (e.g., stronger benefit for the agent with weaker coding-specific pretraining) if the benefit is domain-dependent. Additional metrics beyond pass@1—such as pass@10, which measures whether a correct solution exists in the model's top-10 samples—would test whether HACPO improves solution diversity or only solution quality, connecting to the "diversity vs. quality" question from direction 4 above.

Practical Applications and Downstream Use Cases

1. Cost-efficient post-training for organizations deploying models at multiple scales. The paper's heterogeneous size experiments (Table 1, Table 7) demonstrate that a smaller model (1.7B) improves by 2.9 points and a larger model (4B) improves by 2.3 points when trained collaboratively rather than independently—and that the GSPO×2 baseline (doubling self-generated data for a single agent) underperforms HACPO by 1.8–2.6 points. For an organization that deploys both a 4B model (for high-quality batch processing) and a 1.7B model (for low-latency user-facing applications), collaborative training with HACPO provides a concrete recipe: instead of running two independent GSPO training jobs (each consuming G=8 rollouts per prompt, total 16 rollouts), run one HACPO job where both models share rollouts (same total 16 rollouts, but each model sees all 16). The organization gets improved performance on both models (the 4B model for batch processing becomes more accurate, the 1.7B model for user-facing applications becomes more capable) with no additional sampling cost. The only additional cost is the cross-agent forward passes for importance ratio computation—and if the organization is already running both training jobs on the same GPU cluster, these forward passes can be pipelined with existing computation. The 3.3% average accuracy improvement is modest in absolute terms but represents the kind of gain that, in competitive LLM deployment, can determine whether a small model crosses a usability threshold (e.g., 1.7B model going from 46.7% to 49.3% average on math benchmarks might be the difference between "not deployable" and "acceptable for simple queries").

2. Cross-organization model improvement without sharing model weights or training data. The heterogeneous model setting (Qwen + Llama, Table 1 bottom section) demonstrates that HACPO works across entirely different model families with incompatible tokenizers and pretraining corpora. This enables a deployment scenario that is politically and legally significant: two organizations with different proprietary models can collaboratively improve both models by sharing rollouts (text outputs and correctness labels) without sharing model weights, training data, or even tokenizers. Organization A deploys Qwen-based models, Organization B deploys Llama-based models. They agree on a shared task (mathematical reasoning with publicly available MATH training prompts) and a shared verifier (the MATH ground-truth answers). Each organization runs HACPO locally: Organization A's Qwen model generates responses, Organization B's Llama model generates responses, they exchange only the response texts and correctness labels, and each organization computes cross-agent importance ratios locally using their own model's forward passes. Both models improve without either organization revealing their model architecture, weights, or internal training data. This is a privacy-preserving collaborative training paradigm that is not achievable with standard MARL (which requires model access for inference-time coordination) or KD (which requires one organization to cede the teacher role). The Llama3.2-3B model's 3.9-point improvement from collaborating with Qwen3-4B—while its own GSPO×2 baseline degrades performance—is a concrete existence proof that this cross-organizational scenario can provide unique value.

3. Improving small on-device models via collaboration with large cloud models during training only. The paper's asymmetric benefit pattern—weaker agents consistently gain more from collaboration than stronger agents—has direct implications for on-device deployment. A company might train a large cloud model (e.g., 8B parameters) and a small on-device model (e.g., 1.7B parameters) collaboratively using HACPO. The 1.7B model, which will eventually run on users' phones with no cloud connectivity, learns from the 8B model's high-quality rollouts during training. At inference time, the 1.7B model operates completely independently—no cloud calls, no multi-agent coordination, no additional latency. Yet it carries knowledge transferred from the 8B model that it could never have acquired through self-exploration alone (the paper shows the 1.7B model improving from 46.7% to 49.3% average, driven largely by AIME2025 gains from 14.8% to 22.5%—hard problems where the 1.7B model's own rollouts rarely contain correct solutions). This training-time collaboration, inference-time independence model is precisely the use case that MARL cannot address (MARL requires multi-agent coordination at inference) and that KD partially addresses but without bidirectional benefit (the cloud model also improves, making it better for the tasks it handles directly). The practical deployment architecture is: train collaboratively, then deploy the 1.7B model to edge devices and the 8B model to cloud servers. Both are better than they would have been if trained independently, and users who fall back to the cloud model for hard queries get a model that has also improved.

4. Data-efficient fine-tuning for specialized domains with limited training data. The paper uses 7.5k MATH training questions—a relatively small dataset by LLM pretraining standards—and trains for only one epoch. In specialized domains where labeled data is scarce (medical reasoning, legal analysis, scientific problem-solving), the sample efficiency multiplier from cross-agent sharing could be decisive. If an organization has two models (perhaps a general-purpose model and a domain-specialized model) and only 1,000 domain-specific training examples with ground-truth answers, running HACPO effectively gives each model access to 2,000 training trajectories (its own 1,000 plus the other agent's 1,000) rather than 1,000. This doubling of effective data, with the HACPO corrections ensuring that the additional data helps rather than harms, could be the difference between a usable domain-specific model and one that overfits or underfits. The paper's results on AIME2025—a competition math benchmark where training data is inherently scarce (only past exam problems are available)—showing the 4B model improving from 48.5% to 62.2% in the heterogeneous state setting, suggests that HACPO's data-efficiency benefits are particularly pronounced on hard, data-scarce problems.