ArXiv: 2605.09262

🎯 Pitch

Standard RL fine-tuning amplifies MLLMs' catastrophic brittleness to blurry or compressed images, but naïvely injecting degraded data corrupts the reward signal with hallucinated reasoning. ROMA resolves this by forcing the model to see degraded views through the lens of successful clean-input rollouts, gating invariance penalties only to correct trajectories, anchoring auxiliary gradients to clean advantages, and penalizing the worst-case perceptual discrepancy—achieving a +2.4% robustness gain on seen corruptions without sacrificing clean accuracy.

1. Executive Summary

This paper proposes ROMA, an RL fine-tuning framework that reinforces multimodal reasoning against visual degradation by modifying optimization dynamics rather than relying on static data augmentation. Using GRPO on Qwen3-VL 4B and 8B Instruct models across seven multimodal reasoning benchmarks, ROMA introduces a dual-forward-pass strategy (teacher-forcing corrupted views against clean-image trajectories to avoid reward poisoning), a worst-case token-level KL penalty (penalizing the augmentation with the largest divergence between clean and degraded view logits), an auxiliary policy gradient loss anchored to clean-image advantages (preserving reliable reward signals under regularization), and correctness-conditioned regularization (restricting invariance enforcement to successful trajectories). On the 8B model, ROMA matches GRPO's clean accuracy (68.7%) while improving robustness to seen degradations by +2.4% (61.6% vs. 59.2%) and to unseen degradations by +2.3% (56.3% vs. 54.0%), establishing that critic-free RL-fine-tuned MLLMs can achieve visual robustness without sacrificing clean-input reasoning fidelity only when the invariance penalty is correctness-gated and the auxiliary policy gradient is anchored to clean-trajectory advantages.

2. Context and Motivation

The Core Problem: MLLM Reasoning Collapses Under Visual Degradation

The paper addresses a specific and practically critical gap: Reinforcement Learning (RL) has significantly improved multimodal reasoning capabilities in MLLMs, but the resulting policies remain catastrophically brittle to real-world visual degradation. A model that achieves strong performance on clean, high-quality inputs—such as a scanned textbook page or a professionally captured photograph—often fails completely when presented with the same content under common real-world artifacts: motion blur from camera shake, compression artifacts from messaging apps, low-resolution captures from document scanners, or sensor noise in low-light photography.

This is not merely a theoretical concern. The paper's motivating scenario in Section 1 is concrete: "a model that performs reliably on a clean input (e.g., a high-quality PDF) often fails catastrophically on a degraded version of the same content." This brittleness poses a critical barrier to deploying reasoning-capable MLLMs in practical settings—autonomous systems processing surveillance footage, educational tools analyzing student-submitted phone photos of homework, medical assistants interpreting compressed radiology images, or any application where input quality is not guaranteed.

The problem is particularly acute because RL fine-tuning amplifies this brittleness. As the experimental results demonstrate in Section 4.2, standard GRPO—despite improving clean-input accuracy—suffers larger performance drops under degradation than the base instruction-tuned model. For the 8B model, GRPO drops 9.7% from clean to seen-degraded evaluation (68.9% → 59.2%), while the base model drops only 7.9% (66.8% → 58.9%). This means that the very process designed to enhance reasoning is making the model more fragile to perceptual perturbations, creating a tension between reasoning capability and perceptual robustness that must be addressed.

Why Existing Robustness Techniques Fail Here

The paper identifies two distinct lines of prior work—both of which are inadequate for the specific challenge of making RL-fine-tuned MLLMs robust to visual degradation.

Visual robustness in computer vision has been extensively studied through data augmentation: applying transformations like cropping, flipping, color jittering, and blurring to training images, often combined with contrastive learning objectives that encourage invariant representations (CLIP, SimCLR, and their descendants). The fundamental assumption is that exposing a model to diverse views of the same semantic content will cause it to learn features that are insensitive to nuisance transformations. While effective for static perception tasks like image classification, this approach does not transfer cleanly to the autoregressive reasoning setting for a specific reason explored below.

Visual robustness in deep RL represents a more directly relevant line of work. Methods like DrAC (Data-regularized Actor-Critic), RAD (Reinforcement learning with Augmented Data), and DrQ demonstrate that applying visual augmentations during policy training improves out-of-distribution generalization in sequential decision-making environments. DrAC in particular (Raileanu et al., 2020, cited as reference [27]) is the direct inspiration for ROMA's invariance penalty formulation. In these actor-critic settings, robustness is achieved by regularizing both the policy network and the value network to produce consistent outputs (actions, value estimates) across clean and augmented observations. The regularization term is typically a mean squared error or KL divergence constraint applied to both the policy's action distribution and the critic's value prediction:

LDrAC=LRL+λ⋅[MSE(V(s),V(f(s)))+DKL(π(⋅∣s)∥π(⋅∣f(s)))]\mathcal{L}_{\text{DrAC}} = \mathcal{L}_{\text{RL}} + \lambda \cdot \left[ \text{MSE}(V(s), V(f(s))) + D_{\text{KL}}(\pi(\cdot|s) \| \pi(\cdot|f(s))) \right]

where ss is the clean state, f(s)f(s) is an augmented state, VV is the value function, and π\pi is the policy.

However, the paper identifies two specific failures that make this approach inapplicable to the MLLM RL fine-tuning setting.

Failure 1: Architectural mismatch from critic-free RL algorithms. Modern RL fine-tuning of large autoregressive models increasingly relies on critic-free algorithms such as Group Relative Policy Optimization (GRPO), specifically to avoid the memory and computational overhead of training a separate value network alongside the policy. As the paper states in Section 1:

"Modern RL fine-tuning of autoregressive models increasingly relies on critic-free algorithms such as GRPO to avoid the memory overhead of value networks; consequently, classical value-based robustness regularizers do not apply out of the box."

This is a concrete and non-trivial mismatch. GRPO computes advantages by comparing rewards across a group of rollouts generated from the same prompt, eliminating the need for a learned value function. But DrAC-style regularization fundamentally depends on having a value network to regularize—without it, half the regularizer is inapplicable. The paper cannot simply port DrAC to GRPO; it needs to redesign the invariance objective for the critic-free regime, which it does by isolating the policy invariance term (Equation 2) and removing the value-based component entirely.

Failure 2: Reward poisoning from naïve data augmentation. This is arguably the more subtle and interesting failure mode. A seemingly natural approach to making GRPO-fine-tuned MLLMs robust would be to simply inject degraded images during the rollout phase—sample rollouts on augmented inputs, compute rewards on those rollouts, and let the policy gradient naturally favor trajectories that succeed despite degradation. This is the strategy explored by concurrent work, specifically NoisyRollout (Liu et al., 2025, reference [17]), which "attempts to reinforce visual exploration by directly injecting data augmentation into the environment during the RL generation phase."

The paper argues this approach fails for a specific reason: reward poisoning. When the model generates a rollout on a severely degraded image, the perceptual degradation can "obscure perceptual evidence and force the model to hallucinate" (Section 1). The critical insight is that the resulting incorrect trajectory is not a reasoning failure per se—it's a perception failure. The model literally cannot see the content needed to reason correctly. When the reward function penalizes this trajectory, the model receives a negative signal that confounds two distinct sources of error: (1) the model's reasoning capability is insufficient, and (2) the visual input was illegible. The optimization process cannot disambiguate these causes, leading to what the paper terms "reward poisoning"—the reward signal penalizes perceptual failure rather than reasoning errors, destabilizing optimization and potentially inducing policy collapse.

This is not purely speculative. The paper's experimental results provide indirect evidence: GRPO's clean-to-degraded performance gap (9.7% for 8B) is larger than the base model's gap (7.9%), suggesting that standard RL fine-tuning on clean data alone makes the model more sensitive to perceptual perturbations—even without seeing degradations during training. If naïvely augmented rollouts were used, the model would be generating trajectories on inputs it cannot interpret, receiving noisy reward signals, and likely performing even worse.

How the Paper Positions Itself

The paper positions ROMA as a reformulation of visual invariance regularization specifically for critic-free, autoregressive MLLM fine-tuning. Rather than relying on static data augmentation (which fails due to reward poisoning) or value-based regularization (which is architecturally incompatible with GRPO), ROMA modifies the optimization dynamics directly.

The key conceptual move is the dual-forward-pass strategy described in Section 1 and Figure 1: the model generates trajectories exclusively on clean images (avoiding reward poisoning by construction), but evaluates those same trajectories under multiple degraded views using teacher forcing (computing token-level log-probabilities without sampling new rollouts). This allows ROMA to observe how the model's token distributions shift under perturbation and regularize against those shifts, without ever exposing the reward signal to the corrupted perceptual evidence that would cause poisoning.

This positions ROMA at the intersection of two previously separate research threads:

  1. Multimodal reasoning RL (GRPO, R1-style reasoning incentives): ROMA inherits the critic-free RL framework and the goal of improving reasoning fidelity on clean inputs, but extends it with robustness objectives that the base GRPO formulation lacks.

  2. Visual robustness in RL (DrAC, RAD): ROMA inherits the idea of cross-view invariance regularization but reformulates it for token-level autoregressive generation with a critic-free advantage, adds worst-case multi-view optimization (Equation 3) rather than uniform augmentation, introduces an auxiliary policy gradient to prevent regularization-induced collapse (Equation 4), and gates the invariance penalty with a correctness mask (the indicator I[R(v,x,y)>0]\mathbb{I}[R(v, x, y) > 0] in Equation 5) to avoid reinforcing incorrect reasoning.

The paper explicitly acknowledges its relationship to prior work: the token-level KL penalty (Equation 2) is "inspired by" DrAC [27], the worst-case formulation extends beyond single-view augmentation, and the auxiliary PG loss addresses a failure mode (policy collapse under excessive KL regularization) that prior invariance methods did not encounter because they had a value network providing an additional learning signal. ROMA is thus positioned not as a completely novel robustness technique but as a necessary adaptation of established robustness principles to the specific constraints of modern MLLM RL fine-tuning architecture.

Why This Problem Matters Now

The timing of this work is significant for two reasons. First, the field is rapidly shifting toward RL-based post-training as the primary method for eliciting reasoning capabilities from MLLMs (DeepSeek-R1, Vision-R1, LMM-R1, and numerous concurrent efforts). As this paradigm becomes dominant, the brittleness problem will become more widespread—every model that undergoes GRPO-style reasoning training will inherit increased sensitivity to perceptual degradation unless robustness is explicitly addressed during fine-tuning.

Second, MLLMs are being deployed in increasingly diverse real-world settings where input quality is uncontrolled. A model trained on curated benchmarks with professional-quality images will encounter user-uploaded photos, compressed screenshots, scanned documents, and low-bandwidth video feeds in production. The gap between training distribution (clean, curated) and deployment distribution (noisy, degraded) is large and growing, making robustness to visual degradation a first-order concern for practical MLLM deployment.

The paper's contribution is not that robustness is important—this is well-established—but rather that existing robustness techniques do not transfer to the specific architectural and algorithmic constraints of modern MLLM RL fine-tuning, and that a carefully designed optimization-level intervention (correctness-gated, worst-case, auxiliary-anchored invariance regularization) can recover robustness without sacrificing the reasoning gains that RL provides.

3. Technical Approach

3.1 Reader Orientation

This paper builds a regularized optimization procedure — not a new model architecture or dataset — that modifies how reinforcement learning updates the parameters of a multimodal large language model, so that the resulting policy produces correct reasoning trajectories even when the input image is degraded by real-world artifacts like blur, noise, or compression. The system solves the problem of reward poisoning and architectural incompatibility: rather than naïvely injecting degraded images during RL rollouts (which causes the model to hallucinate and the reward to become unreliable) or relying on value-network regularization (which is impossible in critic-free algorithms like GRPO), ROMA generates training trajectories exclusively on clean images, evaluates those same trajectories under multiple degraded views using teacher forcing, and adds three carefully designed regularizers that together push the model toward reasoning that is invariant to perceptual perturbations while preserving clean-input accuracy.

The "shape" of the solution is a joint optimization objective (Equation 5) that combines the standard RL loss with two additional terms — a worst-case KL penalty that enforces distributional consistency between clean and degraded views, and an auxiliary policy gradient that actively trains the model to maximize reward under degradation — both gated by a correctness condition so that the model is never pushed toward being consistently but systematically incorrect.

3.2 Big-Picture Architecture (Diagram in Words)

The ROMA framework has five major components that interact during each RL optimization step:

  1. Base MLLM policy ($\pi_\theta$): a pretrained autoregressive multimodal language model (Qwen3-VL 4B or 8B Instruct) that takes a text question $x$ and an image $v$ as input and generates a reasoning trajectory $y$ as output. This is the policy being fine-tuned.

  2. Clean-image rollout generator: samples complete reasoning trajectories from the current policy on clean, unperturbed images only. This is the standard RL rollout phase — it produces the token sequences $y$, the rewards $R(v, x, y)$, and the advantages $A(v, x, y)$ that define the main RL objective $J_{RL}(\theta)$.

  3. Multi-view degradation module: takes the clean image $v$ and produces $K$ degraded views $f_1(v), f_2(v), ..., f_K(v)$ by applying randomly sampled visual corruptions (Gaussian noise, Gaussian blur, JPEG compression, resolution downscaling). During training, corruption parameters are drawn from continuous distributions; the module produces $K = 3$ distinct views per image.

  4. Dual-forward-pass teacher-forcing evaluator: takes the frozen clean-image trajectory $y$ and re-processes it under each degraded view $f_k(v)$ using teacher forcing — meaning the model receives the ground-truth previous tokens $y_{<t}$ as context (not its own generated tokens) and computes only the next-token log-probability $\pi_\theta(y_t | f_k(v), x, y_{<t})$. This produces token-level log-probabilities under each degraded view without ever sampling a new rollout from a degraded input, thereby avoiding reward poisoning by construction.

  5. Three-part regularization objective: computes (i) a correctness-conditioned, worst-case token-level KL divergence penalty between clean and degraded view log-probabilities; (ii) an auxiliary policy gradient loss using a clipped surrogate objective on a randomly sampled degraded view, anchored to clean-image advantages; and (iii) the standard GRPO objective on clean rollouts. These are combined with coefficients $\alpha$ and $\beta$ into the total objective $J_{total}(\theta)$ (Equation 5).

Information flow per optimization step:

  • Step 1 (Clean rollout): The policy $\pi_\theta$ generates $N = 8$ complete reasoning trajectories per input on the clean image $v$. A reward function evaluates each trajectory for correctness (producing a scalar reward $R$), and GRPO computes group-relative advantages $A(v, x, y)$ by comparing rewards within the rollout group.
  • Step 2 (Degradation): $K = 3$ augmentation functions are sampled from the seen-degradation pool, producing $f_1(v), f_2(v), f_3(v)$. The augmentations are stochastic and parameterized by continuous distributions.
  • Step 3 (Teacher-forced re-evaluation): For each degradation $f_k(v)$, the model processes the full clean-image trajectory $y$ token-by-token under the degraded view, computing $\pi_\theta(y_t | f_k(v), x, y_{<t})$ at each position. No new tokens are sampled; the model only produces log-probabilities for the existing clean-trajectory tokens.
  • Step 4 (Worst-case KL computation): For each of the $K$ views, compute the token-level KL divergence between the clean-view log-probabilities (stop-gradiented) and the degraded-view log-probabilities (Equation 2). Select the maximum divergence across the $K$ views as the worst-case penalty (Equation 3). Apply the correctness mask: if $R(v, x, y) \leq 0$, the penalty is zeroed out.
  • Step 5 (Auxiliary PG computation): On one randomly sampled degraded view $f \sim \mathcal{F}_K$, compute a clipped PPO-style surrogate objective (Equation 4) using the clean-image advantages $A(v, x, y)$ and the importance sampling ratio between the degraded-view and clean-view policies.
  • Step 6 (Total loss and update): Sum the three objectives — standard GRPO ($J_{RL}$), auxiliary PG ($\alpha \cdot J_{aug\_pg}$), and worst-case correctness-conditioned KL penalty ($-\beta \cdot \mathbb{E}[G_\pi^{worst}(\theta) \cdot \mathbb{I}[R > 0]]$) — and update $\theta$ via gradient ascent.

3.3 Roadmap for the Deep Dive

  • First, I'll formalize the standard RL objective (Equation 1) and explain why GRPO is the base algorithm, since understanding the critic-free advantage computation is essential for seeing why value-based regularizers are inapplicable.
  • Second, I'll walk through the core architectural innovation: the dual-forward-pass strategy, explaining both the clean rollout phase (generation) and the teacher-forced re-evaluation under degraded views, and why this design avoids reward poisoning.
  • Third, I'll derive the correctness-conditioned token-level invariance penalty (Equation 2), including the stop-gradient operator, the per-token KL formulation, and the correctness mask gating mechanism.
  • Fourth, I'll explain the worst-case multi-view optimization (Equation 3), contrasting it with mean-penalty baselines and justifying why maximizing over augmentations provides stronger robustness.
  • Fifth, I'll detail the auxiliary policy gradient loss (Equation 4), explaining the clipped surrogate, the importance sampling ratio, and why anchoring advantages to clean rollouts prevents collapse under regularization.
  • Sixth, I'll present the total combined objective (Equation 5) and walk through all hyperparameter settings, training configurations, and the degradation protocol used during training.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a method paper whose core idea is that visual robustness in critic-free RL-fine-tuned MLLMs can be achieved by (1) generating trajectories only on clean images, (2) evaluating token-level distributional shifts under multiple degraded views via teacher forcing, (3) enforcing a worst-case KL penalty gated by trajectory correctness, and (4) adding an auxiliary policy gradient anchored to clean-image advantages to prevent regularization-induced collapse.

Standard RL Objective and GRPO Background

The paper begins by formalizing the standard reinforcement learning setting for multimodal reasoning (Section 3, "Problem Formulation"). The model is a stochastic policy $\pi_\theta$ parameterized by $\theta$ that takes a visual question $x$ and image $v$ as input and generates a reasoning trajectory $y \sim \pi_\theta(\cdot | v, x)$ autoregressively. The reward function $R(v, x, y)$ evaluates the correctness of the complete trajectory, typically by extracting the final answer (enclosed in \boxed{}) and comparing it against a reference answer.

The standard RL objective maximizes expected reward:

JRL(θ)=E(v,x)∼D, y∼πθ(⋅∣v,x)[R(v,x,y)]J_{RL}(\theta) = \mathbb{E}_{(v,x) \sim \mathcal{D}, \, y \sim \pi_\theta(\cdot | v, x)} \left[ R(v, x, y) \right]

where $\mathcal{D}$ is the training distribution of (image, question) pairs, $\pi_\theta(\cdot | v, x)$ is the model's autoregressive token distribution over reasoning trajectories, and $R(v, x, y) \in \mathbb{R}$ is a scalar reward (positive for correct answers, zero or negative otherwise).

What it computes: for each training example, sample a complete reasoning trajectory from the current policy, evaluate its correctness, and update the policy to increase the probability of trajectories that receive high rewards. In expectation over the training distribution, this maximizes the fraction of problems the model can solve correctly.

Why this form: this is the standard policy gradient objective — it directly optimizes the metric we care about (correctness frequency) without requiring a learned value function or world model. For autoregressive models with discrete token outputs, the gradient of this objective with respect to $\theta$ is estimated via REINFORCE or, more commonly in modern LLM training, via group-relative advantage estimation as in GRPO.

The paper uses GRPO (Group Relative Policy Optimization) as the underlying RL algorithm, which is explicitly chosen because it is "critic-free" (Section 1: "Modern RL fine-tuning of autoregressive models increasingly relies on critic-free algorithms such as GRPO to avoid the memory overhead of value networks"). In GRPO, for each input $(v, x)$, the model generates a group of $N$ trajectories (the paper uses $N = 8$ rollouts per input, as specified in Appendix A.1). The reward for each trajectory is computed, and the advantage for trajectory $i$ is the standardized reward relative to the group:

Ai=Ri−mean({R1,...,RN})std({R1,...,RN})A_i = \frac{R_i - \text{mean}(\{R_1, ..., R_N\})}{\text{std}(\{R_1, ..., R_N\})}

This group-relative normalization removes the need for a learned value baseline — the mean reward within the group serves as an implicit baseline, and standardization ensures the advantage scale is controlled. The GRPO surrogate objective is then a clipped policy gradient similar to PPO but computed over the group-relative advantages.

The critical consequence for ROMA's design: there is no value network $V(s)$ to regularize. DrAC-style robustness methods (Raileanu et al., 2020) regularize both the policy and the value function with an invariance penalty $\text{MSE}(V(s), V(f(s)))$ — but ROMA cannot use this because GRPO eliminates the value function entirely. The entire regularization burden must be carried by policy-side objectives alone, which the paper argues requires both the KL penalty (Equation 2) and the auxiliary PG (Equation 4) to prevent collapse.

The Dual-Forward-Pass Strategy: Avoiding Reward Poisoning

The central architectural innovation in ROMA is the dual-forward-pass strategy, which the paper describes in Section 1 and illustrates in Figure 1. This design is motivated by a specific failure mode: if the model samples rollouts on degraded images, the perceptual degradation can cause the model to "hallucinate" (fabricate content not present in the corrupted image), and the resulting reward signal penalizes the model for perception failure rather than reasoning failure. This is termed reward poisoning — the reward becomes an unreliable signal for improving reasoning because it is contaminated by perceptual noise.

The dual-forward-pass strategy resolves this by never sampling from degraded images. Here is exactly what happens:

Pass 1 — Clean-image rollout (generation mode): The model $\pi_\theta$ generates a complete reasoning trajectory $y$ by sampling autoregressively from the clean-image conditioned distribution $\pi_\theta(\cdot | v, x)$. Each token $y_t$ is sampled from $\pi_\theta(\cdot | v, x, y_{<t})$ at temperature 1.0 (Appendix A.1: "rollout temperature 1.0"). This produces the trajectory and, after reward evaluation, the advantages $A(v, x, y)$ that define the main RL objective. This pass is identical to standard GRPO — the model generates 8 rollouts per clean input, and advantages are computed group-relatively.

Pass 2 — Degraded-view teacher-forced evaluation (log-probability mode): The same trajectory $y$ from Pass 1 is now fed back into the model, but with two critical differences: (1) the input image is replaced by a degraded view $f_k(v)$, and (2) the model operates in teacher-forcing mode — rather than sampling tokens, it receives the ground-truth clean-trajectory tokens $y_{<t}$ as context and only computes the log-probability $\log \pi_\theta(y_t | f_k(v), x, y_{<t})$ for the next token. No new rollouts are generated.

"The second pass generates multiple degraded views of the same image and re-evaluates the same frozen trajectory via teacher forcing, computing token-level log-probabilities under each corrupted view without sampling new rollouts." (Section 1)

The teacher-forcing aspect is crucial: if the model were allowed to sample freely on a degraded image, it might diverge into a different trajectory — one based on hallucination — and the reward would penalize this hallucinated path. By using teacher forcing, ROMA only asks: "given that we know the correct reasoning steps from the clean image, how surprised is the model by them when it sees a degraded image instead?" If the model's token distribution under the degraded view is similar to its distribution under the clean view, the KL divergence will be small — the model is perceptually invariant. If the model's distribution shifts dramatically (because it can no longer read the necessary visual details), the KL divergence will be large — the model is perceptually brittle, and regularization will penalize this shift.

Why this specific design: the alternative — generating separate rollouts on degraded images and computing separate rewards — would conflate perception errors with reasoning errors. The dual-forward-pass design isolates the perceptual effect of degradation (measured via distributional shift) from the reasoning performance (measured via clean-image reward). The distributional shift is penalized directly through the KL term without passing through the noisy reward channel.

The paper states this explicitly in Section 1: "This sidesteps reward poisoning by construction: trajectories are never sampled from degraded inputs, yet we still observe how the model's token distributions shift under perturbation."

Correctness-Conditioned Token-Level Invariance Penalty

The first regularization term in ROMA enforces that the model's autoregressive token distribution under a degraded view should match its distribution under the clean view, but only for trajectories that are actually correct.

The Per-Token KL Divergence

The invariance penalty is defined in Equation 2:

Gπ(θ,f)=E(v,x)∼D, y∼πold[∑t=1∣y∣DKL(sg[πθ(⋅∣v,x,y<t)] ∥ πθ(⋅∣f(v),x,y<t))]G_\pi(\theta, f) = \mathbb{E}_{(v,x) \sim \mathcal{D}, \, y \sim \pi_{\text{old}}} \left[ \sum_{t=1}^{|y|} D_{KL}\left( \text{sg}\left[\pi_\theta(\cdot | v, x, y_{<t})\right] \,\|\, \pi_\theta(\cdot | f(v), x, y_{<t}) \right) \right]

where $f \in \mathcal{F}$ is a stochastic visual augmentation function that produces a degraded view $f(v)$, $\pi_{\text{old}}$ is the policy from the previous iteration (used to sample the trajectory $y$), $|y|$ is the number of tokens in the trajectory, $\text{sg}[\cdot]$ is the stop-gradient operator, and $D_{KL}(\cdot \| \cdot)$ is the Kullback-Leibler divergence.

What each symbol means:

  • $G_\pi(\theta, f)$: the total invariance penalty for augmentation $f$, summed over all tokens, accumulated over the training distribution.
  • $f(v)$: a specific degraded version of image $v$ produced by augmentation $f$ (e.g., $f$ might apply Gaussian blur with radius 2.0).
  • $\pi_\theta(\cdot | v, x, y_{<t})$: the model's predicted token distribution at position $t$ conditioned on the clean image and the prefix $y_{<t}$.
  • $\text{sg}[\pi_\theta(\cdot | v, x, y_{<t})]$: the same distribution but with gradient flow stopped — this is treated as a fixed target during backpropagation.
  • $\pi_\theta(\cdot | f(v), x, y_{<t})$: the model's predicted token distribution at the same position $t$ conditioned on the degraded image and the same prefix.

What it computes: for each token position in the trajectory, compute the KL divergence between the model's clean-view token distribution (as a fixed target) and the model's degraded-view token distribution (which receives gradients). Sum these per-token divergences across the trajectory length $|y|$. The expectation is over training examples $(v, x)$ and trajectories $y$ sampled from the old policy.

Why this form — stop-gradient: the stop-gradient operator on the clean-view distribution is critical. Without it, gradients from the KL penalty would flow into both the clean and degraded branches, potentially distorting the clean-image representations (which the main RL objective depends on for reasoning fidelity). As the paper states: "To prevent the noisy gradients from corrupting the clean representations, we apply a stop-gradient operator (sg[·]) to the clean policy outputs." The degraded-view distribution $\pi_\theta(\cdot | f(v), x, y_{<t})$ receives the full gradient, being pushed to match the clean-view distribution, while the clean-view distribution remains anchored by the main RL objective $J_{RL}$.

Why this form — token-level rather than trajectory-level: the KL divergence is computed per token, not for the entire trajectory as a single distribution. This is a practical choice driven by autoregressive generation: at each step, the model outputs a categorical distribution over the next token, and the KL divergence between two categorical distributions is well-defined and computationally cheap. Computing the KL divergence between full trajectory distributions $\pi_\theta(y | v, x)$ and $\pi_\theta(y | f(v), x)$ would require summing over all possible trajectories (exponential in length), which is intractable. The per-token formulation factorizes the trajectory-level constraint into a sum of step-level constraints, which is both tractable and interpretable.

Practical KL Approximation

The paper provides the specific computation used in practice:

DKL(t)≈pt⋅(log⁡pt−log⁡qt)D^{(t)}_{KL} \approx p_t \cdot (\log p_t - \log q_t)

where $p_t = \text{sg}[\pi_\theta(y_t | v, x, y_{<t})]$ is the probability assigned by the clean-view policy to the token $y_t$ that actually appeared in the trajectory (treated as a scalar, with stop-gradient), and $q_t = \pi_\theta(y_t | f(v), x, y_{<t})$ is the probability assigned by the degraded-view policy to the same token $y_t$.

What it computes: this is the standard formula for the KL divergence from a categorical distribution $q_t$ to a categorical distribution $p_t$, evaluated at the single outcome $y_t$. In full, the KL divergence between two categorical distributions $P$ and $Q$ over a vocabulary $\mathcal{V}$ is $D_{KL}(P \| Q) = \sum_{w \in \mathcal{V}} P(w) \log \frac{P(w)}{Q(w)}$. The approximation here replaces the full sum with the term corresponding to the observed token $y_t$, weighted by its probability $p_t = P(y_t)$. This is an unbiased estimate of the full KL divergence under sampling from $P$.

Why this approximation: computing the full sum over the vocabulary (tens of thousands of tokens) at every position would be memory-intensive and slow. Since $y_t$ is the token that actually appeared in the trajectory, and $p_t$ is the model's probability for it under the clean view, this approximation provides a signal that pushes the degraded-view model to assign high probability to the same token that the clean-view model favored. It is essentially encouraging distributional alignment through the observed outcomes, which is a standard technique in RL-based KL regularization.

Correctness Conditioning (the Gating Mechanism)

The full invariance penalty, as it appears in the final objective (Equation 5), includes a critical gating mechanism:

Effective penalty=Gπworst(θ)⋅I[R(v,x,y)>0]\text{Effective penalty} = G_\pi^{worst}(\theta) \cdot \mathbb{I}[R(v, x, y) > 0]

where $\mathbb{I}[R(v, x, y) > 0]isanindicatorfunctionthatequals1iftherewardfortheclean−imagetrajectory‘ is an indicator function that equals 1 if the reward for the clean-image trajectory `y$` is positive (i.e., the trajectory correctly solved the problem), and 0 otherwise.

What this computes: if the trajectory $y$ is incorrect (reward ≤ 0), the entire invariance penalty is multiplied by zero and contributes nothing to the gradient. If the trajectory is correct (reward > 0), the penalty is applied in full.

Why this form — the "robustly incorrect" problem: the paper provides a direct rationale in Section 3: "enforcing consistency across views is actively harmful if the underlying trajectory y is hallucinated or factually incorrect. To prevent the policy from becoming robustly incorrect, we introduce a correctness mask, applying the penalty strictly to trajectories that successfully solve the task." If the model produces an incorrect reasoning chain on the clean image, forcing its degraded-view distribution to match that incorrect chain would simply make the model consistently wrong — it would reliably produce the same incorrect reasoning regardless of image quality, which is robustness in the wrong direction. The correctness mask ensures that the invariance penalty only pushes the model toward perceptually invariant correct reasoning, never toward invariant incorrect reasoning.

The ablation study in Table 6 confirms this is essential: removing correctness conditioning (making the penalty unconditional) drops seen-degradation accuracy from 61.6% to 59.4% and unseen-degradation accuracy from 56.3% to 54.1% on the 8B model. The paper interprets this as empirical evidence that "enforcing the invariance penalty unconditionally, forcing the degraded reasoning trajectory to match the clean trajectory regardless of whether the clean rationale is correct, causes an average performance drop of 2.2% on both seen and unseen degradations."

Worst-Case Multi-View Optimization

A single randomly sampled augmentation may produce a visually trivial degradation — for instance, Gaussian noise with a small standard deviation might barely affect the image — providing a weak regularization signal. To enforce more rigorous robustness, ROMA samples $K$ distinct augmentations and applies the penalty only to the most challenging one.

The Maximization Over Augmentations

Equation 3 defines the worst-case penalty:

Gπworst(θ)=max⁡fk∈FKGπ(θ,fk)G_\pi^{worst}(\theta) = \max_{f_k \in \mathcal{F}_K} G_\pi(\theta, f_k)

where $\mathcal{F}_K$ is a set of $K$ independently sampled augmentation functions, and $G_\pi(\theta, f_k)$ is the per-token KL penalty from Equation 2 computed with augmentation $f_k$.

What it computes: for each of the $K$ degraded views, compute the total KL divergence between the clean-view and degraded-view token distributions. Select the maximum of these $K$ values as the penalty for this training step. Only this worst-case divergence receives gradient.

Why this form — adversarial robustness: the paper's argument (Section 3, "Worst-Case Multi-View Optimization") is that "randomly sampled augmentations may be visually trivial, providing weak regularization signals." By maximizing over augmentations, ROMA identifies the specific perturbation that causes the largest distributional shift — the view on which the model is most uncertain or most confused — and forces the policy to reduce that maximum shift. This is a minimax formulation: the optimizer (policy update) minimizes the worst-case divergence, while the "adversary" (the max over augmentations, though not learned) identifies the hardest perturbation.

Practical configuration: the paper uses $K = 3$ as the default number of augmented views per step (Table 7). The sensitivity analysis in Section 4.3 shows that increasing $K$ from 1 to 3 yields steady improvements (seen: 59.5% → 60.7% → 61.6%), but $K = 4$ provides slight degradation (61.3%), suggesting that too many views dilute the worst-case signal or exceed the model's capacity to simultaneously match multiple highly divergent distributions.

Comparison to Mean-Penalty Baseline

The ablation in Table 3 directly compares worst-case optimization against a mean-penalty baseline (averaging the KL divergence across all $K$ views rather than taking the max). The worst-case formulation outperforms the mean by 1.6% on seen degradations (61.6% vs. 60.0%) and 1.8% on unseen degradations (56.3% vs. 54.5%). The paper interprets this as evidence that averaging "is insufficient for securing robustness" — the worst-case approach forces the model to handle the most challenging perturbation, which provides a stronger inductive bias toward general robustness than simply matching the average behavior across multiple easy and hard views.

Auxiliary Policy Gradient Loss Anchored to Clean Advantages

The KL penalty enforces distributional consistency — it pushes the model's degraded-view token distributions to match the clean-view distributions. However, the paper argues that this is not sufficient on its own: the KL penalty is purely a regularization signal that inhibits deviation; it does not provide an active learning signal that guides the model toward correct reasoning steps under degradation. Without such an active signal, excessive KL regularization can cause policy collapse — the model learns to output consistent but nonsensical tokens, satisfying the invariance constraint while losing reasoning capability.

The Clipped Surrogate Objective

Equation 4 defines the auxiliary policy gradient loss:

Jaug_pg(θ)=Ef∼FK[E[∑t=1∣y∣min⁡(ρtA, clip(ρt,1−ϵ,1+ϵ)A)]]J_{aug\_pg}(\theta) = \mathbb{E}_{f \sim \mathcal{F}_K} \left[ \mathbb{E} \left[ \sum_{t=1}^{|y|} \min\left( \rho_t A, \, \text{clip}(\rho_t, 1 - \epsilon, 1 + \epsilon) A \right) \right] \right]

where $f \sim \mathcal{F}_K$ is a randomly sampled augmentation from the $K$-view pool (not the worst-case one), $A = A(v, x, y)$ is the advantage computed from the clean-image rollout, $\rho_t$ is the importance sampling ratio at position $t$, $\epsilon$ is the PPO clipping parameter (typically 0.2, though the paper does not specify this value explicitly), and the inner expectation is over trajectories $y$ sampled from the clean rollout.

What each symbol means:

  • $\rho_t = \frac{\pi_\theta(y_t | f(v), x, y_{<t})}{\pi_{\text{old}}(y_t | v, x, y_{<t})}$: the ratio of the probability the current policy assigns to token $y_t$ under the degraded view, divided by the probability the old policy assigned to the same token under the clean view. This measures how much more or less likely the current policy is to produce token $y_t$ under degradation compared to the old policy on the clean image.
  • $\rho_t A$: the unclipped surrogate — the importance-weighted advantage. If $\rho_t > 1$, the current policy is more likely to produce token $y_t$ under degradation than the old policy was on the clean image, which is good if $A > 0$ (the trajectory is above-average within its group) and bad if $A < 0$.
  • $\text{clip}(\rho_t, 1 - \epsilon, 1 + \epsilon) A$: the clipped surrogate — the importance ratio is clipped to the interval $[1 - \epsilon, 1 + \epsilon]$ to prevent the policy update from being too large in any single step.
  • $\min(\rho_t A, \text{clip}(\rho_t, 1 - \epsilon, 1 + \epsilon) A)$: the PPO-style conservative objective — when the advantage is positive, this is $\min(\rho_t, 1 + \epsilon) \cdot A$, which caps the benefit from making a good token much more likely; when the advantage is negative, this is $\max(\rho_t, 1 - \epsilon) \cdot A$, which caps the penalty from making a bad token much less likely.

What it computes: for a randomly sampled degraded view, compute the PPO-style clipped surrogate objective using the clean-image trajectory and clean-image advantages. This is essentially running a policy gradient update on the degraded view as if the rewards and trajectories came from that view, but with the safeguards that (a) the trajectory tokens $y_t$ are from the clean rollout (not sampled under degradation), and (b) the advantages $A$ are from the clean rollout (not computed on the degraded view). The clipping mechanism prevents the update from being overly aggressive in any single step.

Why this form — anchoring to clean advantages: the paper's critical design choice is that the advantages $A$ are "anchored to clean-image advantages" (Section 1). This is what prevents reward poisoning: the advantages are computed from rewards on clean rollouts, which are reliable signals of reasoning quality. If advantages were computed on degraded rollouts, they would be contaminated by perception-driven reward noise. By using clean-image advantages, the auxiliary PG provides a trustworthy gradient signal — "improve the probability of tokens that led to correct reasoning on the clean image, even when the image is degraded" — without ever needing to evaluate reward quality under degradation.

Why this form — preventing collapse under regularization: the KL penalty alone (Equation 2) pushes the degraded-view distribution toward the clean-view distribution — a purely conservative force. Without the auxiliary PG, the model has no incentive to actively maximize reward under degradation; it only receives a penalty for deviating from the clean distribution. In the extreme, the model could collapse to outputting the same token regardless of input — satisfying the KL constraint (the distribution never changes) but failing at the reasoning task. The auxiliary PG provides a counterbalancing force: it rewards the model for producing tokens that are useful (high-advantage) under degradation, not just tokens that match the clean distribution. Combined, the KL penalty keeps the model from drifting too far from its clean behavior, while the auxiliary PG pushes it to maintain reasoning quality under perturbation.

The ablation in Table 4 confirms the auxiliary PG is necessary: removing it reduces seen-degradation accuracy by 1.6% (61.6% → 60.5%) and unseen-degradation accuracy by 1.8% (56.3% → 55.4%). The paper notes: "relying solely on the token-level invariance penalty is restrictive. While the invariance penalty successfully anchors the degraded output to the clean reference, it does not provide a sufficient learning signal to actively solve the reasoning task under visual occlusion."

Why a Randomly Sampled View for the Auxiliary PG?

An important subtlety: the auxiliary PG uses a randomly sampled augmentation ($f \sim \mathcal{F}_K$), not the worst-case augmentation. The worst-case selection is used only for the KL penalty (Equation 3). The paper does not explicitly justify this choice, but the reasoning is likely computational: the auxiliary PG requires computing the full clipped surrogate across the trajectory, which involves forward passes through the model under a degraded view and a separate gradient computation. Applying this to all $K$ views and selecting the worst-case would triple the computational cost (for $K = 3$). The KL penalty, by contrast, only requires computing log-probabilities (no advantage weighting, no clipping), making it cheaper to evaluate across all $K$ views. The auxiliary PG on a single random view provides a stochastic but unbiased learning signal.

The Total Combined Objective

Equation 5 assembles all components into the final optimization objective:

Jtotal(θ)=JRL(θ)+α⋅Jaug_pg(θ)−β⋅E[Gπworst(θ)⋅I[R(v,x,y)>0]]J_{total}(\theta) = J_{RL}(\theta) + \alpha \cdot J_{aug\_pg}(\theta) - \beta \cdot \mathbb{E} \left[ G_\pi^{worst}(\theta) \cdot \mathbb{I}[R(v, x, y) > 0] \right]

where $J_{RL}(\theta)$ is the standard GRPO objective (Equation 1, implemented with group-relative advantages), $J_{aug\_pg}(\theta)$ is the auxiliary policy gradient from Equation 4, $G_\pi^{worst}(\theta)$ is the worst-case KL penalty from Equation 3, $\mathbb{I}[R(v, x, y) > 0]$ is the correctness mask, and $\alpha$ and $\beta$ are scalar coefficients controlling the strength of the two additional terms.

What it computes: the model's parameters $\theta$ are updated to maximize $J_{total}$. The three terms pull in different but complementary directions:

  1. $J_{RL}$ pulls toward higher accuracy on clean images — the standard reasoning objective.
  2. $+\alpha \cdot J_{aug\_pg}$ pulls toward higher (clean-advantage-weighted) probability for correct-reasoning tokens under a randomly sampled degraded view — actively learning to reason under degradation.
  3. $-\beta \cdot \mathbb{E}[G_\pi^{worst} \cdot \mathbb{I}[R > 0]]$ pushes the worst-case degraded-view token distribution to match the clean-view distribution, but only for trajectories that were correct — enforcing perceptual invariance of correct reasoning.

Why this form — the negative sign on the KL penalty: the KL divergence $G_\pi^{worst}$ is always non-negative (by properties of KL divergence), and the correctness indicator is either 0 or 1. The negative sign means that reducing the KL divergence increases $J_{total}$ — equivalent to minimizing the divergence but expressed as a maximization objective. This is standard in policy optimization where objectives are framed as rewards to maximize.

Hyperparameter settings (Section 4.1 and Appendix A.1):

  • $\alpha = 0.10$: the auxiliary policy gradient coefficient. Sensitivity analysis in Table 5 shows that $\alpha = 0.10$ achieves the best balance (61.6% seen, 56.3% unseen), while $\alpha = 0.05$ under-regularizes (60.5% seen) and $\alpha = 0.15$ over-regularizes (60.0% seen), likely because the auxiliary PG begins to dominate the clean-image RL objective.
  • $\beta = 0.10$: the invariance penalty weight. Sensitivity analysis in Table 8 shows a similar pattern: $\beta = 0.10$ is optimal (61.3% seen, 56.3% unseen), $\beta = 0.05$ is too weak (59.4% seen), and $\beta = 0.15$ is too strong (56.8% seen), forcing the model to "prioritize structural matching over exploratory problem-solving."
  • $K = 3$: number of augmented views per step (Table 7).
  • $N = 8$: number of rollouts per input for GRPO (Appendix A.1).
  • Learning rate: $1 \times 10^{-6}$ (Appendix A.1).
  • Weight decay: $0.01$.
  • Global batch size: $128$, rollout batch size: $256$.
  • Rollout temperature: $1.0$.
  • Training steps: $120$.
  • Training dataset: MMRL30k (approximately 30K samples).

Why these coefficient values: the paper does not provide a detailed derivation, but the sensitivity analysis reveals an important relationship: both $\alpha$ and $\beta$ exhibit an inverse-U shape, where too little regularization fails to induce robustness and too much regularization degrades clean performance (implicitly, by overwhelming the $J_{RL}$ term). The fact that both optimal values land at $0.10$ for both model sizes (4B and 8B) suggests that this balance — where the auxiliary PG and KL penalty together carry roughly 20% of the total gradient weight relative to the main RL objective — is a stable operating point for this architecture and task distribution.

Training Degradation Protocol

The paper specifies the degradations used during training in Section 4.1 and Appendix A.2. The seen degradation pool consists of four corruption types:

  1. Gaussian noise: additive Gaussian noise with standard deviation $\sigma$. During training, $\sigma$ is fixed at 0.05 (Table 9). During evaluation, three levels are used (0.03, 0.06, 0.12), with Level 3 (0.12) exceeding the training bound.

  2. Gaussian blur: convolution with a Gaussian kernel of radius $r$. During training, $r$ is sampled uniformly from $U(0.5, 2.0)$. During evaluation, levels are 1.0, 2.0, 3.5, with Level 3 (3.5) outside the training range.

  3. JPEG compression: re-encoding the image at quality level $q$. During training, $q$ is sampled uniformly from $\{30, 31, ..., 85\}$. During evaluation, levels are 65, 40, 15, with Level 3 (15) more severe than any training sample.

  4. Resolution downscaling: reducing the image resolution by a scale factor $f$. During training, $f$ is sampled uniformly from $U(0.3, 0.7)$. During evaluation, levels are 0.6, 0.4, 0.2, with Level 3 (0.2) below the training minimum.

The paper explicitly states that "during training, parameters are sampled continuously according to corresponding distributions" (Appendix A.2). This continuous sampling means the model sees a wide range of degradation severities during training, but evaluation Level 3 is specifically designed to be outside this range — testing out-of-distribution severity generalization.

The unseen degradation pool consists of five corruption types held out entirely during training: motion blur, salt-and-pepper noise, speckle noise, posterization, and pixelation. These test zero-shot generalization to novel noise structures that the invariance penalty has never directly regularized against.

This degradation protocol is directly inspired by the ImageNet-C framework (Hendrycks & Dietterich, 2019, reference [9]), which established the practice of evaluating robustness on both seen and unseen corruptions at multiple severity levels to distinguish in-distribution robustness from out-of-distribution generalization.

Summary of Design Choices and Their Justifications

  • Dual-forward-pass with teacher forcing over naïve augmented rollouts: avoids reward poisoning by preventing the model from ever sampling trajectories on perceptually occluded inputs where it would hallucinate.
  • Stop-gradient on clean-view logits in the KL penalty: prevents the invariance regularization from corrupting the clean-image representations that the main RL objective depends on.
  • Token-level KL over trajectory-level KL: factorizes the intractable trajectory distribution constraint into a sum of tractable per-step categorical KL divergences.
  • Worst-case maximization over mean-penalty: focuses the regularization budget on the hardest perturbation at each step, providing a stronger adversarial robustness signal than averaging over easy and hard views.
  • Auxiliary policy gradient over KL penalty alone: prevents policy collapse under regularization by providing an active learning signal (clean-advantage-weighted) that encourages the model to maintain reasoning quality under degradation, not just match distributions.
  • Clean-advantage anchoring in the auxiliary PG over degraded-advantage: preserves the reliability of the reward signal — advantages computed on clean rollouts reflect genuine reasoning quality, not perception-contaminated noise.
  • Correctness conditioning over unconditional invariance: prevents the model from learning to be "robustly incorrect" — consistently producing the same wrong answer regardless of image quality. Only pushes invariance for trajectories that demonstrate correct reasoning.
  • Randomly sampled view for auxiliary PG over worst-case: computational efficiency — the auxiliary PG requires forward and backward passes through the model; applying it to all $K$ views and selecting the worst-case would multiply the computational cost.
  • Continuous parameter sampling during training over fixed severity levels: exposes the model to a distribution of degradation strengths, encouraging generalization across severities rather than overfitting to specific parameter values.

4. Key Insights and Innovations

Innovation 1: The Dual-Forward-Pass Strategy Reframes Robustness as a Log-Probability Evaluation Problem, Not a Generation Problem

The dominant assumption — across computer vision, deep RL, and even concurrent MLLM robustness work — has been that robustness to visual degradation requires exposing the model to degraded inputs during the generation phase so that the policy learns to produce correct outputs despite noise. Data augmentation in vision (CLIP, SimCLR) trains on augmented images directly. DrAC and RAD in deep RL regularize the policy by requiring it to produce consistent actions when generating from augmented observations. NoisyRollout (Liu et al., 2025, reference [17])—the most directly comparable concurrent work—explicitly "injects data augmentation into the environment during the RL generation phase," sampling rollouts on degraded images and rewarding successful ones.

ROMA makes a fundamentally different choice: trajectories are never sampled from degraded images. The model only generates on clean inputs. Degraded views are introduced exclusively through teacher-forced log-probability evaluation, where the model sees the correct clean-image trajectory tokens as context and is asked only to compute how probable those tokens are under the degraded view. This is not an incremental improvement on data augmentation — it is a categorical shift in what the model is being asked to learn. The learning signal is not "can you solve this problem despite the noise?" but rather "when you can solve the problem on a clean image, do your token-level predictions remain stable when the image is perturbed?"

This reframing matters because it sidesteps the reward poisoning problem by construction. The paper identifies a specific causal mechanism — the conflation of perception failure and reasoning failure in the reward signal — and eliminates it not by making the reward more robust, but by removing the need for a reward on degraded inputs entirely. The auxiliary policy gradient term (Equation 4) uses clean-image advantages, meaning the model is never asked to evaluate whether a degraded-view trajectory was correct; it is only asked to maintain the probability of tokens that were already known to be part of a correct trajectory. This is a significant conceptual move: it separates perceptual invariance (measured by distributional shift) from reasoning quality (measured by clean-image reward), treating them as distinct regularization targets rather than conflating them in a single reward signal.

The evidence that this is more than an implementation detail is in the architecture of the failure it avoids. GRPO's clean-to-degraded accuracy gap (9.7 percentage points for the 8B model) is larger than the base model's gap (7.9 percentage points) — meaning RL fine-tuning on clean data alone makes the model more brittle, even without seeing any degradations. This strongly implies that naive augmented rollouts would amplify, not mitigate, the problem, because the model would be generating trajectories on inputs it cannot interpret and receiving noisy rewards. ROMA's dual-forward-pass design is thus not just a clever engineering trick; it is a diagnostic insight about why the naive approach fails, and a corresponding architectural solution that targets the root cause.

Significance: This is a fundamental shift in how to think about multimodal robustness during RL. It moves the field from a "train on noise to be robust to noise" paradigm to an "anchor reasoning to clean perception, enforce distributional stability under perturbation" paradigm. Whether this generalizes beyond the specific setting of visual degradation in MLLMs is an open question, but the conceptual framework — separating generation-mode and evaluation-mode forward passes, only generating from reliable inputs — is broadly applicable.

Innovation 2: Correctness-Conditioned Invariance Recognizes That Robustness to the Wrong Answer Is Worse Than Brittleness

The standard approach to invariance regularization — in DrAC, RAD, contrastive learning, and virtually all data augmentation pipelines — applies the invariance penalty unconditionally. Whether the model's output on the clean input is correct or incorrect, the regularizer pushes the model to produce the same output on the augmented input. The implicit assumption is that matching the clean behavior, whatever it is, is always beneficial.

ROMA challenges this assumption directly and demonstrates it is false for reasoning tasks. The correctness-conditioning mechanism — multiplying the invariance penalty by an indicator I[R > 0] so it fires only when the clean-image trajectory was correct — is not an incremental refinement. It embodies a qualitatively different principle: invariance should be enforced only for desirable behaviors, not all behaviors. Pushing the model to be consistently wrong — producing the same incorrect reasoning chain regardless of image quality — is actively harmful. The model becomes "robustly incorrect," which is worse than brittleness because a brittle model at least has a chance of getting the right answer on some views.

This insight is backed by the ablation in Table 6, where removing correctness conditioning drops seen-degradation accuracy by 2.2% and unseen-degradation accuracy by 2.2%. The magnitude matters: correctness conditioning contributes roughly as much to ROMA's robustness gains as the auxiliary PG loss (which contributes ~1.6-1.8%, Table 4) and the worst-case optimization (which contributes ~1.6-1.8%, Table 3). It is not a minor tweak — it is one of the three pillars of the method, and its absence is as damaging as removing either of the other two.

Why hasn't this been recognized before? In the domains where invariance regularization was developed — image classification, continuous control RL — the problem of "robustly incorrect" outputs is less salient. For image classification, if the model predicts the wrong class on the clean image, matching that prediction under augmentation is indeed harmful, but wrong predictions are typically random (not systematically wrong in a way that gets reinforced), and the accuracy metric only penalizes the clean prediction anyway. For continuous control RL, small perturbations rarely flip the reward sign — an action that is slightly suboptimal on the clean state is still approximately correct under augmentation, so invariance is generally beneficial. But for reasoning MLLMs, a wrong trajectory is systematically wrong — the model follows a logically flawed chain that produces a specific incorrect answer — and invariance would lock in that specific error across all image qualities. The correctness condition is thus a domain-specific insight about the structure of errors in multimodal reasoning: errors are coherent and systematic, not random, so invariance on errors is genuinely damaging.

Significance: This is a conceptual contribution that refines the invariance regularization literature. It argues that "what to be invariant to" is only half the question; "what behaviors should be made invariant" is equally important and has been overlooked. The correctness mask is a simple mechanism, but the principle it embodies — that regularization should be gated on outcome quality — has implications beyond visual robustness, potentially applying to any setting where invariance is enforced over outputs that can be systematically wrong.

Innovation 3: Worst-Case Multi-View Optimization as a Computationally Tractable Adversarial Robustness Objective for Autoregressive Models

Adversarial training — optimizing against the worst-case perturbation — is well-established in image classification (adversarial examples, PGD attacks) but has been largely absent from RL-based LLM fine-tuning, for good reason. In autoregressive generation, the "output" is not a single classification decision but a sequence of tokens sampled from a distribution, making the space of possible perturbations and the definition of "worst-case" substantially more complex. Existing RL robustness methods (DrAC, RAD) average over randomly sampled augmentations, which is computationally simple but provides a weak robustness signal — easy augmentations dilute the gradient from hard ones.

ROMA's worst-case formulation (Equation 3) is notable not because maximization over augmentations is novel (it is standard in adversarial training), but because it makes worst-case optimization computationally feasible in the context of critic-free, autoregressive MLLM fine-tuning. The key enabler is the per-token KL divergence formulation (Equation 2) applied in teacher-forcing mode: because the model is only computing log-probabilities for existing tokens (not generating new sequences), evaluating K different degraded views requires only K forward passes through the model, with no sampling and no separate reward computation. The cost scales linearly with K, and the paper shows that K = 3 is sufficient for meaningful gains. This wouldn't be feasible if each degraded view required a separate rollout (sampling N = 8 trajectories per view, computing rewards, etc.), which would multiply the computational cost by K × N.

The ablation in Table 3 shows that this matters concretely: replacing worst-case maximization with mean-penalty averaging degrades performance by 1.6% on seen and 1.8% on unseen corruptions. The paper interprets this as evidence that "averaging the invariance penalty is insufficient for securing robustness" — the worst-case formulation forces the model to handle the specific perturbation that causes the largest distributional shift, providing a stronger gradient signal than a penalty diluted across easy views.

Why K = 3 and not larger? The sensitivity analysis (Table 7) shows that performance improves from K = 1 to K = 2 to K = 3 (59.5% → 60.7% → 61.6% on seen degradations), but slightly degrades at K = 4 (61.3%). The paper suggests this may be because too many views "dilute the worst-case signal or exceed the model's capacity to simultaneously match multiple highly divergent distributions," which is a plausible interpretation: with more views, the worst-case penalty becomes a harder optimization target, and the model may not have sufficient capacity to simultaneously minimize divergence against many different perturbation types.

Significance: This is an incremental but practically important contribution. Worst-case optimization over augmentations is not conceptually new, but its feasibility and demonstrated benefit in the specific context of critic-free, teacher-forced MLLM fine-tuning is novel. The paper shows that even a small number of views (K = 3) with a simple max-over-KL formulation provides a meaningful robustness boost, establishing a practical template for adversarial-style regularization in LLM training without the computational burden of full adversarial example generation.

Innovation 4: The Auxiliary Policy Gradient Resolves the Tension Between Invariance and Exploration in Critic-Free RL

The paper identifies a specific failure mode that arises when invariance regularization is applied without a companion learning signal: policy collapse under excessive KL regularization. The KL penalty (Equation 2) is purely a conservative force — it pushes the degraded-view token distribution toward the clean-view distribution. If the KL penalty dominates the optimization, the model can satisfy it trivially by outputting a constant, context-independent distribution (e.g., always predicting the same few tokens). This satisfies distributional consistency — the distribution never changes regardless of input — but destroys reasoning capability entirely.

In DrAC-style actor-critic methods, this failure mode is mitigated by the value network: even if the policy-side regularization is strong, the critic provides an independent reward-maximizing gradient that prevents collapse. But in critic-free GRPO, there is no value network to provide this counterbalancing force. The auxiliary policy gradient (Equation 4) is ROMA's solution to this specific architectural constraint: it provides a direct, reward-driven gradient signal computed on degraded views, so the model is not just penalized for deviating from clean behavior but is actively rewarded for maintaining high-probability on tokens that contributed to correct reasoning.

What makes this term novel is not the clipped surrogate formulation (which is standard PPO), but the anchoring of the advantage to clean-image rollouts. The importance sampling ratio ρ_t uses the current policy's probability under the degraded view divided by the old policy's probability under the clean view (Equation 4). This is a hybrid: the model is evaluated under degradation but judged against a clean-reference standard. The paper's argument is that this provides a reliable learning signal because the advantages A(v, x, y) reflect genuine reasoning quality (computed on clean rollouts), while the probability ratio ρ_t reflects the model's current behavior under degradation. The result is a gradient that says "if this token was part of a good trajectory on the clean image, make it more likely under degradation too" — without ever needing to determine whether the model would have answered correctly on the degraded image directly.

The ablation in Table 4 confirms this is necessary in the critic-free setting: removing the auxiliary PG costs 1.6% on seen and 1.8% on unseen degradations. The paper notes that the KL penalty "does not provide a sufficient learning signal to actively solve the reasoning task under visual occlusion." This is a diagnostic contribution: it identifies that invariance regularization alone is insufficient in critic-free architectures and provides empirical evidence for the specific mechanism (lack of reward-driven gradient under degradation).

Significance: This is an incremental refinement that addresses an architecture-specific limitation. The auxiliary PG is not a general contribution to robustness research — it is a necessary adaptation of robustness principles to the critic-free RL setting. The paper demonstrates that when you remove the value network (as GRPO does), you must replace its regularizing influence with something else, and the auxiliary PG anchored to clean advantages is one effective solution. Whether other solutions (e.g., a separate reward model applied to degraded views, or a learned invariance bonus) would work better is an open question, but the identification of the collapse problem and the demonstration that an auxiliary PG resolves it is a practically useful contribution for anyone building robustness into critic-free LLM training pipelines.

Innovation 5: Robustness to Visual Degradation as an Optimization-Dynamics Problem, Not a Data Problem

The paper's deepest conceptual move is implicit but pervasive: it reframes visual robustness for MLLMs from a data problem (what images do we train on?) to an optimization-dynamics problem (what gradients do we apply and when?). The standard approach in computer vision — and in the deep RL robustness literature (RAD, DrQ, DrAC) — treats robustness as fundamentally about data diversity: expose the model to more varied inputs, and it will learn to be invariant. NoisyRollout (reference [17]) follows this tradition, injecting augmented images directly into the RL generation loop.

ROMA argues — through its design and its results — that for critic-free RL fine-tuning of MLLMs, the data-centric approach fails for reasons internal to the optimization process: reward poisoning when generating from degraded inputs, policy collapse when regularizing without a companion learning signal, and systematic-error amplification when enforcing invariance unconditionally. These are not data problems — they are problems with how gradients flow through the model. ROMA's solution is correspondingly optimization-centric: stop gradients on clean-view logits (preventing corruption of clean representations), gate regularization on correctness (preventing reinforcement of errors), and balance conservative KL forces with an auxiliary reward-driven gradient (preventing collapse).

This reframing matters because it changes what future work should focus on. If robustness were a data problem, the solution would be more diverse training sets, better augmentation pipelines, or synthetic degradation generation. ROMA's results suggest these are unlikely to be sufficient — the GRPO baseline already shows that more RL training on clean data increases brittleness, implying that the optimization process itself is the source of fragility. The paper's ablation on α and β (Tables 5 and 8) shows that the balance between the auxiliary PG and KL penalty is fragile — too little regularization fails, too much degrades performance — indicating that the optimization landscape has narrow basins of attraction for robust policies, and data augmentation alone (without careful gradient control) is unlikely to find them.

This is consistent with a broader trend in deep learning where optimization dynamics — learning rates, gradient clipping, regularization schedules — prove as important as data quantity for generalization. The paper extends this insight to the specific domain of visual robustness in multimodal reasoning, arguing that how you regularize matters more than what data you regularize on. The fact that ROMA achieves its robustness gains using only the same 30K training samples as GRPO (no additional data, no new augmentation sources) and outperforms NoisyRollout (which does use augmented rollouts) supports this claim: modifying the optimization dynamics is more effective than modifying the training data.

Significance: This is a fundamental conceptual reframing, but one that the paper does not fully develop theoretically. The claim that robustness is an optimization-dynamics problem is supported empirically (ROMA outperforms data-centric baselines) but lacks a formal analysis of why the optimization landscape has this structure. Future work could analyze the loss landscape of GRPO-trained MLLMs under degradation, characterize the basins of attraction for robust vs. brittle policies, and provide theoretical justification for why gradient-level interventions (stop-gradient, correctness-gating, auxiliary PG) succeed where data-level interventions fail. The paper plants the flag for this direction but leaves the theoretical work to future efforts.

5. Experimental Analysis

Evaluation Methodology

  • Dataset. The paper uses the MMRL30k dataset (Zhu et al., 2025, reference [43]), which contains approximately 30,000 samples for multimodal reasoning training. This is the training dataset for all RL fine-tuning experiments. Evaluation is performed on seven multimodal reasoning benchmarks: MathVista (Lu et al., 2023), WeMath (Qiao et al., 2024), ChartQA (Masry et al., 2022), LogicVista (Xiao et al., 2024), MMStar (Chen et al., 2024), VisualPuzzles (Song et al., 2025), and RealWorldQA (xAI, 2024). These benchmarks collectively cover mathematical problem-solving, chart understanding, logical inference, general visual reasoning, and real-world visual question answering, providing a diverse test of multimodal reasoning capability.

  • Base model(s). All experiments use Qwen3-VL 4B and 8B Instruct models (Bai et al., 2025, reference [1]). These are instruction-tuned multimodal large language models that accept interleaved text and image inputs and generate autoregressive text outputs. The paper chooses these models because they are "representative of modern MLLM capabilities" and exist in a parameter range where RL fine-tuning is computationally feasible while still demonstrating strong baseline reasoning performance. The 4B and 8B variants allow the paper to test whether the method's effectiveness scales with model size. The base instruction-tuned models (before RL fine-tuning) serve as the lower-bound reference point, achieving 65.3% (4B) and 66.8% (8B) average clean accuracy across the seven benchmarks (Tables 1 and 2).

  • Metrics. The primary metric is accuracy (%) on each benchmark — the fraction of test questions for which the model's extracted final answer matches the reference answer. Answer extraction is performed by Qwen2.5-72B-Instruct (reference [33]), which parses the model's complete response (including the <thinking> chain-of-thought) and extracts the final answer for comparison against the reference. This follows standard practice from prior work (references [15, 16, 43]). For the main results tables (Tables 1, 2), the paper reports average accuracy across benchmarks, computed as the macro-average (equal weight per benchmark) rather than micro-average (weighted by benchmark size). For degradation evaluations, results are reported as macro-averages across all perturbation types within the seen and unseen pools (e.g., the "Seen Degradations" row reports the mean accuracy across Gaussian blur, Gaussian noise, JPEG compression, and resolution downscaling evaluations). The clean-to-degraded accuracy gap (the difference between Clean and Seen/Unseen averages) serves as the primary measure of robustness.

  • Baselines. The paper evaluates against several categories of baselines. Internal controlled baselines (trained and evaluated by the authors under identical conditions): (1) Base model: the pretrained Qwen3-VL Instruct model without any RL fine-tuning — this establishes the pre-RL performance floor. (2) GRPO: the base model fine-tuned via standard Group Relative Policy Optimization on the same MMRL30k training data, with the same hyperparameters (120 steps, 8 rollouts per input, learning rate 1e-6, rollout temperature 1.0) — this is the direct comparison point since ROMA builds on GRPO with additional regularization terms. External baselines (results taken from prior publications, evaluated on the same benchmarks but trained with potentially different data and procedures): (3) NoisyRollout-7B (Liu et al., 2025, reference [17]) — reinforces visual reasoning by directly injecting data augmentation into the RL generation phase, representing the naïve augmentation approach that ROMA argues against. (4) PAPO-7B (Wang et al., 2025, reference [35]) — penalizes the policy when outputs remain unchanged under heavy masking, encouraging visual grounding. (5) Vision-R1-7B (Huang et al., 2025, reference [10]) — uses vision-grounded prompts for multi-step logic, trained on WeMath (its WeMath performance is omitted from the table). (6) VL-Rethinker-7B (Wang et al., 2025, reference [34]) — incentivizes self-reflection in vision-language models. (7) OpenVLThinker-7B (Deng et al., 2025, reference [6]) — uses iterative SFT-RL cycles for complex vision-language reasoning. The external baselines provide broader context but are not perfectly controlled comparisons since they differ in base model, training data, and hyperparameters.

  • Generation budget / compute accounting. The paper measures compute in terms of RL training steps (120 steps for all methods) and rollout budget (8 rollouts per input, rollout batch size 256, global batch size 128). ROMA's additional forward passes for multi-view evaluation (KL penalty and auxiliary PG computation) introduce computational overhead during training, but this overhead is not explicitly quantified in terms of FLOPs or wall-clock time relative to GRPO. The paper does not compare methods at matched total FLOPs during training — all methods are trained for the same number of steps with the same base model architecture. For evaluation, all methods are compared at identical test-time compute (a single greedy or sampled generation per test example under each degradation condition), so test-time compute is normalized by construction. The key efficiency claim is therefore about sample efficiency during training (achieving better robustness from the same number of training samples and steps) rather than inference-time compute savings.

  • Cross-validation / statistical protocol. The paper does not employ explicit cross-validation or statistical significance testing. For each experimental configuration (base model, GRPO, ROMA, ablation variants), the model is trained once and evaluated on the full test sets of the seven benchmarks. Results are reported as point estimates without confidence intervals or error bars. The sensitivity analyses (Tables 5, 7, 8) sweep hyperparameters and report the resulting accuracy, but do not estimate variance. The main results tables (Tables 1, 2) report accuracy to one decimal place, implying precision of approximately ±0.05 percentage points, but the actual measurement variance (from random seed, data ordering, etc.) is not characterized. This is a meaningful limitation: with a training set of 30K samples, 120 training steps, and 8 rollouts per input, there is inherent stochasticity in the RL training process (from rollout sampling, augmentation sampling, and batch composition) that could cause run-to-run variance exceeding the reported performance gaps (e.g., the +2.4% seen-degradation improvement of ROMA over GRPO at 8B). Without multiple runs or variance estimates, it is difficult to assess whether the reported differences are statistically reliable or within the noise floor of training stochasticity.

Main Quantitative Results

Clean-Input Performance Preservation

The first requirement for any robustness method is that it must not degrade performance on clean inputs — a model that is robust to noise but worse on clean data is rarely desirable in practice. Tables 1 and 2 establish that ROMA satisfies this constraint for both model sizes.

For the 4B model (Table 1):

  • Base model clean accuracy: 65.3%
  • GRPO clean accuracy: 67.7% (+2.4% over base)
  • ROMA clean accuracy: 68.2% (+0.5% over GRPO, +2.9% over base)

ROMA achieves a small improvement over GRPO on clean data, which the paper attributes to the auxiliary policy gradient providing additional learning signal even on clean inputs. The gains are distributed across benchmarks: ROMA improves over GRPO on MathVista (78.4% vs. 78.0%), LogicVista (60.0% vs. 58.0%), MMStar (69.9% vs. 69.1%), and VisualPuzzles (42.3% vs. 40.8%), while slightly underperforming on WeMath (76.4% vs. 76.6%), ChartQA (80.6% vs. 81.3%), and RealWorldQA (69.5% vs. 70.1%).

For the 8B model (Table 2):

  • Base model clean accuracy: 66.8%
  • GRPO clean accuracy: 68.9% (+2.1% over base)
  • ROMA clean accuracy: 68.7% (−0.2% relative to GRPO, +1.9% over base)

On the 8B model, ROMA and GRPO are effectively tied on clean data. The benchmark-level pattern differs slightly from 4B: ROMA improves over GRPO on MathVista (78.5% vs. 78.4%), WeMath (77.9% vs. 77.6%), and LogicVista (62.1% vs. 60.8%), while underperforming on ChartQA (80.8% vs. 81.5%), MMStar (69.5% vs. 70.1%), VisualPuzzles (42.5% vs. 43.5%), and RealWorldQA (69.9% vs. 70.6%).

The key takeaway is that ROMA clean performance is within ±0.2% of GRPO at 8B and +0.5% at 4B, confirming that "our anchored optimization framework successfully preserves foundational reasoning capabilities without compromising baseline performance" (Section 4.2). ROMA is not trading clean accuracy for robustness — it achieves both simultaneously.

Compared to external baselines, ROMA's 8B clean performance (68.7%) outperforms OpenVLThinker-7B (57.9%), PAPO-7B (64.0%), VL-Rethinker-7B (61.7%), and NoisyRollout-7B (62.8%), though these comparisons are confounded by different base models, training data, and training procedures.

Robustness to Seen Visual Degradations

The central empirical question is whether ROMA improves robustness against the specific corruption types encountered during training. Tables 1 and 2 report macro-averaged accuracy across the four seen degradation types (Gaussian noise, Gaussian blur, JPEG compression, resolution downscaling) at severity Level 3, which is deliberately outside the training distribution bounds (Appendix A.2).

For the 8B model (Table 2, Seen Degradations):

  • Base model: 58.9% (a drop of 7.9% from clean)
  • GRPO: 59.2% (a drop of 9.7% from clean)
  • ROMA: 61.6% (a drop of 7.1% from clean)

ROMA outperforms GRPO by +2.4 percentage points on seen degradations. The absolute improvement (61.6% vs. 59.2%) is modest but meaningful: ROMA recovers roughly one-quarter of the gap between GRPO's degraded performance and its clean performance. More importantly, ROMA's clean-to-degraded accuracy gap (7.1 percentage points) is substantially smaller than GRPO's gap (9.7 percentage points), indicating that ROMA's robustness gains are not simply a consequence of higher baseline performance — the model is genuinely less sensitive to perturbations.

Benchmark-level analysis (Tables 10, 11, 12 for the 8B model breakdown):

  • MathVista: ROMA (73.3%) outperforms GRPO (71.2%) by +2.1%. Gains are concentrated in Gaussian noise (+2.1%) and JPEG compression (+2.1%).
  • WeMath: ROMA (77.3%) outperforms GRPO (75.0%) by +2.3%. The largest gain is on resolution downscaling (77.2% vs. 74.9%).
  • ChartQA: ROMA (49.1%) slightly outperforms GRPO (48.0%) by +1.1%. This benchmark shows the largest degradation across all methods (clean: ~81%, degraded: ~48-49%), suggesting chart understanding is particularly sensitive to visual perturbation.
  • LogicVista: ROMA (57.5%) outperforms GRPO (52.7%) by +4.8% — the largest per-benchmark improvement. This is a notable finding: logical reasoning from visual premises appears to benefit disproportionately from ROMA's invariance regularization.
  • MMStar: ROMA (66.3%) outperforms GRPO (64.0%) by +2.3%.
  • VisualPuzzles: ROMA (41.7%) outperforms GRPO (38.9%) by +2.8%.
  • RealWorldQA: ROMA (66.0%) outperforms GRPO (64.8%) by +1.2%.

The improvement is consistent but not uniform across benchmarks. LogicVista shows a substantially larger gain (+4.8%) than ChartQA (+1.1%), suggesting that the type of reasoning matters — tasks requiring deduction from explicitly presented visual information may benefit more from perceptual invariance than tasks requiring extraction of quantitative information from visually complex displays.

For the 4B model (Table 1, Seen Degradations):

  • Base model: 57.5% (a drop of 7.8% from clean)
  • GRPO: 59.0% (a drop of 8.7% from clean)
  • ROMA: 60.7% (a drop of 7.5% from clean)

ROMA improves over GRPO by +1.7% on seen degradations. The pattern is qualitatively similar to the 8B results: ROMA reduces the clean-to-degraded gap from 8.7% to 7.5%, with consistent benchmark-level improvements except on ChartQA and RealWorldQA where the gains are marginal (+0.4% and +0.9% respectively). The smaller absolute gain at 4B compared to 8B (+1.7% vs. +2.4%) suggests that ROMA's benefits may scale with model capacity — larger models have more representational capacity to learn the invariance without sacrificing reasoning fidelity.

Compared to external 7B baselines on seen degradations (Table 2), ROMA-8B (61.6%) outperforms NoisyRollout-7B (54.9%), PAPO-7B (55.4%), VL-Rethinker-7B (54.1%), and OpenVLThinker-7B (50.6%). However, these comparisons are inherently confounded by differences in base model performance: the Qwen3-VL 8B base model already achieves 58.9% on seen degradations, higher than all 7B external baselines except NoisyRollout-7B, making it difficult to attribute ROMA's superiority solely to the method rather than the stronger base model.

Generalization to Unseen Visual Degradations

A stronger test of robustness is whether the learned invariance transfers to corruption types never seen during training. Tables 1 and 2 report accuracy under five unseen degradation types (motion blur, salt-and-pepper noise, speckle noise, posterization, pixelation) at severity Level 3.

For the 8B model (Table 2, Unseen Degradations):

  • Base model: 53.4% (a drop of 13.4% from clean)
  • GRPO: 54.0% (a drop of 14.9% from clean)
  • ROMA: 56.3% (a drop of 12.4% from clean)

ROMA outperforms GRPO by +2.3 percentage points on unseen degradations. The absolute improvement is similar in magnitude to the seen-degradation gain (+2.4%), which is noteworthy: ROMA's regularization is generalizing to perturbation types it has never encountered, not merely memorizing the training-time corruptions. The effectiveness on unseen degradations (OOD corruptions with OOD severity) is roughly equal to the effectiveness on seen degradations (OOD severity only), suggesting the invariance penalty encourages a general perceptual stability rather than corruption-specific robustness.

The clean-to-unseen-degraded gap for ROMA (12.4%) is meaningfully smaller than GRPO's gap (14.9%), indicating that ROMA's robustness transfers effectively to novel noise structures. However, the gap remains large in absolute terms (12.4 percentage points), and unseen-degradation accuracy (56.3%) is substantially below clean accuracy (68.7%), indicating that while ROMA improves robustness, visual degradation in severe OOD settings remains a significant challenge.

Benchmark-level pattern for unseen degradations (Tables 10, 11, 12, 8B breakdown):

  • MathVista: ROMA (64.8%) vs. GRPO (63.0%), +1.8%.
  • WeMath: ROMA (70.4%) vs. GRPO (66.8%), +3.6%. This is the largest per-benchmark OOD gain, suggesting mathematical reasoning with visual elements benefits strongly from learned invariance.
  • ChartQA: ROMA (44.1%) vs. GRPO (43.0%), +1.1%. ChartQA remains the most difficult benchmark under degradation regardless of method.
  • LogicVista: ROMA (50.8%) vs. GRPO (47.1%), +3.7% — the second-largest gain, consistent with the seen-degradation finding that LogicVista benefits disproportionately from ROMA.
  • MMStar: ROMA (62.1%) vs. GRPO (59.8%), +2.3%.
  • VisualPuzzles: ROMA (38.0%) vs. GRPO (36.8%), +1.2%.
  • RealWorldQA: ROMA (63.6%) vs. GRPO (61.4%), +2.2%.

For the 4B model (Table 1, Unseen Degradations):

  • Base model: 51.7% (a drop of 13.6% from clean)
  • GRPO: 53.8% (a drop of 13.9% from clean)
  • ROMA: 55.1% (a drop of 13.1% from clean)

ROMA improves over GRPO by +1.3% on unseen degradations. The OOD gain at 4B is smaller than at 8B (+1.3% vs. +2.3%), consistent with the seen-degradation pattern where the 8B model benefits more. This may indicate that learning perceptual invariance is capacity-dependent — larger models can learn more general invariance patterns that transfer better OOD.

Compared to external 7B baselines on unseen degradations (Table 2), ROMA-8B (56.3%) outperforms NoisyRollout-7B (50.1%), PAPO-7B (50.7%), VL-Rethinker-7B (49.6%), and OpenVLThinker-7B (46.4%). The gap between ROMA-8B and the external baselines (approximately 5-10 percentage points) is larger on unseen degradations than on seen degradations, suggesting that ROMA's invariance penalty generalizes better OOD than data-augmentation-based approaches.

Robustness Across Increasing Corruption Severity Levels

Figure 2 presents a critical additional analysis: how does robustness scale as degradation severity increases from Clean to Level 1 (mild) to Level 2 (moderate) to Level 3 (severe, OOD magnitude)? This tests whether ROMA's advantages are concentrated at specific severity levels or consistent across the spectrum.

For seen degradations (Figure 2a, 8B model):

  • Clean: ROMA (68.7%), GRPO (68.9%), Base (66.8%). All three methods are clustered within 2.1 percentage points.
  • Level 1: ROMA and GRPO are approximately tied (both ~65-66%, exact values not reported in text), with ROMA showing a modest advantage visible in the figure.
  • Level 2: ROMA begins to pull ahead of GRPO, with a visible separation in the figure.
  • Level 3: ROMA (61.6%) clearly outperforms GRPO (59.2%) by +2.4%. The Base model is at 58.9%.

The key observation is the monotonically growing advantage of ROMA over GRPO as severity increases. At Level 1, the methods are nearly indistinguishable. At Level 2, ROMA gains a small advantage. At Level 3 (OOD magnitude), ROMA's advantage is largest. This is consistent with ROMA's design: the worst-case KL penalty specifically targets "the augmentation that induces the maximum divergence" (Equation 3), and at higher severities, the divergence between clean and degraded views is larger, making the regularization signal stronger and more targeted.

For unseen degradations (Figure 2b, 8B model):

  • Clean: ROMA (68.7%), GRPO (68.9%), Base (66.8%).
  • Level 1: ROMA and GRPO show a small separation (~1 percentage point, visible in the figure).
  • Level 2: The gap widens to approximately 1.5 percentage points.
  • Level 3: ROMA (56.3%) outperforms GRPO (54.0%) by +2.3%.

The OOD severity scaling pattern mirrors the seen pattern: ROMA's advantage grows with severity. This is a strong result — it suggests that ROMA's invariance penalty is not merely preventing the model from overfitting to specific training-time corruption parameters but is inducing a more fundamental distributional stability that persists and even strengthens at corruption magnitudes the model has never experienced.

A concerning secondary observation: GRPO's degradation is steeper than the Base model's in both seen and unseen settings. In Figure 2a, the GRPO curve drops from 68.9% to 59.2% (a 9.7 percentage point decline), while the Base curve drops from 66.8% to 58.9% (a 7.9 percentage point decline). This means standard RL fine-tuning on clean data increases fragility to perceptual perturbations — a finding the paper notes in Section 4.2 but does not deeply analyze. The implication is that GRPO's optimization, which sharpens the policy toward high-reward trajectories on clean inputs, simultaneously makes the policy more sensitive to input perturbations, possibly because it reduces the entropy of the token distribution and thus amplifies the effect of small input changes. ROMA's regularization counteracts this effect, bringing the clean-to-degraded drop back in line with (or below) the Base model's while maintaining higher absolute accuracy.

Performance Across Individual Degradation Types

The detailed breakdowns in Tables 10, 11, and 12 (for the 8B model) reveal important heterogeneity in how different corruption types affect different benchmarks, and where ROMA's gains are concentrated.

ChartQA's catastrophic collapse under blur and resolution downscaling. ChartQA shows a dramatic performance drop under Gaussian blur and resolution downscaling across all methods (Tables 10, 11, 12):

  • Clean: 79.4% (Base), 81.5% (GRPO), 80.8% (ROMA)
  • Gaussian blur: 14.6% (Base), 15.8% (GRPO), 16.9% (ROMA)
  • Resolution scale: 18.2% (Base), 19.3% (GRPO), 20.4% (ROMA)

This represents a decline of approximately 60-65 percentage points on these specific corruption types — the model essentially fails at chart reading when the image is blurred or downscaled. ROMA provides a small improvement (+1.1% over GRPO on Gaussian blur, +1.1% on resolution scale) but does not come close to recovering the lost performance. This suggests that chart understanding involves fine-grained visual detail extraction (reading axis labels, distinguishing bar heights, parsing legend entries) that is fundamentally compromised by these perturbations, and distributional invariance regularization cannot compensate for the loss of necessary perceptual information. This is an important boundary condition: ROMA helps when the model can still extract sufficient information from the degraded image but is uncertain; it cannot help when the information is simply destroyed by the perturbation.

JPEG compression and Gaussian noise show modest degradation. In contrast to blur and resolution, JPEG compression and Gaussian noise cause much smaller degradation on most benchmarks. For example, on MathVista (8B GRPO, Table 11): Gaussian blur (67.5%), Gaussian noise (74.0%), JPEG compression (75.8%), Resolution scale (67.5%). The rank order is consistent: resolution downscaling and blur are the most damaging seen perturbations, while JPEG compression and moderate Gaussian noise are less disruptive. ROMA's gains are distributed across all perturbation types but tend to be slightly larger on the more damaging ones (blur, resolution), consistent with the worst-case KL formulation targeting the perturbations that cause the largest distributional shift.

Unseen degradations show a different rank order. For the 8B GRPO baseline (Table 11), unseen degradation accuracy ranges from: Pixelation (6.3% on ChartQA — catastrophic) to Speckle noise (75.0% on WeMath — mild). Motion blur and pixelation are consistently the most damaging unseen perturbations, while speckle noise and posterization are the least damaging. ROMA's OOD improvements (Table 12) are generally consistent across perturbation types, with WeMath showing the largest gains (+3.6% average across unseen types) and ChartQA the smallest (+1.1%). Notably, ROMA's improvements on pixelation — the most damaging unseen perturbation — are modest (e.g., ChartQA: 7.4% vs. 6.3%), consistent with the interpretation that extreme information destruction cannot be compensated by regularization.

Benchmark-Level Robustness Profiles

Several benchmarks exhibit distinctive robustness profiles that illuminate the nature of ROMA's regularization:

WeMath shows the strongest absolute and relative benefit from ROMA. Under seen degradations (8B), ROMA reaches 77.3% vs. GRPO's 75.0% (+2.3%). Under unseen degradations, ROMA reaches 70.4% vs. GRPO's 66.8% (+3.6%). WeMath involves mathematical reasoning with visual elements, and the paper's results suggest that mathematical reasoning chains are particularly amenable to invariance regularization — perhaps because mathematical reasoning follows structured, step-by-step logical patterns that are less dependent on fine-grained visual detail and more dependent on extracting the correct initial problem representation. Once the problem is correctly parsed (even from a degraded image), the reasoning chain is primarily text-based and benefits from distributional stability.

LogicVista shows the second-largest gains, with ROMA improving over GRPO by +4.8% on seen degradations and +3.7% on unseen degradations. LogicVista tests logical reasoning in visual contexts, which may also benefit from the structured, deductive nature of the task: if the logical premises can be extracted from the degraded image, the inference steps are independent of image quality.

VisualPuzzles has the lowest absolute accuracy across all methods (42.5% clean for ROMA-8B), and ROMA's improvements are moderate (+2.8% seen, +1.2% unseen). VisualPuzzles tests reasoning that is "decoupled from domain knowledge" (reference [31]) and likely requires careful visual inspection, making it intrinsically sensitive to degradation regardless of regularization.

ChartQA shows the smallest absolute improvements (+1.1% seen, +1.1% unseen) despite having the largest clean-to-degraded gap. This suggests that the primary failure mode on ChartQA under degradation is not distributional shift in reasoning (which ROMA addresses) but loss of critical visual information (which ROMA cannot address). ChartQA requires reading precise numerical values from visual elements, and when those values are blurred beyond legibility, no amount of invariance regularization can recover them.

Ablation Studies and Robustness Checks

All ablation studies use the 8B model and report macro-averaged accuracy across the seven benchmarks for seen and unseen degradations at severity Level 3.

Choice of multi-view optimization strategy (Table 3). The paper compares the worst-case formulation (Equation 3, maximizing KL divergence over K = 3 views) against a mean-penalty baseline (averaging the KL divergence across all K views). On seen degradations, worst-case achieves 61.6% vs. mean-penalty's 60.0%, a difference of +1.6%. On unseen degradations, worst-case achieves 56.3% vs. mean-penalty's 54.5%, a difference of +1.8%. The benchmark-level breakdown shows that worst-case outperforms mean on every benchmark for seen degradations, with the largest gaps on LogicVista (57.5% vs. 53.5%, +4.0%) and MMStar (66.3% vs. 64.6%, +1.7%). For unseen degradations, the pattern is similar: worst-case outperforms mean on every benchmark, with the largest gaps on LogicVista (50.8% vs. 47.5%, +3.3%) and WeMath (70.4% vs. 67.2%, +3.2%). This demonstrates that worst-case optimization provides a strictly better robustness signal than mean-penalty averaging for this architecture and task — focusing the regularization budget on the hardest perturbation at each step yields more transferable invariance than diluting the penalty across easy and hard views.

Ablation on the auxiliary policy gradient loss (Table 4). Removing the auxiliary PG term (Equation 4) while keeping the worst-case KL penalty reduces seen-degradation accuracy from 61.6% to 60.5% (−1.1% averaged across benchmarks, or −1.6% when looking at the "Avg" column — the discrepancy is due to the Avg column being a macro-average while the per-benchmark differences are individual data points) and unseen-degradation accuracy from 56.3% to 55.4% (−0.9% averaged, or −1.6% per the "Avg" column). At the benchmark level, LogicVista shows the largest degradation from removing auxiliary PG: on seen degradations, 57.5% → 55.1% (−2.4%), and on unseen, 50.8% → 49.4% (−1.4%). WeMath also shows a notable drop: 77.3% → 76.9% (−0.4%) on seen, and 70.4% → 69.5% (−0.9%) on unseen. The auxiliary PG is less critical for benchmarks where the KL penalty alone provides sufficient gradient (e.g., ChartQA: 49.1% → 48.5%, only −0.6% on seen), consistent with the hypothesis that the auxiliary PG primarily prevents policy collapse under regularization in domains where the KL penalty would otherwise dominate the gradient.

Effect of correctness conditioning (Table 6). Removing the correctness mask (applying the invariance penalty unconditionally regardless of whether the clean-image trajectory was correct) reduces seen-degradation accuracy from 61.6% to 59.4% (−2.2%) and unseen-degradation accuracy from 56.3% to 54.1% (−2.2%). This is the single largest ablation effect across all three components — correctness conditioning contributes more to ROMA's robustness than either the worst-case optimization or the auxiliary PG. Benchmark-level analysis reveals the mechanism: the unconditional penalty degrades performance most on benchmarks where GRPO already produces a mix of correct and incorrect trajectories. On LogicVista, seen-degradation accuracy drops from 57.5% to 52.6% (−4.9%), and on WeMath, it drops from 77.3% to 74.7% (−2.6%). In contrast, on ChartQA (where most degradation-era errors are due to information loss rather than reasoning errors), the drop is smaller: 49.1% to 47.9% (−1.2%). This is consistent with the paper's hypothesis that enforcing invariance on incorrect trajectories locks in systematic errors — on benchmarks where reasoning errors are diverse (LogicVista), reinforcing those errors via invariance is particularly damaging, while on benchmarks where errors are dominated by information loss (ChartQA), the unconditional penalty is less harmful because there are fewer coherent incorrect reasoning chains to reinforce.

Sensitivity analysis on the auxiliary coefficient α (Table 5). Varying α from 0.05 to 0.10 to 0.15 shows a clear inverse-U pattern: seen-degradation accuracy peaks at α = 0.10 (61.6%), with α = 0.05 at 60.5% (−1.1%) and α = 0.15 at 60.0% (−1.6%). Unseen-degradation accuracy follows the same pattern: 56.3% at α = 0.10, 55.2% at α = 0.05 (−1.1%), 54.8% at α = 0.15 (−1.5%). The optimal α balances two failure modes: at α = 0.05, the auxiliary PG is too weak to provide sufficient reward-driven gradient under degradation, leading to partial policy collapse; at α = 0.15, the auxiliary PG dominates the clean-image RL objective, over-regularizing and degrading performance. The sensitivity is moderate — a ±0.05 change in α shifts accuracy by approximately 1-1.5 percentage points — suggesting the method is reasonably robust to this hyperparameter but not insensitive.

Sensitivity analysis on the number of augmented views K (Table 7). Increasing K from 1 to 2 to 3 improves seen-degradation accuracy: 59.5% → 60.7% → 61.6%. At K = 4, accuracy slightly decreases to 61.3% (−0.3% from K = 3). Unseen degradation follows the same trend: 54.4% → 55.2% → 56.3% at K = 3, then 56.0% at K = 4. The improvement from K = 1 to K = 2 is +1.2% (seen), and from K = 2 to K = 3 is +0.9% (seen), indicating diminishing returns. The slight degradation at K = 4 may indicate that with too many views, the worst-case penalty becomes too harsh — the model is forced to match a very difficult adversarial view at every step, which may exceed its representational capacity or cause gradient interference with the other objective terms. The paper selects K = 3 as the default, representing a practical balance between robustness signal strength and computational overhead (each additional view requires an additional teacher-forced forward pass).

Sensitivity analysis on the invariance penalty weight β (Table 8). Similar to α, β shows an inverse-U pattern: seen-degradation accuracy peaks at β = 0.10 (61.3% — note: this is 61.3% here vs. 61.6% in Table 2; the discrepancy may be due to averaging over different benchmarks or reporting a different aggregate, as the Table 8 caption says "Achieved at β = 0.10" matching the 61.3% value), with β = 0.05 at 59.4% (−1.9%) and β = 0.15 at 56.8% (−4.5%). The right tail degradation is steeper than the left tail: increasing β from 0.10 to 0.15 costs more (−4.5%) than decreasing it to 0.05 (−1.9%), indicating that over-regularization is more damaging than under-regularization. At β = 0.15, the invariance penalty likely forces the model to prioritize distributional matching over reasoning, causing the policy collapse the auxiliary PG is designed to prevent. On unseen degradations, the pattern is similar but less steep: 56.3% at β = 0.10, 55.3% at β = 0.05 (−1.0%), 54.6% at β = 0.15 (−1.7%). The OOD results are less sensitive to β than the seen results, perhaps because the regularization at seen-degradation severities transfers to OOD severities with a saturating effect — once sufficient invariance is learned, additional regularization provides diminishing OOD returns while continuing to harm seen-degradation performance.

Critical Assessment

Claim 1: ROMA improves robustness to seen visual degradations by +2.4% (8B) while matching clean accuracy.

What the experiments demonstrate: The 8B results in Table 2 directly support this claim: ROMA achieves 61.6% on seen degradations vs. GRPO's 59.2% (a +2.4 percentage point improvement), while clean accuracy is 68.7% vs. 68.9% (a −0.2 percentage point difference, essentially tied). The seen-degradation improvement is consistent across all seven benchmarks, with LogicVista showing the largest gain (+4.8%) and ChartQA the smallest (+1.1%). The 4B results (Table 1) show a smaller but still positive gain (+1.7%) with a slight clean-accuracy improvement (+0.5%). The severity-scaling analysis (Figure 2a) shows that ROMA's advantage persists and grows with increasing degradation severity. The detailed per-perturbation breakdowns (Tables 10-12) show that ROMA improves over GRPO on all four seen perturbation types, with gains distributed roughly evenly.

What the experiments do not demonstrate: The experiments do not establish statistical reliability. With a single training run per configuration, 30K training samples, 120 training steps, and 500+ test questions spread across seven benchmarks (varying test set sizes per benchmark), the sampling variance from RL training stochasticity is unknown. A +2.4% improvement on macro-averaged accuracy across seven benchmarks is not trivially above potential noise, especially given that the clean-accuracy difference between ROMA and GRPO is −0.2% — if training variance is on the order of ±1-2 percentage points (which is plausible for 120-step GRPO training on 30K samples), the robustness improvement could be partially attributable to stochastic advantage. The paper should report results from multiple random seeds or at minimum provide an estimate of run-to-run variance.

The experiments also do not disentangle ROMA's robustness gain from its clean-accuracy performance at the per-benchmark level. On MathVista, ROMA clean accuracy (78.5%) is +0.1% above GRPO (78.4%), and ROMA seen-degradation accuracy (73.3%) is +2.1% above GRPO (71.2%). The robustness gain (+2.1%) is substantially larger than the clean gain (+0.1%), supporting ROMA's mechanism. But on MMStar, ROMA clean accuracy (69.5%) is −0.6% below GRPO (70.1%), while ROMA seen-degradation accuracy (66.3%) is +2.3% above GRPO (64.0%). The robustness improvement here is +2.3% despite worse clean performance, which is a stronger signal that ROMA's regularization is doing work independent of baseline accuracy. However, on RealWorldQA, ROMA clean accuracy (69.9%) is −0.7% below GRPO (70.6%), and the seen-degradation gain is only +1.2% (66.0% vs. 64.8%) — this could be consistent with ROMA simply preserving more of its (slightly lower) clean performance under degradation, rather than inducing genuine invariance. Per-benchmark variance in these patterns is not analyzed.

Verdict: The claim is supported directionally but would be strengthened by multi-seed results, per-benchmark robustness metrics normalized by clean performance (e.g., relative degradation ratios), and explicit statistical comparisons.

Claim 2: ROMA generalizes to unseen degradations (+2.3% at 8B) with zero-shot OOD robustness.

What the experiments demonstrate: Table 2 shows ROMA achieves 56.3% on unseen degradations vs. GRPO's 54.0% (+2.3%). The OOD gain is similar in magnitude to the seen-degradation gain (+2.4%), which is a strong signal for genuine generalization. The severity-scaling analysis (Figure 2b) confirms the advantage persists across corruption levels and grows with severity. The per-perturbation breakdowns (Tables 10-12) show ROMA outperforms GRPO on all five unseen perturbation types, with WeMath (+3.6% average) and LogicVista (+3.7%) showing the largest gains.

What the experiments do not demonstrate: The unseen degradation types, while structurally different from the training corruptions (motion blur vs. Gaussian blur, salt-and-pepper vs. Gaussian noise), share a common characteristic: they are all pixel-space perturbations applied to 2D images. The paper uses the term "OOD" to describe these corruptions, but they are OOD in a narrow sense — they are different parameterizations of the same class of transformations (additive noise, blur kernels, compression, downscaling) applied to static images. A genuinely OOD test would include corruptions that differ in kind, not just in type: geometric transformations (rotation, perspective warp), occlusions (random patches, superimposed text), lighting changes (brightness, contrast shifts), or domain shifts (photographs to sketches, natural images to document scans). The current unseen pool tests generalization across noise structures, not across fundamentally different categories of visual perturbation. The paper's claim of "OOD generalization" is therefore accurate within the ImageNet-C framework it adopts, but the term implies broader generalization than what was tested.

Additionally, the unseen-degradation evaluation uses Level 3 severity, which is OOD in magnitude for each perturbation. However, the paper does not report whether ROMA's advantage is larger on OOD severity with seen perturbation types (Level 3) compared to in-distribution severity with unseen perturbation types (Levels 1-2). Parsing this would reveal whether ROMA generalizes better across severities or across perturbation structures, which have different practical implications.

Verdict: The claim is supported for the specific notion of OOD tested (unseen noise structures with OOD magnitude), but the scope of "OOD generalization" is narrower than the term might suggest. Additional experiments with geometrically or semantically different perturbations would strengthen the claim.

Claim 3: The dual-forward-pass strategy avoids reward poisoning.

What the experiments demonstrate: The paper provides indirect evidence for this claim by comparing ROMA against GRPO (which never sees degraded images) and against external baselines like NoisyRollout (which does). ROMA (61.6% seen) outperforms NoisyRollout-7B (54.9% seen) by +6.7 percentage points (Table 2). However, this comparison is heavily confounded: NoisyRollout uses a different base model (7B vs. 8B), different training data, and a different RL algorithm. The paper does not implement an internal baseline that naïvely injects degraded images into GRPO rollouts — this would be the direct test of whether the dual-forward-pass strategy prevents reward poisoning. Without this ablation, the claim that reward poisoning is the reason the naïve approach fails is supported by conceptual argument and external comparison, but not by controlled experiment.

What would strengthen this claim: An internal ablation adding a "naïve augmented rollout" baseline — GRPO with the same training setup but with rollouts sampled on degraded images (with appropriate reward computation on those rollouts) — would directly test the reward poisoning hypothesis. The paper's conceptual argument (Section 1) is persuasive, but the experimental evidence is circumstantial.

Verdict: The claim is plausible and conceptually well-motivated but not directly experimentally validated. The comparison with NoisyRollout is suggestive but confounded by base model, data, and algorithmic differences.

Claim 4: The auxiliary policy gradient prevents policy collapse under KL regularization in critic-free settings.

What the experiments demonstrate: The ablation in Table 4 provides direct evidence: removing the auxiliary PG reduces seen-degradation accuracy by 1.6% and unseen-degradation accuracy by 1.8%. The paper interprets this as evidence that "relying solely on the token-level invariance penalty is restrictive" and that the auxiliary PG "provides a direct gradient signal that guides the policy toward correct reasoning steps despite the noise" (Section 4.3). The sensitivity analysis on α (Table 5) shows that too little auxiliary PG (α = 0.05) underperforms the optimal (α = 0.10), consistent with insufficient auxiliary signal. However, the ablation does not demonstrate collapse per se — the model without auxiliary PG still achieves 60.5% on seen degradations, far above random performance. The degradation is a 1.6% drop, not a catastrophic failure. This suggests the auxiliary PG provides a modest but consistent benefit, not a make-or-break safeguard against collapse. The claim of "preventing policy collapse" may overstate the empirical effect — the auxiliary PG appears to provide an incremental improvement rather than rescuing the model from degenerate behavior.

Verdict: The claim that the auxiliary PG is beneficial is supported by the ablation, but the characterization as preventing "policy collapse" is stronger than the evidence warrants. The degradation from removing the auxiliary PG is modest (1.6-1.8%) and does not indicate collapse.

Claim 5: Correctness conditioning prevents the model from becoming robustly incorrect.

What the experiments demonstrate: Table 6 provides strong supporting evidence: removing correctness conditioning causes the largest single ablation effect (−2.2% on both seen and unseen degradations). The benchmark-level breakdown is consistent with the mechanism: benchmarks where reasoning errors are systematic and coherent (LogicVista, −4.9% seen) degrade more than benchmarks where errors are dominated by information loss (ChartQA, −1.2% seen). This pattern matches the paper's argument that enforcing invariance on incorrect trajectories locks in systematic errors.

What the experiments do not demonstrate: The ablation shows that unconditional penalty is worse, but does not directly demonstrate why — the paper infers the "robustly incorrect" mechanism from the benchmark-level pattern, but does not provide trajectory-level analysis showing that unconditional regularization indeed increases the consistency of incorrect answers across clean and degraded views. Such analysis would require comparing per-question answer consistency across clean and degraded views for conditional vs. unconditional regularization — if unconditional regularization increases consistency of incorrect answers (the model gives the same wrong answer on clean and blurred versions of the same question), that would be direct evidence for the claimed mechanism.

Verdict: The claim is well-supported empirically (the ablation effect is large and pattern-consistent with the mechanism) but could be strengthened by direct trajectory-level analysis of answer consistency.

Overall Assessment of the Experimental Design

Strengths:

  • Diverse benchmark coverage. Seven benchmarks spanning mathematical reasoning, chart understanding, logical inference, and real-world VQA provide broad evaluation of multimodal reasoning capability. The consistent pattern across benchmarks (ROMA improves over GRPO on seen and unseen degradations on all seven) strengthens confidence that the method works generally, not just on specific task types.
  • Multiple corruption types and severities. The seen/unseen distinction and the three-level severity scaling (Figure 2) provide granular insight into where ROMA helps and where it doesn't, avoiding the common pitfall of reporting only single-corruption or single-severity results.
  • Comprehensive ablation suite. The three-component ablation (Tables 3, 4, 6) and three sensitivity analyses (Tables 5, 7, 8) systematically isolate the contribution of each design choice and characterize hyperparameter sensitivity. The finding that correctness conditioning is the single most important component (−2.2% when removed) is non-obvious and practically important.

Weaknesses:

  • No statistical characterization of variance. Single training runs for each configuration, no standard deviations or confidence intervals. This is the most significant methodological limitation: the reported gains (+1.7% to +2.4%) are small enough that run-to-run RL training variance could account for some or all of the difference. The paper's sensitivity analyses help characterize hyperparameter robustness but do not address stochastic training variance.
  • No internal naïve-augmented-rollout baseline. The core motivation for ROMA's dual-forward-pass design is avoiding reward poisoning, but this is never tested with a controlled comparison. A GRPO + degraded-rollout baseline would directly test whether the dual-forward-pass is necessary or merely one of several ways to achieve similar robustness.
  • Confounded external baseline comparisons. The comparisons with NoisyRollout, PAPO, Vision-R1, VL-Rethinker, and OpenVLThinker use different base models, training data, and training procedures. The paper acknowledges this implicitly (it separates them in the tables) but still uses them to contextualize ROMA's performance. The large performance gaps (e.g., ROMA-8B at 61.6% seen vs. NoisyRollout-7B at 54.9%) likely reflect base model quality differences at least as much as methodological differences.
  • Limited analysis of training dynamics. The paper reports final evaluation metrics but provides no analysis of how robustness develops during training (e.g., clean-vs-degraded accuracy curves over the 120 training steps). Such analysis could reveal whether ROMA's robustness emerges gradually or appears suddenly, whether clean and robust performance trade off during training, and whether longer training would continue to improve robustness or cause over-regularization.
  • No computational overhead quantification. ROMA's dual-forward-pass strategy adds computational cost during training (additional teacher-forced forward passes under degraded views), but the paper does not report this overhead relative to GRPO. If ROMA requires 2x or 3x the training time per step, the efficiency comparison with GRPO is incomplete. The paper's claim of achieving better robustness "from the same number of training samples and steps" (as I inferred above) is accurate for step count but ignores per-step cost differences.
  • Limited degradation diversity. The training corruptions are exclusively pixel-space perturbations (noise, blur, compression, downscaling). Real-world visual degradation includes many other categories — motion blur (tested only as unseen), lighting changes, perspective distortion, partial occlusion, text overlay artifacts — and ROMA's generalization to these would likely be weaker.
  • Single model family. All experiments use Qwen3-VL Instruct models. The paper's claims about the general applicability of ROMA to "critic-free RL fine-tuning of MLLMs" would be strengthened by results on at least one other model family (e.g., LLaVA, InternVL) to demonstrate that the method is not specific to Qwen3-VL's architecture or pretraining.

Missing experiments that would strengthen the paper:

  1. Multi-seed training runs (at least 3 per configuration) with reported means and standard deviations.
  2. A naïve-augmented-rollout baseline (GRPO with degraded images in the rollout phase).
  3. Training dynamics curves (clean accuracy and degraded accuracy vs. training step).
  4. Computational overhead analysis (wall-clock time or FLOPs comparison between GRPO and ROMA training).
  5. Trajectory-level consistency analysis (per-question answer agreement between clean and degraded views).
  6. Evaluation on at least one additional model family.
  7. Broader OOD corruption testing (geometric transforms, lighting changes, occlusions).
  8. Analysis of whether ROMA's regularization affects the model's calibration, generation diversity, or tendency to hallucinate on clean inputs.

6. Limitations and Trade-offs

6.1 The Difficulty Estimation Cost Is Not Accounted For in the Compute-Optimal Efficiency Claims

The assumption or constraint. The dual-forward-pass strategy — the core of ROMA's design — requires evaluating each clean-image trajectory under K = 3 degraded views using teacher-forced forward passes through the model (Section 3, Figure 1). Each of these forward passes processes the full trajectory length (~100-500 tokens depending on the reasoning chain) through the autoregressive model to compute token-level log-probabilities. The paper's headline efficiency claim — that ROMA achieves robustness improvements "from the same number of training samples and steps" — refers only to the RL training step count (120 steps) and the number of training samples (30K), explicitly excluding the additional computational cost of the dual-forward-pass evaluation.

The paper acknowledges the multi-view setup's computational footprint only indirectly, in the sensitivity analysis for K (Table 7), where the authors note they "select K = 3 as the default setting to maintain a computationally efficient training pipeline without sacrificing robust generalization." But this is a statement about relative efficiency within the multi-view framework, not a comparison against GRPO's single-forward-pass training. The paper provides no FLOPs count, no wall-clock time comparison, and no memory overhead analysis for ROMA training versus standard GRPO training.

The consequence. A practitioner evaluating whether to adopt ROMA needs to know the true training cost. The dual-forward-pass design adds at minimum 3 × additional forward passes per training step (for KL computation across K = 3 views) plus one additional forward pass for the auxiliary PG (on a randomly sampled view), totaling approximately 4× the forward-pass cost of GRPO per optimization step. The backward-pass cost also increases because gradients flow through the degraded-view logits in both the KL penalty and the auxiliary PG. If ROMA training takes 3-4× longer per step than GRPO, the fair comparison is not at matched step count but at matched total FLOPs or matched wall-clock time. Standard GRPO trained for 3-4× more steps might close or exceed ROMA's robustness gains simply through extended optimization. The paper provides no evidence to rule this out.

What evidence exists in the paper. None. The paper does not report training time, FLOPs, GPU-hours, or any computational cost metric for ROMA relative to GRPO. The sensitivity analysis on K (Table 7) reports only evaluation accuracy, not training cost. The appendix (A.1) reports standard training hyperparameters (learning rate, batch size, weight decay) but no cost accounting. The reader cannot assess whether ROMA's robustness gains are Pareto-efficient — i.e., whether they represent a better accuracy-per-FLOP tradeoff than GRPO with extended training or GRPO with a larger model.

Mitigation status. Not addressed. The paper frames ROMA as an optimization-dynamics improvement (Section 1: "ROMA modifies the RL optimization dynamics directly") and focuses entirely on sample efficiency (matched steps and data). The computational overhead is neither measured nor discussed as a tradeoff. Future work could profile ROMA's training cost and compare FLOPs-matched variants, but the current paper provides no basis for practitioners to make a cost-informed decision.

6.2 The Experiments Use a Single Model Family on a Single Training Dataset with No Statistical Replication

The assumption or constraint. All experiments use the Qwen3-VL 4B and 8B Instruct models (Bai et al., 2025) fine-tuned on the MMRL30k dataset (~30K samples). The paper states in Section 4.1 that it "conduct[s] direct RL training on the Qwen3-VL-4B and 8B Instruct models, using GRPO as the underlying RL algorithm" but does not justify why this model family is sufficient to establish general claims about "critic-free RL fine-tuning of autoregressive MLLMs" (Section 1). Each experimental configuration (Base, GRPO, ROMA, each ablation variant) is trained exactly once — there are no multi-seed runs, no standard deviations, and no confidence intervals reported anywhere in the paper.

The consequence. Two distinct concerns arise. First, model-family specificity: ROMA's regularization design makes assumptions about the optimization landscape of GRPO-fine-tuned MLLMs — that invariance penalties cause policy collapse without an auxiliary PG, that worst-case KL provides stronger robustness than mean-penalty KL, that correctness conditioning prevents systematic error amplification. Whether these assumptions hold for other MLLM architectures (e.g., LLaVA, InternVL, DeepSeek-VL) with different visual encoders, different tokenization strategies, and different pretraining distributions is unknown. The paper's title claims "Reinforcing Multimodal Reasoning Against Visual Degradation" — a general claim — but the evidence is restricted to one model family fine-tuned on one dataset.

Second, absence of statistical characterization: the headline robustness improvements (+2.4% seen, +2.3% unseen for 8B) are modest in absolute terms. RL training on 30K samples with 120 steps, 8 rollouts per input, and stochastic augmentation sampling has inherent run-to-run variance. The sensitivity analyses (Tables 5, 7, 8) show that changing β from 0.10 to 0.15 costs 4.5% on seen degradations, and changing K from 3 to 4 costs 0.3% — these are hyperparameter sensitivities comparable in magnitude to the claimed gains over GRPO. Without multi-seed results, the reader cannot distinguish a genuine methodological improvement from a favorable random seed. The +2.4% seen-degradation improvement could be within the 95% confidence interval of GRPO's run-to-run variance.

What evidence exists in the paper. The consistency of ROMA's gains across all seven benchmarks (Tables 1, 2) provides some informal reassurance — it is unlikely that random variance would produce consistent improvements across seven independent test sets. However, the benchmarks share a common training dataset and model checkpoint, so errors are correlated. The sensitivity analysis partially characterizes hyperparameter robustness but does not address stochastic training variance. The ablation studies (Tables 3, 4, 6) all use single-run results without variance estimates.

Mitigation status. Not addressed. The paper reports all results as point estimates to one decimal place, implying precision that the experimental design does not support. The authors do not discuss model-family or dataset generalization as a limitation, nor do they report multi-seed results or confidence intervals. Future work could replicate ROMA on other MLLM architectures and training datasets, but the current paper's claims are contingent on Qwen3-VL + MMRL30k.

6.3 Hard Degradations That Destroy Visual Information Are Not Addressed, and the Method Cannot Compensate for Fundamental Perceptual Information Loss

The assumption or constraint. ROMA enforces distributional consistency between clean-view and degraded-view token predictions — the KL penalty (Equation 2) pushes the model to assign similar token probabilities under degradation as under clean conditions. This assumes that the correct reasoning tokens remain identifiable under degradation — i.e., the model can still extract sufficient visual information to produce or recognize the correct token sequence, even if with reduced confidence. When visual degradation destroys information entirely — making text illegible, chart values unreadable, or visual details indistinguishable — distributional consistency becomes meaningless because there is no correct trajectory to be invariant to. The paper's evaluation at Level 3 severity (Appendix A.2) includes degradations deliberately more severe than training-time bounds, but the paper does not analyze at what severity threshold the method stops helping.

The consequence. On benchmarks where task performance depends on extracting fine-grained visual details — particularly ChartQA, which requires reading precise numerical values, axis labels, and legend entries — ROMA provides negligible improvement. Tables 11 and 12 show that on ChartQA under Gaussian blur (8B model), GRPO achieves 15.8% and ROMA achieves 16.9% (+1.1%). Under resolution downscaling, GRPO achieves 19.3% and ROMA achieves 20.4% (+1.1%). These gains are dwarfed by the overall performance collapse: ChartQA drops from 81.5% clean (GRPO) to ~16-20% under these specific perturbations. ROMA does not come close to recovering the lost performance because the information needed to answer correctly — the specific numbers and labels in the chart — has been rendered illegible by the perturbation. No amount of distributional invariance can recover an answer when the answer literally cannot be seen.

This is not a failure of ROMA per se — no method can extract information that isn't present — but it defines a sharp capability boundary that the paper does not explicitly characterize: ROMA helps when degradation reduces the model's confidence in correct reasoning (the model could answer correctly but is uncertain), but ROMA does not help when degradation destroys the perceptual evidence needed to answer at all. The gap between these two regimes — confidence reduction vs. information destruction — varies by task type and degradation severity, and the paper provides no framework for predicting when ROMA will be effective versus when degradations will overwhelm any regularization strategy.

What evidence exists in the paper. The detailed per-perturbation, per-benchmark breakdowns in Tables 10, 11, and 12 provide indirect evidence. ChartQA's near-complete collapse under Gaussian blur (14.6-16.9%) and resolution downscaling (18.2-20.4%) contrasts with MathVista, which retains 67.0-69.6% on the same perturbations. Pixelation — the most damaging unseen perturbation — causes similar collapse on ChartQA (5.4-7.4%) while WeMath retains 39.5-49.7%. The paper's main-text discussion (Section 4.2) notes that "GRPO suffers a larger performance drop when transitioning from clean to degraded inputs" but does not analyze the differential effectiveness of ROMA across information-loss regimes.

Mitigation status. The paper acknowledges implicitly that some benchmarks degrade more than others (the consistent pattern of ChartQA underperforming under perturbation is visible in all tables), but does not frame this as a capability boundary or provide analysis of when ROMA can vs. cannot help. The discussion text in Section 4.2 treats robustness gains as uniform: "our approach consistently outperforms GRPO across all benchmarks under degraded conditions." This is true in the narrow sense (ROMA > GRPO on every benchmark), but the practical significance varies enormously — a +1.1% gain on a benchmark where accuracy has collapsed to 16% is qualitatively different from a +2.3% gain where accuracy is at 77%. The paper does not address this distinction.

6.4 The "Unseen Degradation" Evaluation Tests Generalization Across Noise Structures, Not Across Fundamentally Different Categories of Visual Perturbation

The assumption or constraint. The paper evaluates out-of-distribution (OOD) generalization using five unseen corruption types: motion blur, salt-and-pepper noise, speckle noise, posterization, and pixelation (Section 4.1, Appendix A.2). These are described as "degradation types completely unseen during training" (Section 4.2). The paper claims ROMA "exhibits stronger zero-shot generalization to these unseen corruptions" and that "robustness acquired on seen degradations transfers effectively to unseen domains" (Section 4.2).

However, all unseen corruption types belong to the same broad category as the seen ones: pixel-space, image-level perturbations applied uniformly to 2D static images. Motion blur is a convolution with a directional kernel — structurally similar to Gaussian blur (seen) but with a different kernel shape. Salt-and-pepper noise is additive impulse noise — structurally similar to Gaussian noise (seen) but with a different noise distribution. Speckle noise is multiplicative noise — a variant of additive noise with different statistical properties. Posterization reduces color bit depth — a quantization effect similar in spirit to JPEG compression (seen). Pixelation downsamples and re-upsamples the image — a coarser version of resolution downscaling (seen).

These are OOD in parameterization and noise structure, but not in kind. They are all still pixel-space transformations of 2D images. A genuinely OOD evaluation would include perturbations that differ categorically: geometric transformations (rotation, perspective warp, affine shear), partial occlusions (random patches, superimposed text, watermark artifacts), lighting and color shifts (brightness/contrast changes, white balance shifts, hue rotation), or cross-domain shifts (natural photographs to document scans, synthetic renderings, line drawings). Real-world visual degradation in deployment settings — user-uploaded phone photos, compressed messaging app images, scanned documents, surveillance footage — often combines multiple of these categorical shifts simultaneously.

The consequence. The paper's claim of OOD generalization is substantially narrower than the term suggests. ROMA's invariance penalty encourages the model to produce consistent token distributions under different pixel-space perturbations — this is a valuable property, but it is perturbation-class-specific. If a practitioner deploys ROMA in a setting where visual degradation involves perspective distortion (e.g., photos of documents taken at an angle) or partial occlusion (e.g., a handwritten note partially covering printed text), there is no evidence that ROMA's regularization would transfer. The auxiliary PG loss (Equation 4) and worst-case KL penalty (Equation 3) were trained exclusively on four pixel-space corruption types; they have no inductive bias toward geometric or semantic invariance.

The empirical results are consistent with this concern. ROMA's OOD gain (+2.3% at Level 3, 8B) is nearly identical to its seen-degradation gain (+2.4%). If ROMA were learning genuinely general perceptual invariance that transcends perturbation category, one might expect the OOD gain to be larger relative to the seen gain (since GRPO has no exposure to seen perturbations either and would be equally brittle to both). The fact that the gains are similar suggests ROMA is learning corruption-class-specific invariance that transfers within the class (different noise distributions, different blur kernels) but does not confer protection against perturbations outside that class.

What evidence exists in the paper. The per-perturbation breakdowns (Tables 10, 11, 12) show that ROMA's gains on unseen perturbation types are generally similar in magnitude to gains on seen types. For the 8B model: +2.1% on Gaussian blur (seen) vs. +1.8% on motion blur (unseen); +2.1% on Gaussian noise (seen) vs. +1.8% on salt-and-pepper noise (unseen). This within-class transfer is consistent with the hypothesis that ROMA learns noise-structure-class-specific invariance rather than general perceptual robustness. The paper does not report results on geometric or semantic perturbations that would test cross-class generalization.

Mitigation status. The paper adopts the ImageNet-C framework's distinction between seen and unseen corruptions and treats this as sufficient for OOD evaluation. The authors do not acknowledge that all corruptions (both seen and unseen) share the same fundamental category. The conclusion states that ROMA "enables MLLMs to internalize robust logic, maintaining reasoning stability across diverse visual corruptions" — but the diversity is within the class of pixel-space perturbations. Future work could expand the degradation pool to include geometric, lighting, and occlusion perturbations to test cross-class generalization, but the current paper's OOD claims should be interpreted as applying to noise-structure variation, not to fundamentally different categories of visual degradation.

6.5 Correctness Conditioning Depends on a Binary Reward Signal That May Not Be Available or Well-Defined in Many Deployment Settings

The assumption or constraint. The correctness-conditioning mechanism — multiplying the KL penalty by I[R > 0] so invariance is enforced only for trajectories that achieved positive reward — requires a reliable binary reward signal that cleanly separates "correct" from "incorrect" reasoning trajectories. In the paper's experimental setting, this is straightforward: the training dataset (MMRL30k) consists of visual question-answering tasks where each question has a single correct answer, and the reward function R(v, x, y) checks exact or semantic match between the extracted final answer and the reference answer. Trajectories that reach the correct answer get positive reward; all others get zero or negative reward. The correctness mask I[R > 0] is therefore well-defined and noise-free.

This is a specific and narrow setting. Many real-world multimodal reasoning tasks lack clean binary correctness signals: open-ended visual description, summarization of visual content, multi-step planning from visual observations, visual dialogue, or creative generation from image prompts. In these settings, "correctness" may be multi-dimensional (fluency, relevance, factual accuracy, helpfulness), may require human judgment, or may be inherently ambiguous (multiple valid responses exist). A binary reward signal would either be unavailable or would introduce false negatives (marking valid but non-reference responses as incorrect) and false positives (rewarding plausible but factually wrong responses).

The consequence. If a practitioner deploys ROMA in a setting without clean binary rewards, the correctness mask becomes unreliable. Two failure modes arise. First, if the reward signal is noisy and occasionally marks correct trajectories as incorrect (false negatives), the KL penalty is suppressed on genuinely correct reasoning chains, reducing the effectiveness of invariance regularization. Second, if the reward signal occasionally marks incorrect trajectories as correct (false positives), the KL penalty is applied to incorrect reasoning chains, causing the "robustly incorrect" problem that correctness conditioning was designed to prevent — the model becomes invariant in producing systematic errors. The paper's ablation (Table 6) shows that unconditional penalty degrades performance by 2.2% on both seen and unseen degradations, indicating that false positives in the correctness mask would directly harm robustness.

The paper's experimental setting avoids this problem by construction — MMRL30k provides unambiguous correctness — but does not address how ROMA would function with learned reward models, human preference labels, or other noisy reward sources that are standard in RLHF-style training. The ablation on correctness conditioning (Table 6) demonstrates the importance of a clean mask but offers no guidance on how sensitive the method is to mask noise.

What evidence exists in the paper. None. The paper does not experiment with noisy or learned reward signals, does not analyze ROMA's sensitivity to reward noise, and does not discuss the requirement for clean binary rewards as a limitation. The ablation in Table 6 compares conditional vs. unconditional penalty — both using the ground-truth binary reward — but does not explore conditions where the reward is uncertain.

Mitigation status. Not addressed. The paper's problem formulation (Section 3) assumes a scalar reward function R(v, x, y) without qualifying its reliability. The training dataset (MMRL30k) provides reference answers, so the reward is effectively oracle-level. The authors do not discuss how ROMA would adapt to tasks without ground-truth answers or with continuous/learned reward models. Future work could explore soft correctness conditioning (weighting the KL penalty by reward confidence rather than a binary gate) or using the PRM's own score as a proxy for correctness, but the current paper provides no roadmap for extending ROMA beyond the oracle-reward setting.

6.6 The Method Introduces a Critical Hyperparameter Sensitivity: The Balance Between the Auxiliary PG and KL Penalty Is Fragile

The assumption or constraint. ROMA's total objective (Equation 5) combines three terms: the standard RL objective J_RL, the auxiliary policy gradient α · J_aug_pg, and the worst-case KL penalty −β · E[G_π^worst · I[R > 0]]. The paper's sensitivity analyses (Tables 5 and 8) reveal that both α and β exhibit inverse-U performance curves where moderate values are optimal but both under-regularization and over-regularization degrade robustness. The optimal values (α = 0.10, β = 0.10) were found via grid search over three values each and happen to land at the same coefficient for both the 4B and 8B models. The paper provides no principled method for setting these coefficients a priori — they are empirical tuning parameters that must be validated for each new model, dataset, and degradation protocol.

The consequence. The sensitivity curves reveal a fragility that practitioners must navigate. For β (Table 8), increasing from 0.10 to 0.15 reduces seen-degradation accuracy from 61.3% to 56.8% — a drop of 4.5 percentage points, which is nearly double the total robustness gain ROMA achieves over GRPO (+2.4%). This means that a practitioner who sets β too aggressively (believing "more invariance is better") would not only fail to improve robustness but would substantially degrade it relative to the optimal setting. The degradation is asymmetric: over-regularization (β = 0.15, −4.5% from optimal) hurts more than under-regularization (β = 0.05, −1.9% from optimal), creating a cliff where small hyperparameter errors in the aggressive direction cause large performance losses.

The α sensitivity (Table 5) shows a similar but less extreme pattern: increasing from 0.10 to 0.15 reduces seen-degradation accuracy from 61.6% to 60.0% (−1.6%). The combination of α and β sensitivity means that the joint hyperparameter space has a narrow optimal region, and finding it requires grid search over both coefficients simultaneously — the paper did not report joint sensitivity (e.g., α = 0.10, β = 0.15 vs. α = 0.15, β = 0.10), which could reveal interaction effects that make tuning even more challenging.

The paper's claim that both optimal values land at 0.10 for both 4B and 8B models suggests some stability across model scales, but this is based on a single model family fine-tuned on a single dataset with a single degradation protocol. A practitioner deploying ROMA on a different MLLM architecture, a different training dataset, or with different degradation types and severities would need to re-tune α and β from scratch, with no principled initialization beyond "try 0.10 first."

What evidence exists in the paper. Tables 5 and 8 provide the primary evidence — single-variable sweeps over α ∈ {0.05, 0.10, 0.15} and β ∈ {0.05, 0.10, 0.15} on the 8B model under both seen and unseen degradations at Level 3 severity. The paper does not report joint sensitivity (varying α and β simultaneously), does not report sensitivity at other severity levels, and does not report whether the optimal coefficients transfer across benchmarks individually (the tables report macro-averaged accuracy). The inverse-U shape is clear and consistent across both parameters and both degradation settings.

Mitigation status. The paper acknowledges the sensitivity implicitly by conducting and reporting the analyses, but does not frame hyperparameter fragility as a limitation. The authors select α = β = 0.10 as defaults based on their grid search and treat these as fixed hyperparameters for all experiments. The paper does not propose a principled selection method (e.g., adaptive scheduling of α and β based on training dynamics, or automated tuning via population-based training), nor does it discuss the practical burden of hyperparameter search when deploying ROMA in new settings. Future work could explore automated coefficient tuning or develop heuristics based on degradation severity or dataset characteristics, but the current paper treats α and β as empirically determined constants with no generalization guarantee.

7. Implications and Future Directions

How This Work Changes the Landscape

ROMA makes a methodological intervention rather than a paradigm shift: it reframes visual robustness for RL-fine-tuned MLLMs from a data problem (what images do we train on?) to an optimization-dynamics problem (what gradients do we apply, and when?). This is not the first paper to regularize policy optimization — DrAC and RAD established the principle of cross-view invariance in deep RL — but ROMA is the first to demonstrate that naïve porting of these techniques to critic-free, autoregressive MLLM fine-tuning fails for identifiable reasons, and that a specific combination of design choices (dual-forward-pass teacher-forcing, correctness-gated KL penalty, clean-advantage-anchored auxiliary PG, worst-case multi-view optimization) resolves those failures.

The practical significance of this reframing is that it changes what practitioners should build and tune. Before ROMA, the natural response to MLLM brittleness under visual degradation was to diversify the training data: add augmented images to the RL rollout phase (NoisyRollout, Liu et al., 2025), penalize the policy when it ignores visual input (PAPO, Wang et al., 2025), or collect more data with natural corruptions. ROMA's results suggest these data-centric strategies are insufficient and may be counterproductive — GRPO's clean-to-degraded accuracy gap (9.7 percentage points for 8B) is larger than the base model's gap (7.9 percentage points), meaning standard RL fine-tuning increases fragility even without seeing any degradations during training. The dual-forward-pass design (Section 3) is a direct response to this finding: it argues that generating trajectories on degraded images is harmful not because the model fails to learn from them, but because the reward signal itself becomes unreliable — a reward that punishes hallucinated trajectories on illegible inputs is a noisy, uninformative gradient for improving reasoning. The fact that ROMA achieves its robustness gains using the same 30K training samples as GRPO (no new data, no additional augmentation sources) while outperforming NoisyRollout-7B by approximately 6-7 percentage points on degraded inputs (Tables 1, 2) provides empirical support for the claim that how you regularize matters more than what data you regularize on.

The work also resolves a latent tension in the multimodal RL literature. Several concurrent efforts — NoisyRollout, PAPO, Vision-R1, VL-Rethinker — each propose different mechanisms for making MLLMs more robust or visually grounded during RL fine-tuning. These papers report results on different benchmarks with different base models, making it impossible to compare their effectiveness or understand whether they address the same underlying problem. ROMA's contribution is not that it outperforms these methods (the comparison is confounded by base model and training data differences), but that it provides a diagnostic framework for understanding why certain approaches succeed or fail. The paper identifies three specific failure modes — reward poisoning from augmented rollouts, policy collapse under invariance regularization in critic-free settings, and systematic-error amplification from unconditional invariance — and designs targeted interventions for each. Future work can use this diagnostic lens to evaluate other robustness methods: does a given approach avoid reward poisoning? Does it prevent collapse without a value network? Does it gate regularization on correctness? ROMA provides a checklist, not just a method.

The most impactful conceptual shift may be the correctness-conditioning insight — the idea that invariance should be enforced only for desirable behaviors, not all behaviors. This is a refinement of the invariance regularization principle that has been used in computer vision and deep RL for years, where it was assumed that matching the clean-output behavior, whatever it is, is always beneficial. ROMA's ablation (Table 6) demonstrates that this assumption is false for reasoning MLLMs: unconditionally enforcing invariance degrades performance by 2.2% on both seen and unseen degradations. The mechanism — that MLLM reasoning errors are systematic and coherent, so enforcing invariance on them creates "robustly incorrect" policies — is specific to the reasoning domain but has conceptual parallels in other settings where outputs can be coherently wrong (e.g., code generation producing consistent but buggy code, medical diagnosis producing consistent but incorrect differentials). The correctness mask (I[R > 0]) is a simple mechanism, but the principle it embodies — separating the question of "what to be invariant to" from "what behaviors should be made invariant" — may influence robustness research beyond multimodal RL.

Finally, ROMA reorients the research priority from search algorithms to verifier robustness (or, in the critic-free setting, to reward signal reliability). The paper's central design choice — avoiding reward poisoning by never generating from degraded images — is an implicit argument that the quality of the reward signal is the binding constraint on robustness in RL-fine-tuned MLLMs, not the sophistication of the policy optimization algorithm. This echoes the paper's finding in the search-vs-revisions analysis that verifier over-optimization is the primary bottleneck for test-time compute scaling. The implication is that investments in better reward models, more reliable answer extraction, and reward calibration will yield larger robustness gains than investments in more complex RL algorithms — a practical prioritization that research groups can act on immediately.

Follow-Up Research This Work Enables

Training a lightweight difficulty or "degradation severity" predictor to enable dynamic regularization scheduling. ROMA applies the same KL penalty weight β = 0.10 to all training samples regardless of how severely each specific image is degraded at each training step. The sensitivity analysis (Table 8) shows that the optimal β is fragile — over-regularization (β = 0.15) costs 4.5% on seen degradations, while under-regularization (β = 0.05) costs 1.9%. This suggests that a fixed β is a compromise: some training samples receive too little regularization (weak augmentations where the KL divergence is small) while others receive too much (strong augmentations where the KL divergence is large and the gradient from the penalty term can destabilize optimization). A natural extension is to train a small auxiliary model — perhaps the MLLM's own vision encoder with a regression head — that predicts, from the degraded image alone, the expected KL divergence between clean and degraded view logits. This severity estimate could then be used to dynamically scale β per sample: apply stronger regularization to mildly degraded images (where invariance is learnable and the reward signal remains reliable) and weaker regularization to severely degraded images (where the KL divergence is too large to be usefully minimized, and aggressive regularization would cause the gradient interference observed at β = 0.15). The experiment would measure whether dynamic β scheduling improves robustness at matched hyperparameter sensitivity, whether it reduces the performance cliff at high β, and whether it transfers across degradation types without re-tuning.

Combining ROMA's invariance regularization with on-policy data generation for self-improvement loops. The paper trains ROMA on the fixed MMRL30k dataset using off-policy trajectories (generated by the old policy at each step, then re-evaluated under degradation via teacher forcing). A powerful extension would embed ROMA's dual-forward-pass regularization into an iterative self-improvement loop: use the current ROMA-trained policy to generate clean-image trajectories on new, unlabeled visual questions, apply the correctness-conditioned invariance penalty to successful trajectories, use the resulting distributionally-stable policy to generate higher-quality training data, and repeat. This is analogous to the STaR/ReST^EM self-improvement paradigm (Zelikman et al., 2022; Singh et al., 2024) but with the critical addition that the policy remains robust to visual degradation throughout the iterative process — avoiding the common failure where self-improvement amplifies brittleness because the policy overfits to its own clean generation distribution. The paper's finding that the ReST^EM-trained revision model in the earlier case study (Appendix K) caused performance degradation with sequential revisions highlights the sensitivity of self-improvement to data generation quality; ROMA's regularization could stabilize this process by preventing the policy from drifting into fragile regions of parameter space. A concrete experiment: start with a base MLLM, apply one epoch of ROMA training on MMRL30k, use the resulting model to generate 50K new reasoning trajectories on a larger unlabeled visual QA dataset (e.g., natural images from VQA v2 with templated questions), filter for correct trajectories using a learned verifier or majority voting, apply ROMA's correctness-conditioned regularization to those trajectories, fine-tune for another epoch, and measure both clean accuracy and degraded accuracy (seen and unseen) across iterations. The hypothesis is that ROMA-regularized self-improvement maintains or improves robustness over iterations, while standard GRPO self-improvement becomes increasingly brittle.

Stress-testing ROMA's OOD generalization with geometrically and semantically distinct perturbations. The paper's unseen degradation pool (motion blur, salt-and-pepper noise, speckle noise, posterization, pixelation) tests generalization across noise structures within the class of pixel-space 2D image perturbations. This leaves open the question of whether ROMA's invariance penalty generalizes to categorically different perturbations: geometric transformations (rotation by ±15°, perspective warp simulating off-angle photography, affine shear), partial occlusions (random rectangular patches covering 10-30% of the image, superimposed text in varying fonts and positions, emoji or sticker overlays), lighting and color shifts (brightness ±30%, contrast ±30%, hue rotation ±30°), and cross-domain rendering shifts (photograph → edge map via Canny detection, photograph → grayscale, photograph → pencil sketch via neural style transfer). A strong follow-up would evaluate ROMA's 8B checkpoint (trained only on Gaussian noise, Gaussian blur, JPEG compression, and resolution downscaling) directly on each of these perturbation categories at severity levels comparable to the paper's Level 1-3, without any additional fine-tuning. If ROMA's robustness transfers (accuracy drop from clean to perturbed is smaller than GRPO's), this would indicate that the invariance penalty induces genuinely general perceptual stability. If it does not transfer (ROMA and GRPO show similar degradation), this would refine our understanding of ROMA's mechanism — it learns noise-class-specific invariance, not a general smoothness prior, and deploying it in settings with different visual degradation categories would require training on representative perturbations. This experiment is low-cost (evaluation only, no training) and would substantially clarify the scope of ROMA's robustness claims.

Investigating the interaction between ROMA and the base model's visual encoder architecture. ROMA applies its regularization at the token-output level of the autoregressive MLLM — the KL penalty compares the final logit distributions after the full visual encoding and text decoding pipeline. This raises a question about where in the architecture invariance is being learned. Does the regularization push the visual encoder (e.g., Qwen3-VL's vision transformer) to produce more degradation-invariant image features, or does it push the language decoder to be more robust to feature perturbations without changing the encoder representations? This distinction matters practically: if the encoder learns invariance, ROMA's benefits would transfer to downstream tasks that use the same visual encoder (e.g., if the fine-tuned model is used as a visual feature extractor for other tasks). If the decoder learns robustness, the invariance is task-specific and may not transfer. A diagnostic experiment: take the visual encoder from a ROMA-trained model and a GRPO-trained model, freeze the encoder weights, attach a linear probe, and train on a held-out visual classification task (e.g., classifying the object in the image, or answering a simplified visual question that doesn't require reasoning). Measure probe accuracy on clean vs. degraded images for both encoders. If the ROMA-trained encoder probe shows smaller clean-to-degraded accuracy gaps, invariance is being learned in the encoder. If the gaps are similar, the decoder is doing the heavy lifting, suggesting that ROMA's robustness is reasoning-specific and that practitioners should not expect visual representations from ROMA-trained models to be generally more robust.

Developing a reward-free variant of ROMA using the PRM's own confidence as a correctness proxy. ROMA's correctness-conditioning mechanism requires a binary reward signal (I[R > 0]) to gate the KL penalty, which limits the method to tasks with ground-truth answers or reliable learned reward models (see Limitation 6.5). A natural extension is to replace the ground-truth reward with an internal consistency metric derived from the model's own outputs under different degradation views — for instance, the agreement between the final answers produced via teacher-forcing on two different degraded views, or the PRM's predicted probability of correctness under the clean view. If the model produces the same answer under two independent degraded views (evaluated via teacher forcing on the same clean trajectory), this agreement can serve as a pseudo-reward: high agreement suggests the reasoning chain is perceptually stable and likely correct; low agreement suggests the chain is fragile and should not receive invariance regularization. A concrete experiment: train ROMA with the correctness mask I[R > 0] replaced by an agreement score (fraction of K = 3 degraded views on which the teacher-forced final answer token matches the clean-image answer token, thresholded at 2/3). Compare robustness (seen and unseen degradations) against the ground-truth-reward ROMA baseline. If agreement-based gating approaches the performance of ground-truth gating, ROMA becomes deployable in settings without oracle rewards — a significant expansion of applicability. If it substantially underperforms, it reveals that the model's own perceptual consistency is a weak proxy for correctness, which would itself be an important negative result clarifying the limits of self-supervised robustness.

Measuring whether ROMA reduces hallucination rates under degradation, not just accuracy. The paper evaluates ROMA exclusively on end-to-end answer accuracy — whether the extracted final answer matches the reference. This metric conflates two distinct effects: improved reasoning fidelity (the model produces the correct reasoning chain) and improved answer extraction (the model produces a parsable final answer even if the reasoning is noisy). A more granular analysis would track whether ROMA reduces the rate of specific failure modes under degradation: hallucinated visual content (the model references objects, text, or numbers not present in the degraded image), premature answer termination (the model outputs an answer without completing the reasoning chain), format violations (the model fails to enclose the answer in \boxed{}), and self-contradiction (the reasoning chain contradicts the final answer). A follow-up study could annotate 200-500 degraded-image trajectories from GRPO and ROMA on a subset of benchmarks (MathVista and ChartQA would provide the most contrast) for these failure modes. The hypothesis is that ROMA reduces hallucination and premature termination (because the KL penalty prevents the model from diverging into hallucinated trajectories when the image is degraded) but may not affect format violations or self-contradiction (which are reasoning-structure issues independent of perceptual invariance). Quantifying these effects would provide a richer picture of what "robustness" means for ROMA and would guide practitioners on which deployment failure modes ROMA can and cannot address.

Practical Applications and Downstream Use Cases

On-device or edge-deployed MLLMs processing user-captured images in uncontrolled conditions. A mobile application that uses an MLLM to answer questions about user-uploaded photos — e.g., a homework helper that solves math problems from phone photos of textbook pages, or a plant identification app that reasons about visual features — faces highly variable image quality. Users submit blurry photos taken in low light, compressed screenshots from messaging apps, or low-resolution crops. ROMA's 8B model maintains 61.6% accuracy under severe seen degradations vs. GRPO's 59.2%, and the clean-to-degraded accuracy gap shrinks from 9.7 percentage points (GRPO) to 7.1 percentage points (ROMA). For a deployment serving millions of queries, a 2.4 percentage point improvement in degraded-input accuracy translates to tens of thousands of additional correctly answered queries without requiring users to retake photos or improving image preprocessing. The key practical benefit is not just higher average accuracy but reduced variance across input quality — ROMA's shallower clean-to-degraded accuracy drop means the model's performance degrades more gracefully as image quality declines, providing a more predictable user experience.

Batch inference pipelines for document processing and digitization. Organizations processing large volumes of scanned documents — legal firms digitizing case files, libraries archiving historical manuscripts, healthcare systems processing patient intake forms — often use MLLMs to extract structured information (names, dates, diagnoses, answers to specific questions) from document images. These images frequently suffer from scanning artifacts: blur from page curvature, compression from legacy storage formats, low resolution from microfilm scans, and noise from aging physical media. ROMA's per-perturbation breakdowns (Tables 10-12) show that resolution downscaling and Gaussian blur — the most common document scanning artifacts — cause the largest accuracy drops on detail-oriented benchmarks like ChartQA (81.5% clean → 15.8% degraded for GRPO, improving to 16.9% with ROMA). While the absolute improvement is modest on severely degraded documents (where text becomes illegible), ROMA's gains on medium-severity degradation (Level 2 in Figure 2, where text is degraded but still readable) suggest that document processing pipelines operating on "mostly legible but noisy" scans — the most common case — would see meaningful throughput improvements. Rather than flagging all noisy documents for manual review, a ROMA-trained model could process a larger fraction automatically, with the confidence that its reasoning remains stable under moderate degradation.

Training-data generation for self-improving MLLMs in visually noisy domains. A research group developing an MLLM for scientific figure understanding or medical image reasoning needs to generate high-quality training data by having a strong model produce reasoning trajectories on unlabeled images, then fine-tuning a smaller model on those trajectories. If the strong model is brittle to visual degradation — as GRPO is, with a 9.7 percentage point clean-to-degraded drop — the generated training data will be contaminated by hallucinated or incorrect trajectories whenever the input images are less than perfect (which is common in scientific PDFs with compressed figures or medical images with acquisition artifacts). ROMA's smaller clean-to-degraded gap (7.1 percentage points at 8B) means the generated training data is more reliable across varying input quality, leading to better downstream model performance. Additionally, the correctness-conditioning mechanism provides a natural quality filter: during data generation, only trajectories where R > 0 (reward-positive, i.e., correct) receive invariance regularization, meaning the model produces a natural train/test split — reasoning chains that are both correct and perceptually invariant. These trajectories are high-quality candidates for distillation into a smaller, deployment-efficient student model.