Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation

ArXiv: 2604.24763

🎯 Pitch

Ditching the pretrained vision encoder entirely—replacing CLIP and VAEs with raw pixel patches—actually improves multimodal understanding at scale if you train long enough. The catch is that the encoder-free model starts slower, revealing a hidden “early-bootstrap” advantage that standard vision backbones provide but that end-to-end pixel learning eventually overcomes.

---

1. Executive Summary

This paper introduces Tuna-2, a native unified multimodal model that performs visual understanding and generation directly in pixel space by progressively stripping away pretrained vision encoders—first removing the VAE to yield Tuna-R (using only a representation encoder), then eliminating the encoder entirely to operate from raw pixel embeddings via simple patch embedding layers. Evaluated against the MATH of multimodal benchmarks (GQA, RealWorldQA, MMVet, MMMU, GenEval, DPG-Bench, and ImgEdit) using a Qwen2.5-7B-Instruct backbone, Tuna-2 achieves state-of-the-art results among 7B-scale native UMMs on understanding tasks while remaining competitive on generation, and introduces a masking-based visual feature learning scheme (replacing a random subset of image patches with learnable mask tokens during pretraining to regularize both understanding and generation objectives) that stabilizes pixel-space training. In controlled comparisons, Tuna-2's encoder-free design eventually surpasses Tuna-R on understanding benchmarks after sufficient pretraining, establishing that pretrained vision encoders become unnecessary for multimodal modeling at scale—but only when the model is given enough data to overcome the early training advantage conferred by encoder-based semantic priors.

2. Context and Motivation

The Problem: Vision Encoders Are the Default, But We Don't Know If They're Necessary

At the heart of this paper is a deceptively simple question: do we actually need pretrained vision encoders to build effective multimodal models that both understand images and generate them? The field has converged on an architectural consensus that assumes the answer is yes—nearly every major multimodal model chains together a pretrained vision encoder (like CLIP or SigLIP) with a language model decoder, using the encoder's rich semantic features as the bridge between pixels and text. But this consensus emerged from historical path dependence rather than from systematic elimination: vision encoders were built first, they worked well for understanding, so they became the foundation. No one had rigorously tested whether they remain necessary when the entire system is trained end-to-end at scale.

The gap this paper addresses is therefore the lack of empirical evidence on whether pretrained vision encoders are truly essential for unified multimodal modeling, or whether they are merely convenient scaffolding that can be discarded once training reaches sufficient scale. The authors frame this as a "natural but largely unexplored question" (Section 1): can we move beyond pretrained vision encoders altogether and build native unified models through end-to-end training directly from raw pixels?

Why This Matters: Practical, Theoretical, and Architectural Implications

This question matters for several interconnected reasons that go beyond mere architectural aesthetics.

Architectural complexity and fragility. Current unified multimodal models (UMMs) are modular assemblies of independently pretrained components: a VAE for compressing images into a latent space, a representation encoder (like CLIP or SigLIP) for extracting semantic features, a connector layer to translate between encoder and decoder representations, and an LLM decoder for joint vision-language processing. Each component introduces its own design constraints—fixed input resolutions, incompatible feature spaces, separate training objectives—that create friction when the system is trained end-to-end. The authors explicitly note that these modular designs prevent "fully end-to-end optimization from raw pixels" (abstract), meaning the system cannot learn visual representations optimized jointly for the tasks it actually performs. This creates a form of representation mismatch: the VAE is optimized for reconstruction, the representation encoder for semantic alignment, and neither is necessarily optimal for the downstream combination of understanding and generation that the unified model must perform.

Task interference and representation fragmentation. The problem is particularly acute in unified models because understanding and generation impose competing demands on visual representations. Understanding requires high-level semantic features that capture what objects are present, their relationships, and how they relate to language. Generation requires fine-grained, low-level spatial details—texture, edges, precise object boundaries—that a VAE's latent space preserves but a CLIP-style semantic encoder typically discards (pooling away spatial resolution in favor of global semantics). When a model uses separate encoders for these tasks (the "decoupled" approach), the representations live in different spaces entirely—the CLIP feature space and the VAE latent space have fundamentally different geometry, dimensionality, and information content. Even when models move to a "unified" representation (single encoder for both), they still inherit the inductive biases of whichever pretrained encoder they use, which was optimized for a single objective (typically contrastive image-text matching) that may conflict with generation fidelity.

The tantalizing success of encoder-free approaches in adjacent fields. The paper's motivation draws on two parallel developments that independently suggest encoders might be unnecessary. In multimodal understanding, recent models like Mono-InternVL (Luo et al., 2025) and NEO (Diao et al., 2025) have demonstrated that removing the pretrained representation encoder and processing raw pixel patches directly through the LLM decoder can match or exceed encoder-based models. This is a radical departure: instead of using a heavily engineered vision backbone pretrained on billions of image-text pairs, these models learn visual processing from scratch within a single transformer. In visual generation, pixel-space models like PixelFlow (Chen et al., 2025b) and JiT (Li and He, 2025) have shown that bypassing VAE latent spaces—once considered essential for generation quality—and operating directly on pixels can produce state-of-the-art results. The authors note (Section 1) that these pixel-space generation models have demonstrated "increasing flexibility, stronger scalability, and state-of-the-art performance on a wide range of tasks, suggesting that pretrained VAE encoders may no longer be essential even for high-fidelity image synthesis."

However, no prior work had combined these insights into a unified model: an encoder-free architecture that handles both understanding and generation in pixel space. The separate success of encoder-free understanding and pixel-space generation created a compelling hypothesis that both components could be unified, but this remained untested.

Scale as the great equalizer. Underlying the paper's motivation is a hypothesis about scaling: pretrained encoders provide a head start by injecting rich semantic knowledge acquired from massive datasets, but their advantage might diminish as joint training proceeds because the monolithic model can learn visual representations optimized specifically for its downstream tasks. The encoder's inductive biases—which help at small scale by providing structure—might become constraints at large scale, limiting what the model can learn from end-to-end optimization. If true, this would be a profound finding: it would mean the field's investment in ever-larger vision encoders (SigLIP at 400M parameters, InternVL encoders at 6B+) is misallocated for unified modeling, and that compute should instead be directed toward joint pretraining of simpler architectures.

Prior Approaches and Where They Fall Short

The paper identifies a clear taxonomy of existing approaches, each with specific limitations that Tuna-2's design addresses.

Decoupled UMMs: separate encoders, separate spaces. Early unified models like BAGEL (Deng et al., 2025) and Janus-Pro (Chen et al., 2025c) use fundamentally different representations for understanding and generation. The understanding pathway uses a representation encoder (e.g., CLIP, SigLIP) that produces semantic features optimized for text-image alignment. The generation pathway uses a VAE encoder that produces compressed latent codes optimized for reconstruction fidelity. These representations have different dimensionalities, different semantic content, and different scaling properties—the model must either maintain two separate processing streams or introduce adapter layers to translate between them. The result is a system where the understanding and generation capabilities are effectively siloed: the model sees different "views" of the same image depending on which task it's performing, preventing the kind of shared visual understanding that a truly unified model would develop.

The key failure mode is representation mismatch (Section 1): when the model generates an image, it cannot directly use the semantic understanding it developed for answering questions about images, because those semantic features exist in a different representational space than the latent codes the generation head expects. This prevents cross-task transfer—learning to recognize a cat better through the understanding objective doesn't directly improve the model's ability to generate cats, because the two tasks operate on incompatible representations.

Unified representation UMMs: shared encoder, inherited constraints. More recent models like Show-o2 (Xie et al., 2025b) and Tuna (Liu et al., 2025) address the decoupling problem by using a single shared vision encoder for both tasks. This is a significant improvement: the model now develops one unified representation that supports both understanding and generation, enabling direct cross-task transfer. Tuna, the immediate predecessor to this work, uses a representation encoder (SigLIP 2) combined with a VAE to produce unified tokens in a shared latent space.

However, these models still inherit the limitations of their pretrained encoders. The paper specifically highlights several constraints (Section 2.1):

1. Fixed input resolutions. Pretrained encoders are trained at specific resolutions (e.g., SigLIP 2 So400M at 384×384). While techniques like dynamic resolution adaptation exist, they are band-aids on a fundamentally constrained architecture. The encoder's positional embeddings and attention patterns are optimized for a particular spatial grid, making it difficult to handle arbitrary aspect ratios or very high-resolution images natively.

2. Loss of fine-grained low-level details. Representation encoders are designed and trained for semantic tasks—matching images to text descriptions—where low-level visual features (fine textures, precise edges, small text, subtle color gradations) are often irrelevant or even harmful (they add nuisance variance). The training objective (contrastive learning) explicitly encourages invariance to these details. But for generation and fine-grained visual reasoning, these details are essential. The paper notes (Section 3.2) that pixel-centric benchmarks like V*, CountBench, and VisuLogic—which require reasoning about very small objects, counting instances, or detecting subtle visual patterns—are precisely where latent-space UMMs struggle, because the pretrained encoder has already discarded the information needed to answer these questions.

3. Inductive biases that may conflict with generation. The contrastive pretraining objective shapes the encoder's feature space in ways that optimize text-image alignment but may be suboptimal for reconstruction and generation. Features that help distinguish between categories (sharp decision boundaries) may not be the features that help reconstruct an image faithfully (smooth, invertible transformations).

4. Training pipeline complexity. Modular architectures typically require multi-stage training: first pretraining the vision encoder (often on separate data), then training connector layers to align encoder outputs with the LLM's input space, then joint fine-tuning. Each stage introduces engineering complexity and potential points of failure, and the staged training means the system never truly performs end-to-end optimization from pixels to outputs.

Encoder-free understanding models: single-modality success, no generation. Mono-InternVL and NEO demonstrated that encoder-free architectures work for understanding, but these are understanding-only models—they don't attempt generation. The jump to unified understanding and generation introduces substantial new challenges because the model must learn visual representations that simultaneously support high-level semantic reasoning and low-level reconstruction fidelity, without the compartmentalization that modular encoders provide.

Pixel-space generation models: generation-only, often small-scale. Models like PixelFlow and JiT demonstrated high-quality pixel-space generation, but primarily in generation-only settings, often at limited scales (the paper notes these studies are "usually limited to small-scale settings, e.g., class-conditioned generation on ImageNet," Section 4.2). Scaling pixel-space flow matching to the demands of free-form text-to-image generation with natural language conditioning, while simultaneously supporting multimodal understanding, represents a substantial increase in difficulty that hadn't been attempted.

The missing middle: no encoder-free, pixel-space, unified model. The critical gap—and the precise niche this paper fills—is that no prior work had combined all three properties: (1) encoder-free architecture, (2) pixel-space operation (no VAE), and (3) unified understanding and generation. Each simplification had been demonstrated independently (encoder-free for understanding, pixel-space for generation), but combining them is more than the sum of its parts because the two design choices interact. Removing the VAE makes generation harder (operating in high-dimensional pixel space rather than compact latent space) but removes a source of information loss for understanding. Removing the representation encoder makes understanding harder (no pretrained semantic features) but removes constraints on resolution and detail preservation. A unified model without either encoder faces both challenges simultaneously—it must learn semantic representations from scratch while also learning to reconstruct pixel-level detail, all within a single transformer.

How This Paper Positions Itself

Tuna-2 is explicitly framed as a systematic architectural elimination study, not merely a new model proposal. The title's emphasis on "pixel embeddings beat vision encoders" signals that the central contribution is the empirical finding that encoder-free designs are not just viable but potentially superior, rather than a novel architectural innovation per se.

Progressive simplification as experimental design. Rather than jumping directly to the fully encoder-free design, the paper introduces two intermediate architectures that enable controlled comparison (Section 2.1):

• Tuna (the predecessor): uses both VAE and representation encoder—the standard design.
• Tuna-R: removes the VAE but keeps the representation encoder—isolating the effect of latent-space compression on generation quality and fine-grained understanding.
• Tuna-2: removes both encoders entirely—isolating the effect of pretrained semantic features on learning dynamics and final performance.

This A/B/C comparison is methodologically important because it lets the paper distinguish between two separate effects that are often conflated: the VAE's impact on generation quality and detail preservation, and the representation encoder's impact on learning speed and semantic understanding. By comparing Tuna and Tuna-R, the paper shows that removing the VAE improves fine-grained visual understanding (supporting the hypothesis that VAE compression discards useful details). By comparing Tuna-R and Tuna-2, the paper shows that removing the representation encoder eventually improves understanding performance despite an initial learning speed disadvantage, supporting the hypothesis that encoder priors are helpful early but constraining late. This controlled comparison is the paper's primary analytical contribution, providing evidence for why encoder-free designs work, not just that they work.

Pixel-space generation as a deliberate choice, not a compromise. The paper positions pixel-space flow matching not as a regression from more sophisticated latent diffusion approaches, but as an enabling design choice that unlocks end-to-end training and fine-grained visual perception. The authors explicitly argue that the "increased redundancy in pixel-space inputs" that makes learning harder can be addressed through their masking-based feature learning scheme (Section 2.2), which forces the model to develop robust internal representations by training it to reason under partial visual observation and to reconstruct images from masked inputs. This reframes pixel-space operation from a liability to an opportunity: the difficulty of pixel-space learning acts as a form of implicit regularization that produces better representations.

Scale as the critical variable. The paper's positioning around scale is nuanced. It does not claim that encoder-free designs are universally better—rather, it argues that they are better at scale. The finding that "Tuna-R actually outperforms Tuna-2 during the early stage of training" but Tuna-2 catches up and surpasses Tuna-R on understanding (Section 3.5, Figure 6) is central to this argument. The encoder provides a valuable head start by injecting semantic structure, but this advantage is "rented" rather than "owned": the encoder-free model's learned representations eventually become richer because they are optimized end-to-end for the tasks the model actually performs, without the constraints imposed by the encoder's pretraining objective. This positions the paper within the broader discourse on the relationship between architecture, scale, and data in deep learning: as models and datasets grow, simple architectures with weak inductive biases tend to catch up to and surpass more sophisticated architectures with strong priors, a pattern observed in language modeling (the shift from LSTMs with attention to pure transformers) and now, this paper argues, in multimodal modeling.

Connection to the "end-to-end" philosophy. At a conceptual level, the paper aligns with a long-standing tradition in deep learning that favors end-to-end learning over hand-designed intermediate representations. The message is that pretrained vision encoders, like hand-crafted features before them, are useful crutches that become unnecessary—and potentially limiting—when sufficient data and compute enable learning from raw sensory input. The paper's title—"Pixel Embeddings Beat Vision Encoders"—is deliberately provocative in this framing, suggesting that what the field has treated as an essential component of multimodal models is actually a form of premature optimization that restricts the model's ability to develop truly integrated visual representations.

3. Technical Approach

3.1 Reader orientation

Tuna-2 is a single transformer-based neural network that takes raw image pixels and text as input and directly outputs either text answers (for visual understanding tasks) or new images (for generation and editing tasks), without using any pretrained vision encoders as intermediate components. The core problem it solves is that existing unified multimodal models rely on modular, separately-pretrained vision encoders (VAEs and representation encoders like CLIP/SigLIP) that introduce representation mismatches between understanding and generation tasks, impose fixed architectural constraints like input resolution, and prevent fully end-to-end optimization from pixels—Tuna-2's solution is to progressively strip away these encoders (first the VAE, then the representation encoder) and replace them with simple learnable patch embedding layers that convert raw images into token sequences, combined with a pixel-space flow matching objective for generation and a masking-based regularization scheme that forces the model to learn robust unified visual representations directly from high-dimensional pixel inputs.

3.2 Big-picture architecture

The system has four major components arranged in a single end-to-end pipeline:

1. Patchify Layer — a simple embedding layer that divides raw input images into fixed-size patches (e.g., 16×16 pixel grids), linearly projects each patch into a token vector, and produces a sequence of visual tokens. For masked training, some patches are replaced with a learnable mask token embedding before entering the transformer. This is the only visual preprocessing; there is no VAE, no representation encoder, no connector layer.

2. Unified Vision-Language Transformer Decoder — a single autoregressive transformer (Qwen2.5-7B-Instruct) that processes the concatenated sequence of visual tokens and text tokens jointly, producing hidden states that encode unified multimodal representations. This single transformer handles all modalities and all tasks.

3. Language Modeling Head — a linear projection from the transformer's output hidden states to vocabulary logits, used for autoregressive text generation (answers to questions, captions, etc.). This is the standard LLM output head.

4. Flow Matching Head — a separate output head that takes the transformer's hidden states corresponding to image token positions and predicts a clean image in pixel space (the x-prediction head). During inference, this head is used iteratively with an Euler ODE solver to denoise random noise into a generated image.

Information flows as follows: raw image → patchify into token sequence → (optionally apply masking) → concatenate with text tokens → process through unified transformer → route to either language head (for text output) or flow matching head (for image output), depending on the task.

Tuna-R, the intermediate variant, inserts a pretrained representation encoder (SigLIP 2 So400M) between the image and the transformer—the encoder first processes the image into semantic visual tokens, which are then fed (via a trained connector layer) into the transformer. Tuna-2 removes this encoder entirely, feeding raw patch embeddings directly.

3.3 Roadmap for the deep dive

• First, the progressive architectural simplification (Tuna → Tuna-R → Tuna-2), explaining exactly what each variant removes, what replaces it, and why each step isolates a specific hypothesis about what vision encoders contribute.
• Second, the pixel-space flow matching formulation, because it is the generation mechanism that makes VAE-free operation possible, and its design choices (x-prediction, velocity parameterization, v-loss) directly affect the optimization landscape.
• Third, the masking-based feature learning scheme, which is the regularization technique that stabilizes pixel-space training and is applied identically to both understanding and generation examples but plays different functional roles in each.
• Fourth, the training pipeline, covering the two-stage end-to-end training process (pretraining then SFT), the data mixture ratios, the specific hyperparameters, and how Tuna-R's additional connector-alignment stage differs from Tuna-2's fully end-to-end approach.

3.4 Detailed, sentence-based technical breakdown

This is primarily an architectural analysis and simplification paper whose core idea is that pretrained vision encoders—both VAEs for latent-space compression and representation encoders for semantic feature extraction—are not necessary for building high-performing unified multimodal models, and that removing them enables fully end-to-end pixel-space learning that eventually produces stronger visual representations, particularly for fine-grained perception tasks, provided the model is trained at sufficient scale with appropriate regularization.

---

Progressive Architectural Simplification: Tuna → Tuna-R → Tuna-2

The paper's central experimental design is a three-way comparison where each model variant removes one additional encoder component, enabling the authors to isolate the effect of each encoder type on the model's behavior. The starting point is Tuna (Liu et al., 2025), which represents the standard design for native unified multimodal models with unified visual representations.

Tuna (the baseline): Tuna uses both a VAE encoder and a representation encoder. The VAE compresses input images into a compact latent space (typically 1/8 or 1/16 the spatial resolution of the original image with a channel depth determined by the VAE architecture) through an encoder-decoder framework trained with a reconstruction objective plus KL regularization. The representation encoder (SigLIP 2 So400M in this case) processes either the original image or the VAE latent representations and produces semantic feature tokens. These tokens are then combined with text tokens and fed through the LLM decoder for joint vision-language processing. For generation, the model predicts VAE latent codes which are then decoded back to pixel space by the VAE decoder. This architecture inherits all the constraints discussed in the motivation: the VAE's compression discards fine-grained details, the representation encoder's contrastive pretraining shapes the feature space toward semantic alignment rather than reconstruction fidelity, and the two encoders introduce separate processing streams that prevent truly unified representation learning.

Tuna-R (remove the VAE, keep the representation encoder): Tuna-R eliminates the VAE entirely. The representation encoder (still SigLIP 2 So400M) now processes raw pixel inputs directly—not VAE latents—and produces semantic feature tokens. These tokens pass through a trainable connector layer (a small MLP or linear projection) that maps from the encoder's output dimension to the LLM decoder's input dimension, then are concatenated with text tokens and processed by the transformer. For generation, rather than predicting VAE latents that require a separate decoder, Tuna-R uses pixel-space flow matching: the flow matching head (attached to the transformer's output) directly predicts clean images in raw pixel space, bypassing any latent-space compression.

What this removal isolates: by comparing Tuna and Tuna-R, the paper tests whether VAE-based latent compression is necessary for high-quality generation. If Tuna-R's pixel-space generation matches or approaches Tuna's latent-space generation quality, it demonstrates that VAE encoders are not essential for generation—a finding that contradicts the prevailing latent diffusion paradigm. Simultaneously, by operating on raw pixels before the representation encoder (rather than on VAE latents), Tuna-R gives the encoder access to the full pixel-level detail that the VAE would have compressed away, potentially improving fine-grained visual understanding. The connector layer in Tuna-R requires a separate short alignment training stage (3,000 steps at learning rate 5×10⁻⁴ using only image captioning and text-to-image generation data, trained with AdamW) before the main pretraining, which is an additional engineering step that Tuna-2 avoids.

Tuna-2 (remove both encoders, use raw patch embeddings): Tuna-2 eliminates both the VAE and the representation encoder. The only visual processing is a simple patch embedding layer: the input image is divided into a grid of non-overlapping patches (size 16×16 pixels), each patch is flattened into a vector of pixel values (16×16×3 = 768 values for RGB images) and linearly projected to the transformer's hidden dimension. These projected patch embeddings become visual tokens that are directly concatenated with text token embeddings and fed into the unified transformer decoder. There is no separate vision encoder, no connector layer, no pretrained visual features—the transformer must learn all visual processing from scratch, jointly with language understanding.

What this removal isolates: by comparing Tuna-R and Tuna-2, the paper tests whether pretrained semantic features from a representation encoder are necessary or beneficial for unified multimodal learning. Tuna-R gets a head start from SigLIP 2's semantic knowledge (acquired from pretraining on massive image-text datasets with a contrastive objective), while Tuna-2 must learn visual semantics from scratch through the joint multimodal training objective. The hypothesis is that Tuna-R's encoder-provided features help early in training (faster convergence) but may constrain late-training performance because the encoder's fixed inductive biases—its trained feature extractors, its resolution constraints, its semantic invariance to low-level details—limit what the model can learn through end-to-end optimization on the actual tasks.

Design choices and justifications in the simplification:

1. Why remove the VAE first, then the representation encoder? The two-stage simplification (rather than jumping directly to encoder-free) is methodologically critical because removing the VAE and removing the representation encoder test different hypotheses. Removing the VAE tests whether latent compression is necessary for generation quality—a claim that the latent diffusion literature takes as given. Removing the representation encoder tests whether pretrained semantic features help or hurt unified representation learning—a separate claim about the value of transfer learning versus end-to-end learning. If the paper had only compared Tuna and Tuna-2, any performance differences could be attributed to either encoder type, confounding the analysis. The intermediate Tuna-R variant lets the paper attribute specific effects to specific components.

2. Why SigLIP 2 So400M specifically for Tuna-R? The paper follows Tuna's choice and describes SigLIP 2 as a representation encoder with "improved semantic understanding, localization, and dense features" (Tschannen et al., 2025). SigLIP 2 is trained with a sigmoid-based contrastive loss rather than softmax-based contrastive loss (like CLIP), and it incorporates a masked prediction objective similar to the paper's proposed masking scheme, which makes it a particularly strong baseline—if an encoder-free design can match or exceed a model using this encoder, it is a strong statement about encoder necessity.

3. Why a patch size of 16? The paper states this in the experiment setup (Section 3.1) without extended justification, but the choice balances several constraints. Smaller patches (e.g., 8×8) would produce 4× more tokens per image (quadratically more), dramatically increasing the sequence length and computational cost of the transformer's self-attention (which scales quadratically in sequence length). Larger patches (e.g., 32×32) would reduce token count but lose spatial resolution, making it harder for the model to capture fine-grained details. The 16×16 patch size is a standard choice in vision transformers (dating back to ViT, Dosovitskiy et al., 2020) that provides a reasonable trade-off between spatial granularity and computational efficiency. For a 512×512 input image, a 16×16 patch size produces 32×32 = 1024 visual tokens, which is manageable within the 16K-token sequence length used in training.

4. Why Qwen2.5-7B-Instruct as the LLM decoder? The paper selects this model as a "representative" 7B-scale language model with strong instruction-following capabilities from pretraining. The choice of an instruction-tuned model (rather than a base model) matters because the SFT stage builds on the model's existing ability to follow instructions, which is crucial for tasks like image editing where the model must interpret complex editing commands ("replace the background with a beach scene"). The 7B scale also enables direct comparison with other 7B-scale UMMs in Tables 1 and 2.

---

Pixel-Space Flow Matching for Image Generation

The VAE-free design removes the ability to operate in a compressed latent space for generation, meaning the model must generate images directly in the high-dimensional pixel space (e.g., 512×512×3 = 786,432 dimensions for a single image, compared to typical VAE latent spaces of 64×64×4 = 16,384 dimensions). To make this tractable, the paper adopts the x-prediction and v-loss paradigm from JiT (Li and He, 2025) for pixel-space flow matching. This is not a novel contribution of the paper, but explaining it in detail is necessary because it is the mechanism that makes pixel-space generation work.

Rectified flow formulation and the linear schedule. Flow matching (Lipman et al., 2022) defines a continuous transformation between a simple base distribution (typically Gaussian noise) and the data distribution (natural images). The rectified flow variant (Liu et al., 2022) uses a particularly simple interpolation path: given a clean source image x₁ and a randomly sampled noise image x₀ ∼ 𝒩(0, I), the noisy intermediate image at time t is constructed as:

$x_t = t x_1 + (1 - t) x_0, \quad t \in [0, 1]$

where x₁ is the clean target image (a real image from the training data), x₀ is random Gaussian noise sampled independently for each training example, x_t is the intermediate noisy image produced by linearly interpolating between noise and clean image according to the time parameter, and t ∈ [0, 1] is a scalar timestamp sampled uniformly during training that controls the noise level (t=0 gives pure noise, t=1 gives the clean image).

What it computes: this linear interpolation schedule creates a continuous path from pure noise (at t=0) to clean image (at t=1) by simple weighted averaging. At each training step, a random t is sampled, a random noise image x₀ is generated, and x_t is computed as the weighted sum of the clean image and the noise.

Why this form: the rectified flow linear schedule is the simplest possible interpolation path and has the property that the "velocity field" (the derivative of x_t with respect to t, which is x₁ - x₀) is constant along the entire path. This means the model only needs to learn to predict a single constant vector field rather than a time-varying field, which reduces the complexity of the learning problem compared to other diffusion formulations where the drift and diffusion coefficients vary with time. Alternative formulations like DDPM (Ho et al., 2020) use more complex noise schedules where variance changes with t, making the learning target more complicated.

The x-prediction formulation. The model is trained to directly predict the clean image x₁ from the noisy image x_t:

$x_\theta = \pi_\theta(x_t, c, t)$

where x_θ is the model's prediction of the clean image (the θ subscript indicates it is a learned prediction parameterized by model weights θ), π_θ denotes the full unified model (vision-language backbone plus the flow matching head), x_t is the noisy input image at timestep t, c represents conditioning signals (text descriptions for text-to-image generation, or text plus a reference image for image editing tasks), and t is the timestep that tells the model the current noise level.

What it computes: the model takes a noisy image, the text prompt (and optionally an input image for editing), and the timestep as inputs, and produces a prediction of what the clean, noise-free image should look like. During training, this prediction is compared against the ground-truth clean image x₁. During inference (image generation), this prediction is used to estimate the velocity field that guides the denoising trajectory.

Why this form: directly predicting the clean image (x-prediction) rather than predicting the noise (ε-prediction, common in DDPM) or the velocity (v-prediction) means the model's output is always in the space of natural images, which may be easier for the unified transformer to learn because the flow matching head operates on top of representations that are also used for understanding natural images. The x-prediction approach also simplifies the loss computation because the target is simply the clean image itself, with no transformation needed.

Velocity parameterization and v-loss. Although the model predicts x_θ, the paper transforms this prediction into a velocity term for the actual loss computation, following JiT's recommendation:

$v_\theta = \frac{x_\theta - x_t}{1 - t}$

where v_θ is the model's predicted velocity field (derived from the clean image prediction), x_θ is the model's predicted clean image, x_t is the noisy input, and (1-t) is the remaining noise fraction (when t is close to 1, the image is almost clean and the denominator is small, so small prediction errors get amplified; when t is close to 0, the denominator is large, so errors are attenuated).

What it computes: this equation converts the model's clean image prediction into an estimate of the velocity field—the instantaneous direction and magnitude of change needed to move from the current noisy state toward the clean image. The numerator x_θ - x_t is the total displacement from current state to predicted clean image; dividing by (1-t) converts this total displacement into a rate (velocity) by accounting for how much time remains in the denoising process.

Why this form: velocity parameterization with x-prediction provides a more stable training signal. When t is close to 1 (image almost clean), directly predicting the clean image would involve a very small target change, leading to vanishing gradients. The velocity parameterization amplifies these small differences by dividing by the small (1-t), making the optimization landscape more uniform across different noise levels. This is mathematically equivalent to importance sampling in the time domain—it weights losses at different noise levels so that the model pays roughly equal attention to all denoising stages rather than focusing primarily on high-noise regimes (where errors are naturally larger).

The training objective—flow matching loss:

$\mathcal{L}_{\text{flow}} = \mathbb{E}_{t, c, x_1, x_0}\left[ \|v_\theta - v\|_2^2 \right]$

where 𝔼_{t,c,x₁,x₀} denotes the expectation over randomly sampled timesteps t ∼ 𝒰(0,1), conditioning signals c, clean images x₁ from the training data, and noise samples x₀ ∼ 𝒩(0,I), v_θ is the model's predicted velocity (derived from x_θ via the transformation above), v = x₁ - x₀ is the ground-truth constant velocity field for the rectified flow linear path, and ‖·‖₂² is the squared Euclidean (L2) distance between the predicted and true velocity vectors, summed over all pixel positions and color channels.

What it computes: for each training example, the model takes a noisy image x_t (produced by interpolating between random noise and a training image), predicts the velocity field that would transform it to the clean image, and is penalized by the mean squared error between its prediction and the true velocity (which is simply the difference between the clean image and the noise). This is a straightforward regression objective in pixel space.

Why this form: the L2 loss on velocity is the standard flow matching objective, corresponding to maximum likelihood estimation under the assumption of a Gaussian conditional distribution of velocities given the noisy state. The v-loss is particularly appropriate for the rectified flow linear schedule because the true velocity is constant (doesn't depend on t), so the model's prediction target is time-independent—it always tries to predict the same vector field regardless of the noise level, which simplifies learning compared to time-dependent velocity fields. If the paper had used ε-prediction (predicting the noise component x₀), the target would also be constant, but the velocity parameterization provides better numerical properties as discussed above.

Inference with the Euler solver. During generation, the model starts from pure random noise x₀ ∼ 𝒩(0,I) and iteratively denoises using the Euler method:

$x_{t'} = x_t + (t' - t)v_\theta$

where x_t is the current (noisier) image at the current timestep t, x_{t'} is the slightly cleaner image at a later timestep t' > t, (t' - t) is the step size (the amount of time to advance), and v_θ is the model's predicted velocity at the current state (derived from x_θ as v_θ = (x_θ - x_t)/(1-t)).

What it computes: starting from t=0 (pure noise), the model repeatedly: (1) predicts the clean image x_θ from the current noisy state, (2) converts this to a velocity estimate v_θ, (3) takes a small step in the direction of v_θ proportional to the time increment. After enough steps (typically 20–50 for Euler), the state reaches t=1, producing the final generated image.

Why this form: the Euler solver is the simplest possible ODE integrator (first-order, single-step), chosen for its computational efficiency and simplicity. The rectified flow linear schedule means the velocity field is approximately constant, so even a simple Euler method with relatively few steps can produce high-quality results—the model doesn't need sophisticated higher-order solvers (like Runge-Kutta) that would increase inference cost. This is a practical advantage of the x-prediction + rectified flow combination over other diffusion formulations.

The x-prediction to velocity conversion during inference. At inference time, the model produces x_θ, and the velocity is computed as:

$v_\theta = \frac{x_\theta - x_t}{1 - t}$

This is the same transformation used during training, ensuring consistency between training and inference. The inference loop then applies the Euler step to advance the state. At the final step (t=1), the model's x_θ prediction at t close to 1 is taken as the output image directly, since at t=1 the clean image should equal x₁.

Why pixel-space rather than latent-space flow matching. The paper explicitly chooses pixel-space operation because it enables: (1) fully end-to-end training without any separately-trained components that might introduce information bottlenecks, (2) direct access to fine-grained pixel-level details that VAE compression would discard, and (3) a single unified representation space for both understanding and generation (the transformer's hidden states operate on the same input representations regardless of task). The cost is that the generation target space is much higher-dimensional (512×512×3 ≈ 786K dimensions vs. 64×64×4 ≈ 16K for a typical VAE latent), making the regression problem harder. The masking-based feature learning scheme (described next) is partially designed to address this difficulty.

---

Masking-Based Visual Feature Learning

Learning unified visual representations directly in high-dimensional pixel space is fundamentally harder than learning in a VAE's compact latent space because pixel-space inputs contain massive redundancy—neighboring pixels are highly correlated, and large homogeneous regions (sky, walls, backgrounds) carry little information per pixel. This redundancy makes it easy for the model to rely on superficial shortcuts (e.g., memorizing pixel-level patterns rather than learning semantic structure) instead of developing robust visual representations that genuinely capture the content needed for both understanding and generation. The masking-based feature learning scheme is designed to combat this by forcing the model to learn under partial observation.

The masking operation. During training, a subset of image patches is randomly selected and replaced with a shared learnable mask token embedding before the sequence enters the transformer:

1. The input image is first divided into patches (16×16 pixels each) by the patchify layer, producing a grid of patch embeddings.
2. A masking ratio r is sampled uniformly from 0% to 50% (the paper states "randomly sampling the masking ratio from 0% to 50%," Section 2.2).
3. A random subset of patches is selected to be masked according to the sampled ratio, with each patch independently having probability r of being masked.
4. Masked patches have their patch embeddings replaced with a learnable mask token—a trainable embedding vector (initialized randomly and updated during training) that is shared across all masked positions. This is analogous to the [MASK] token in BERT-style language model pretraining.
5. The sequence of patch embeddings (some original, some replaced with mask tokens) is fed into the transformer along with text tokens.

The paper applies masking in 50% of training examples—meaning half the batches use masking and half don't—and this is only activated during the final 40% of the pretraining stage (i.e., after 60% of the pretraining steps have completed). The masking is applied identically to both understanding and generation examples, but plays different functional roles in each case.

Masking for generation examples—creating harder denoising problems. For generation (text-to-image) training examples, the model is asked to predict the clean image patches for both the masked and unmasked positions. This means:

• Visible patches (not masked): the model sees actual pixel values and must learn to denoise them and predict the corresponding clean image patches. This is the standard flow matching denoising task.
• Masked patches (replaced with mask tokens): the model only sees the learnable mask token embedding at those positions, with no pixel information. It must predict the clean image patches at those positions based solely on: (a) the surrounding visible patches (spatial context), (b) the text conditioning signal, and (c) any information the mask token itself has learned to encode during training.

This creates a harder denoising problem than standard flow matching because the model must reconstruct clean image content from partially observed noisy inputs. The mask token is forced to serve as a context-dependent prior—it must learn to encode useful information about what typically appears in a given context (conditioned on the visible patches and text) so that it can help the model predict the missing regions.

Why this helps generation: the masking forces the model to develop robust internal representations that capture the statistical structure of images at a semantic level, rather than simply learning low-level pixel-to-pixel mappings. When a large region is masked, the model cannot rely on local texture continuity; it must use high-level understanding of what objects should be present given the text and visible context. This is essentially an inpainting-like objective that teaches the model about object semantics, spatial layout, and composition—skills that are critical for generating coherent images from text descriptions.

Masking for understanding examples—regularization through partial observation. For understanding (VQA, captioning) examples, the model is asked to produce the ground-truth text response based on the partially masked visual input. The model must answer questions or generate captions while only seeing a fraction of the image patches (with the rest replaced by mask tokens). The mask token at those positions serves as a learned "unknown" representation—it signals to the model that information is missing at those locations, and the model must reason based on the available visual evidence.

Why this helps understanding: this is a form of regularization that prevents the model from overfitting to specific pixel-level patterns and forces it to develop representations that are robust to missing information. For example, if the question is "What color is the car?" and the car region happens to be masked, the model must either: (a) infer the answer from surrounding context (garage setting, other objects), or (b) rely on the mask token having learned (during generation training) to encode predictive information about typical car colors in similar contexts. This is analogous to dropout in neural networks—by randomly removing visual information during training, the model learns redundant, distributed representations that don't depend on any single patch being present. The paper states this serves "as a regularization mechanism that forces the model to perform multimodal reasoning under partial visual observation, leading to more robust visual representations" (Section 2.2).

Relationship to prior masked modeling methods. The paper explicitly draws connections to:

• MAE (Masked Autoencoders, He et al., 2022): which showed that masking a high fraction of image patches and training a vision transformer to reconstruct the missing patches produces strong visual representations for downstream tasks. Tuna-2's scheme differs in that (a) the masking is applied within a multimodal (vision+language) training framework rather than a vision-only reconstruction objective, and (b) the mask token is used for both reconstruction (generation) and semantic reasoning (understanding), whereas MAE only uses it for reconstruction.

• SigLIP 2 (Tschannen et al., 2025): which incorporates a masked prediction objective during its pretraining. The paper explicitly notes that "Tuna-R benefits less from masked training than Tuna-2" (Section 3.4) because its SigLIP 2 encoder was already trained with a similar objective, meaning the encoder has already internalized the benefits of masking-based representation learning.

• MaskGIT (Chang et al., 2022) and DeTok (Yang et al., 2025a): which use masking for visual generation by training models to predict masked tokens in a discrete token space. Tuna-2's approach is analogous but operates in continuous pixel space rather than a discrete VQ-VAE token space.

Design choices and justifications:

1. Why apply masking only during the final 40% of pretraining? The paper states this is because "we expect this strategy to serve as a representation enhancement strategy after the model has acquired basic multimodal knowledge" (Section 3.4). Applying masking from the very beginning of training would be too difficult—the model needs to first learn basic visual processing (edge detection, color recognition, simple shape recognition) before it can meaningfully reason under partial observation. The staged introduction lets the model build foundational capabilities first, then uses masking to refine and robustify those representations. The ablation study (Table 6) confirms that masking applied after 50k steps of standard pretraining improves performance over both no masking and over applying masking from the start.

2. Why sample the masking ratio from 0% to 50% rather than using a fixed ratio? Randomly sampling the ratio exposes the model to a range of difficulty levels—from fully visible images (0% masking, which is standard training) to heavily occluded images (50% masking). This prevents the model from adapting to a single masking level and encourages development of representations that gracefully degrade with missing information. The maximum of 50% (rather than higher ratios like 75–90% used in MAE) likely reflects the fact that the model must simultaneously perform language tasks (generate text) using the same transformer; with very high masking ratios, too much visual information is lost for the model to answer questions meaningfully, making the understanding objective too noisy.

3. Why a shared learnable mask token rather than, say, zero-filling? A learnable mask token allows the model to develop a specialized "unknown" representation that encodes the statistical expectation of what might appear at a masked location given the context. If the model simply filled masked positions with zeros, it would need to distinguish between "this position contains a zero-valued pixel" and "this position is masked"—a difficult disambiguation that wastes representational capacity. The learnable mask token (analogous to the [MASK] token in BERT) cleanly separates these two cases and can be optimized during training to contain information that helps both reconstruction and reasoning.

4. Why apply masking to only 50% of training examples? The 50% probability means that half the time, the model sees fully visible images (standard training), and half the time it sees partially masked images (regularized training). This balance ensures the model doesn't forget how to process unmasked images while still receiving the benefits of masking-based regularization. If masking were applied to 100% of examples, the model might develop representations specialized for partial observation that don't transfer well to standard (unmasked) inference.

---

Training Pipeline

Tuna-2's encoder-free design enables a fully end-to-end training pipeline with only two stages, avoiding the connector-alignment stage required by encoder-based modular architectures.

Stage 1: Full model pretraining (300,000 steps). This is the main training stage where the model learns unified multimodal representations from scratch. The entire model—patch embedding layer, transformer backbone, language modeling head, and flow matching head—is trained jointly from random initialization (for the new components) or from the Qwen2.5-7B-Instruct checkpoint (for the transformer and language head). The training data mixture consists of:

• 550M in-house image-text pairs, split 30% image captioning (multimodal understanding) and 70% text-to-image generation data. The 7:3 generation-to-understanding ratio was determined through ablation experiments (Section 3.3, Figure 5) as the optimal trade-off: "increasing the proportion of either generation or understanding data consistently reduces its corresponding training loss... a generation-to-understanding ratio of 7:3 (7g3u) achieves the best trade-off between the two objectives."
• 20% text-only data from Nemotron (Bercovich et al., 2025), included to maintain the language model's text understanding capabilities and prevent catastrophic forgetting of language skills during the visual-heavy pretraining. The paper states this "accounts for 20% of the total pretraining data."
• Masking-based feature learning is activated only during the final 40% of pretraining steps (i.e., the last 120,000 steps, after 180,000 steps of standard pretraining). During this phase, masking is applied to 50% of training examples with the masking ratio randomly sampled per example from 0% to 50%.

Training hyperparameters: The full model is trained end-to-end for 300,000 steps on 64 nodes using the AdamW optimizer (Loshchilov and Hutter, 2017) with weight decay decoupled from the learning rate update. The learning rate is set to 1×10⁻⁴ (the paper does not specify a learning rate schedule, but the use of the notation "a learning rate of 1×10⁻⁴" without mentioning warmup or decay suggests a constant or flat learning rate throughout pretraining). The input sequence length is padded to 16,000 tokens per GPU, which accommodates both the visual tokens (e.g., 1024 patches for a 512×512 image, plus text tokens for captions or generation prompts) and the target sequence. The batch size is not explicitly stated but can be inferred from the 64-node setup and 300K-step duration as being in the typical range for 7B-scale model training (likely hundreds to low thousands of sequences per global batch).

Stage 2: Supervised finetuning (SFT, 50,000 steps). After pretraining, the full model undergoes supervised finetuning on a curated corpus designed to refine specific capabilities:

• Image instruction-following: 13M conversational examples from the open-source FineVision dataset (Wiedmann et al., 2025). These are multi-turn dialogues where the user asks questions about images and the model provides answers, covering diverse visual reasoning tasks.
• Image editing: ~2M examples from OmniEdit (Wei et al., 2024), covering editing operations such as object addition, removal, replacement, attribute modification, background changes, and style transfer. Each example includes an input image, an editing instruction in natural language, and the target edited image.
• High-quality image generation: the paper mentions including "high-quality image generation" data but does not specify the source or quantity, beyond the general description of the SFT corpus.

This stage is trained for 50,000 steps with a lower learning rate of 2×10⁻⁵ (one-fifth of the pretraining learning rate), using AdamW optimizer, with the same 16K-token sequence length padding. The lower learning rate is standard for SFT to prevent overfitting to the smaller, higher-quality dataset and to avoid disrupting the representations learned during the large-scale pretraining stage.

Design choices and justifications:

1. Why end-to-end training rather than staged training? The paper emphasizes that Tuna-2's encoder-free design "enables fully end-to-end training... without requiring separate stages to train connector layers, which is a common design in encoder-based modular approaches" (Section 2.3). In encoder-based models like Tuna-R, the connector layer between the frozen/semi-frozen encoder and the LLM decoder must be trained first (the "alignment stage") before joint training, because randomly initialized connectors would produce garbage inputs to the transformer, destabilizing training. Tuna-2 avoids this because the patch embedding layer is simple (just a linear projection) and can be trained jointly with the transformer from the start.

2. Why 7:3 generation-to-understanding ratio? The ablation in Figure 5 shows that "the MSE loss is more sensitive to changes in the sampling ratio, while the CE loss varies within a relatively smaller range." This suggests that generation quality (measured by MSE on flow matching) benefits more from additional data than understanding quality (measured by cross-entropy on text)—generation is the harder, higher-dimensional task that requires more examples. The 7:3 ratio provides enough generation data to achieve good image quality while providing enough understanding data to maintain strong VQA and captioning performance. If the ratio were more extreme (e.g., 9:1), understanding would degrade; if more balanced (e.g., 1:1), generation would suffer.

3. Why include 20% text-only data? Multimodal pretraining can cause "catastrophic forgetting" of language capabilities that the base LLM acquired during its text-only pretraining. The text-only data from Nemotron serves as a regularizer that maintains the model's language understanding, instruction-following, and reasoning abilities. Without this, the model might overfit to the visual-conditional distribution (e.g., always expecting an image input) and lose its ability to engage in pure text reasoning or follow complex instructions that don't reference images. The 20% proportion is consistent with practices in other multimodal models (e.g., LLaVA-1.5 uses a similar fraction of text-only data during visual instruction tuning).

4. Why SFT with editing, instruction-following, and high-quality generation data? The pretraining stage uses mainly captioning and text-to-image generation data, which teaches the model basic multimodal associations but doesn't cover instruction-guided editing or complex multi-turn visual dialogue. The SFT stage fills these gaps: editing examples teach the model to modify images based on natural language instructions (a more complex task than generation from scratch because it must preserve irrelevant parts of the input image), instruction-following examples teach the model to engage in multi-turn visual conversations, and high-quality generation examples fine-tune the model on carefully curated images to improve aesthetic quality and prompt adherence beyond what the noisier pretraining data provides.

Tuna-R's additional alignment stage. As noted in Section 2.3, Tuna-R requires an extra step: "For Tuna-R, which includes a connector layer between the representation encoder and the LLM decoder, we add an extra alignment stage before Stage 1. In this stage, we train only the connector layer for a short period using image captioning and text-to-image generation data." This alignment stage is trained for only 3,000 steps with a relatively high learning rate of 5×10⁻⁴ (5 times the pretraining learning rate), using AdamW. The connector layer is a small network (typically an MLP with one or two hidden layers, though the exact architecture is not specified) that maps from the SigLIP 2 encoder's output dimension to Qwen2.5's input dimension. Training only the connector (keeping the encoder and LLM frozen) prevents the randomly initialized connector from producing out-of-distribution inputs that would destabilize the much larger transformer. After this brief alignment, the full model (encoder, connector, LLM, heads) is trained jointly in Stages 1 and 2 with the same settings as Tuna-2. The fact that Tuna-2 avoids this stage entirely is presented as an architectural advantage—simpler training, fewer stages, fully end-to-end from the first gradient step.

4. Key Insights and Innovations

Innovation 1: The Encoder-Free Design Is Not Just Viable — It's Eventually Superior, and the Crossover Point Is the Central Finding

The field's default assumption when building multimodal models is that pretrained vision encoders are necessary infrastructure. This is not a casually held belief — it is embedded in the architecture of virtually every production multimodal system, from LLaVA's CLIP-based encoder to Qwen2.5-VL's ViT to InternVL3.5's massive 6B-parameter vision tower. The logic is straightforward: vision encoders pretrained on billions of image-text pairs inject rich semantic knowledge that a randomly initialized model would take impractically long to acquire. The question the field had not systematically asked was: at what point, if ever, does that pretrained knowledge become a constraint rather than a scaffold?

Tuna-2's defining intellectual contribution is not that it removes encoders — encoder-free models exist in understanding-only settings (Mono-InternVL, NEO) and pixel-space generation exists (JiT, PixelFlow) — but that it demonstrates, through a controlled A/B comparison with Tuna-R, that the encoder's advantage is temporary. The learning curves in Figure 6 (Section 3.5) are the paper's most important single result: Tuna-R, with its SigLIP 2 encoder, starts ahead on understanding benchmarks (OCRBench, MMVP, V*) but is eventually caught and surpassed by the encoder-free Tuna-2. This is not a case of the encoder-free model being "competitive" — it is a case of the encoder-based model having a higher intercept but lower asymptote.

This finding reframes the discourse around transfer learning in multimodal systems. The prevailing narrative treats pretrained encoders as pure assets — free knowledge that bootstraps learning. Tuna-2's evidence suggests they are better understood as speed-vs-ceiling tradeoffs. The encoder's pretrained features accelerate early convergence because they provide a well-structured semantic space that the LLM decoder can quickly learn to query. But those same features are optimized for a different objective (contrastive image-text matching for SigLIP 2) and freeze in a particular resolution, spatial granularity, and representational format that may not be optimal for the joint understanding-and-generation tasks the unified model actually performs. The encoder's inductive biases, so helpful early on, become constraints that prevent the model from developing representations truly optimized for its end-to-end objectives.

The significance here is conceptual, not merely architectural. If encoders were universally beneficial — always helping, never hurting — then the optimal strategy would be to invest more in encoder pretraining. Tuna-2 shows this is not the case: at scale, end-to-end learning from pixels produces visual representations that are better aligned with the downstream tasks than off-the-shelf pretrained features. This is the multimodal analog of a pattern observed in NLP, where simple transformer architectures trained end-to-end on massive data eventually outperform systems that rely on separately pretrained linguistic features (part-of-speech taggers, dependency parsers, named entity recognizers). The pretrained components provided a valuable head start when data was scarce, but became unnecessary — and potentially limiting — when data and compute scaled up. Tuna-2 extends this lesson to the visual domain.

What makes this a fundamental rather than incremental contribution is that it provides causal evidence through controlled comparison. Prior encoder-free models demonstrated viability (showing that encoders aren't strictly necessary), but couldn't answer the "better or worse" question because they lacked a matched encoder-based baseline trained on identical data with identical objectives. Tuna-R is precisely that baseline: same LLM decoder, same training data, same flow matching head, same masking scheme — the only difference is the presence/absence of the SigLIP 2 encoder and its connector layer. Any performance differential can be attributed to the encoder, not to confounding differences in training recipe. This controlled comparison transforms the paper's claim from "encoder-free works" (an existence proof) to "encoder-free is eventually better" (a causal claim with boundary conditions), which is far stronger.

Innovation 2: Masking as a Unified Regularization Strategy That Simultaneously Stabilizes Pixel-Space Training and Improves Fine-Grained Perception

Training a model to both understand images and generate them directly in pixel space faces a fundamental difficulty: pixel-space inputs are extremely high-dimensional and redundant, with neighboring pixels carrying highly correlated information. This redundancy makes it easy for optimization to find shortcuts — the model can achieve low training loss by memorizing low-level pixel statistics rather than learning semantic representations that genuinely capture image content. The prevailing approaches to this problem in the generation literature involve architectural solutions: VAEs to compress images into low-dimensional latents, UNet backbones with careful inductive biases, or discrete tokenization with learned codebooks.

The paper's masking-based feature learning scheme (Section 2.2, Figure 3) takes a fundamentally different approach: rather than reducing input dimensionality through compression, it increases the difficulty of the learning task in a way that forces the model to develop robust internal representations. By randomly replacing image patches with a learnable mask token during training — applied identically to both understanding and generation examples — the model is forced to reason under partial observation and reconstruct missing visual content. This is a regularization-through-difficulty principle: making the task harder during training prevents the model from relying on superficial pixel-level shortcuts and forces it to learn representations that capture the underlying structure of visual scenes.

What distinguishes this from prior masked modeling approaches (MAE, SigLIP 2's masked pretraining, MaskGIT) is its dual-purpose design within a unified architecture. In prior work, masking serves a single function: in MAE, it creates a reconstruction task for representation learning; in SigLIP 2, it augments contrastive pretraining; in MaskGIT, it enables iterative generation. Tuna-2's scheme is the first to apply masking identically to both understanding and generation within a single training process, where it serves complementary roles: for generation, masking creates harder denoising problems that teach the model to inpaint missing regions based on context (improving generation coherence); for understanding, masking acts as a form of dropout-style regularization that forces the model to answer questions from incomplete visual evidence (improving representation robustness). The shared mask token that mediates both objectives becomes a learned representation of visual uncertainty — it encodes the model's estimate of what might be present at a masked location given the visible context and text conditioning.

The evidence for this dual benefit is in Table 6's ablation: masking improves both understanding benchmarks (OCRBench, MMVP, CountBench) and generation benchmarks (GenEval) for both Tuna-R and Tuna-2. Critically, Tuna-2 benefits more from masking than Tuna-R does — the paper attributes this to SigLIP 2 already having been pretrained with a similar masked prediction objective, meaning Tuna-R's encoder has already internalized some of the benefits that masking provides. This differential benefit is evidence that masking is not merely a training trick but a mechanism for developing visual representations that are genuinely absent from pretrained encoders, and that end-to-end models can acquire given appropriate regularization.

The appeal of the masking approach lies in its elegant simplicity and principled motivation. Rather than introducing new modules, loss terms, or architectural complexity, it simply withholds information from the model during training — a technique that requires no additional parameters beyond a single learnable embedding vector. The fact that this simple intervention yields consistent improvements across tasks and model variants suggests it addresses a fundamental challenge of pixel-space multimodal learning rather than a narrow optimization issue.

Innovation 3: Demonstrating That Pixel-Space Generation at Scale Is Competitive with Latent Diffusion, Contradicting the Prevailing Wisdom in Generation Research

For the past several years, the dominant paradigm in image generation has been latent diffusion: compress images into a compact latent space using a pretrained VAE, perform diffusion or flow matching in that latent space, and decode back to pixels. This approach, pioneered by Stable Diffusion (Rombach et al., 2022) and adopted by virtually every major generation system since (FLUX, DALL-E 3, Stable Diffusion 3, Qwen-Image), rests on the premise that operating directly in pixel space is intractably difficult at high resolutions — the dimensionality is too high, the optimization landscape too rugged, the computational cost too extreme. Pixel-space generation was largely relegated to small-scale experiments on datasets like ImageNet (Hoogeboom et al., 2023), with the implicit understanding that scaling to free-form text-to-image generation with natural language conditioning was not practical.

Tuna-2 challenges this consensus not through theoretical argument but through empirical demonstration at scale. The results in Table 2 (GenEval and DPG-Bench) and Table 3 (LLM-judge evaluations) show that Tuna-2's pixel-space generation, using the x-prediction and v-loss approach from JiT, achieves competitive performance with latent-space unified models (Tuna, Mogao, BAGEL) and approaches the quality of dedicated generation-only models. Specifically, on GenEval, Tuna-2 achieves 0.87 overall, compared to Tuna's 0.90 (which uses VAE latents) — a gap that is small enough to challenge the "necessity" of latent compression. The LLM-judge results in Table 3 are particularly striking: Tuna-2 is significantly preferred for diversity (48.4% win rate under GPT-5.4 vs. 30.9% for Tuna-R and 20.6% for Tuna), while remaining competitive on quality. This suggests that pixel-space generation may actually be advantageous for producing varied outputs from the same prompt — possibly because operating in the full pixel space allows the model to explore a richer set of visual variations than the VAE's compressed latent manifold permits.

The image reconstruction experiment in Table 5 and Figure 4 provides additional evidence: Tuna-2's learned pixel-space representations achieve reconstruction quality (rFID 0.15, PSNR 32.80, SSIM 0.93) that approaches the FLUX.1 VAE (rFID 0.06, PSNR 33.65, SSIM 0.93), which is a specialized, separately-trained compression model. A unified model trained for both understanding and generation can reconstruct images this well without a dedicated VAE — its internal representations, developed for multimodal tasks, are invertible enough to support high-fidelity reconstruction. This suggests that the unified training objective produces representations that preserve fine-grained visual information that dedicated VAE training would capture, but without the architectural separation.

What elevates this from an incremental result ("pixel generation works too") to a fundamental insight is its implication for the relationship between understanding and generation in unified models. The latent diffusion paradigm was convenient but architecturally awkward: understanding operated on one representation (semantic features from the CLIP-type encoder) while generation operated on another (VAE latents), creating the representation mismatch that unified models struggled to resolve. By demonstrating that pixel-space generation is competitive, Tuna-2 eliminates the architectural necessity for this separation — a single unified model can process raw pixels for both tasks using a single representation space, without the VAE bottleneck that previously seemed indispensable. The generation quality is slightly lower than latent-space approaches at equivalent scale, but the paper's scaling analysis (Figure 6) shows the gap narrowing with more training, suggesting that pixel-space generation is not fundamentally limited but rather requires more data to match the efficiency of latent-space methods.

Innovation 4: Pixel-Centric Benchmarks Reveal That Encoder-Free Models Excel at the Very Tasks Where Modular Architectures Should Be Strongest

A subtle but important finding emerges from the benchmark disaggregation in Table 1: Tuna-2's strongest advantages over both encoder-based UMMs and understanding-only LMMs appear on the pixel-centric benchmarks — V*, CountBench, and VisuLogic — that explicitly test fine-grained visual perception. These are tasks where the conventional wisdom would predict an encoder-based model to excel, because pretrained encoders are supposed to provide rich visual features. Instead, it is the encoder-free Tuna-2 that achieves the best results: 59.2 on V* (vs. 57.6 for Tuna-R and 52.4 for the latent-space Tuna), 81.7 on CountBench (vs. 77.8 for Tuna-R and 73.5 for Tuna), and 28.8 on VisuLogic (vs. 26.2 for Tuna-R and 22.4 for Tuna). These are substantial margins on benchmarks explicitly designed to expose the limits of multimodal models' visual perception.

This pattern is not obvious a priori. A SigLIP 2 encoder pretrained on massive image-text data with explicit dense feature objectives should capture fine-grained visual details — it was designed to do so. The fact that an encoder-free model with simple patch embeddings eventually outperforms it on these tasks suggests something deeper: pretrained encoders discard information that is useful for these tasks, and joint end-to-end training from pixels can preserve it. SigLIP 2's contrastive pretraining encourages invariance to precisely the kinds of low-level visual features — exact counts, tiny objects, subtle spatial relationships — that CountBench and V* test. The encoder's training objective rewards representations that are stable across variations in viewpoint, lighting, and scale, which means it learns to ignore the pixel-level details that distinguish "seven objects" from "eight objects." Tuna-2, trained end-to-end on tasks that require attending to these details (captioning requires mentioning specific numbers of objects, generation requires rendering the correct count), learns to preserve them because they matter for the training objectives.

This finding inverts the standard narrative about pretrained features. The conventional wisdom says: pretrained encoders provide strong general-purpose visual representations; fine-tuning adapts them to specific tasks. Tuna-2's results suggest: for tasks requiring fine-grained visual discrimination, pretrained encoders' invariance properties — usually considered a feature, not a bug — become a liability, and learning from scratch with task-appropriate objectives produces superior representations. This is significant beyond the specific benchmarks because it identifies a regime where transfer learning is counterproductive, not just inefficient — the pretrained features don't just fail to help, they actively constrain what the model can learn by filtering out information the downstream tasks need.

The attention map visualizations in Figure 7 (Section 3.6) provide qualitative evidence for this interpretation. Tuna-2's attention is more precise and semantically aligned than both encoder-based UMMs and understanding-only LMMs, particularly in challenging settings with misleading linguistic contexts or visual distractors. This precision is exactly what you would expect if the model's visual representations are optimized end-to-end for the tasks it performs, rather than inherited from an encoder optimized for a different objective. The monolithic architecture forces the model to develop visual features that serve both understanding and generation from the same hidden states, and this pressure produces representations that are more faithfully grounded in the image content rather than in the semantic abstractions that contrastive pretraining encourages.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper evaluates on a comprehensive suite of multimodal benchmarks rather than a single dataset. For understanding: GQA (Hudson and Manning, 2019), RealWorldQA (xAI), MMVet (Yu et al., 2023), MMMU (Yue et al., 2024), MMVP (Tong et al., 2024), SEED-Bench2+ (Li et al., 2024b), AI2D (Kembhavi et al., 2016), ChartQA (Masry et al., 2022), and OCRBench (Liu et al., 2024) for general VQA; plus V* (Wu and Xie, 2024), CountBench (Paiss et al., 2023), and VisuLogic (Xu et al., 2025) for pixel-centric fine-grained perception. For generation: GenEval (Ghosh et al., 2023) and DPG-Bench (Hu et al., 2024). For editing: ImgEdit (Ye et al., 2025). For reconstruction: ImageNet validation set (Deng et al., 2009). No single test set size is reported; GenEval and DPG-Bench use standard evaluation sets, the LLM-judge evaluation uses 1.5K sampled text prompts, and understanding benchmarks follow their standard evaluation protocols.

• Base model(s). All Tuna variants use Qwen2.5-7B-Instruct (Qwen et al., 2024) as the LLM decoder backbone. Tuna-R additionally uses SigLIP 2 So400M (Tschannen et al., 2025) as the representation encoder. The 7B scale is chosen as representative of contemporary LLM capabilities and enables direct comparison with other 7B-scale UMMs in the literature. For the ablation study on masking (Section 3.4), the smaller Qwen-2.5-Instruct-1.5B backbone is used to reduce computational cost.

• Metrics. Multimodal understanding is evaluated using each benchmark's standard accuracy metric (exact match or equivalent, as defined by the respective benchmark authors). For generation, GenEval reports per-attribute accuracy (single object, two objects, counting, colors, position, color attribute) and an overall average; DPG-Bench reports per-category scores (global, entity, attribute, relation, other) and an overall average. For the LLM-judge evaluation (Table 3), GPT-5.4 and Claude Opus 4.7 select the best model among three candidates per prompt based on quality (realism, fine-grained detail, texture fidelity) and diversity (visual variation across four images from the same prompt), with win rates reported. For image editing, ImgEdit reports scores per editing type (Add, Adjust, Extend, Replace, Remove, Background, Style, Hybrid, Action) and a total average. For reconstruction, rFID (reconstruction Fréchet Inception Distance, lower is better), PSNR (Peak Signal-to-Noise Ratio, higher is better), and SSIM (Structural Similarity Index Measure, higher is better) are reported on the ImageNet validation set.

• Baselines. The paper compares against three categories of models. Understanding-only LMMs: LLaVA-1.5 (Liu et al., 2023, 7B), Qwen-VL-Chat (Bai et al., 2023, 7B), LLaVA-OV (Li et al., 2024a, 7B), and Qwen2.5-VL (Bai et al., 2025b, 7B). Composite UMMs (those with separate understanding and generation pipelines): TokenFlow-XL (Qu et al., 2025, 14B), BLIP3-o (Chen et al., 2025a, 4B/8B), Tar (Han et al., 2025, 7B), and X-Omni (Geng et al., 2025, 7B). Native UMMs: BAGEL (Deng et al., 2025, 14B), Ming-UniVision (Huang et al., 2025a, 16B), Harmon (Wu et al., 2025d, 1.5B), JanusFlow (Ma et al., 2025b, 1.3B), Emu3 (Wang et al., 2024b, 8B), VILA-U (Wu et al., 2024, 7B), Janus-Pro (Chen et al., 2025c, 7B), Show-o2 (Xie et al., 2025b, 7B), OneCat (Li et al., 2025, 9B), Tuna (Liu et al., 2025, 7B), and Mogao (Liao et al., 2025, 7B). For generation-only baselines: SD3-M (Esser et al., 2024, 2B), FLUX.1 [dev] (Batifol et al., 2025, 12B), LongCat-Image (Team et al., 2025, 6B), Qwen-Image (Wu et al., 2025a, 20B), Seedream 3.0 (Gao et al., 2025), Z-Image-Turbo (Cai et al., 2025, 6B). For editing baselines: FLUX.1, Qwen-Image, OmniGen (Xiao et al., 2025), BAGEL, UniWorld (Lin et al., 2025a), OmniGen2 (Wu et al., 2025b), and GPT-Image. For reconstruction baselines in Table 5: SD-VAE, GigaTok, VA-VAE, DC-AE, MAE-Tok, TexTok, FLUX.1[dev]-VAE (specialized tokenizers), and UniTok, TokenFlow, X-Omni, MingTok, RAE, PS-VAE (unified tokenizers). Tuna-R serves as the internal encoder-based baseline for controlled comparison with Tuna-2.

• Generation budget / compute accounting. All Tuna variants (Tuna, Tuna-R, Tuna-2) are 7B-parameter models trained on identical hardware (64 nodes) and identical data mixtures, making training compute directly comparable. For inference, the paper uses each benchmark's standard evaluation protocol without per-example budget constraints — the models generate answers or images following the benchmark's requirements (e.g., GenEval uses per-attribute evaluation prompts, understanding benchmarks use standard VQA evaluation). No per-example generation budget trading is performed (unlike the compute-optimal scaling paper in the example). For the training dynamics ablation (Section 3.3, Figure 5), compute is measured by the data sampling ratio between generation and understanding examples. For the scaling analysis (Section 3.5, Figure 6), compute is measured by "training dataset size (measured by the number of tokens consumed by the model)," with performance plotted at multiple checkpoints during training.

• Cross-validation / statistical protocol. The paper does not employ cross-validation or report confidence intervals. Results in Tables 1–5 are point estimates from single training runs evaluated on standard test sets. The LLM-judge evaluation (Table 3) uses two judges (GPT-5.4 and Claude Opus 4.7) as a form of inter-annotator reliability check, with consistent findings across both judges providing some robustness. The controlled comparison between Tuna-R and Tuna-2 (Section 3.5) tracks performance at multiple training checkpoints to show trends rather than relying on single-point comparisons, which provides some protection against noise in individual checkpoint evaluations. However, the absence of error bars, confidence intervals, or multiple training runs is a genuine limitation of the experimental methodology.

Main Quantitative Results

Multimodal Understanding Benchmarks

Headline result: Tuna-2 achieves state-of-the-art results among all 7B-scale native UMMs across the comprehensive benchmark suite in Table 1, while also outperforming the encoder-based Tuna-R on most benchmarks. Table 1 reports results across 12 benchmarks. On general VQA benchmarks, Tuna-2 achieves: GQA 65.0 (vs. Tuna-R 63.5, Tuna 63.9, and the next-best native UMM Show-o2 at 63.1), RealWorldQA 67.7 (vs. Tuna-R 67.9 — the one general benchmark where Tuna-R slightly leads), MMVet 51.7 (vs. Tuna-R 46.7, a substantial +5.0 point margin), MMMU 50.7 (vs. Tuna-R 51.1 — essentially tied), MMVP 77.3 (vs. Tuna-R 74.7 and Tuna 70.7, a clear advantage), SEED-Bench2+ 61.1 (vs. Tuna-R 58.4), AI2D 79.6 (vs. Tuna-R 79.4 — effectively tied), ChartQA 85.6 (vs. Tuna-R 85.6 — tied), and OCRBench 79.7 (vs. Tuna-R 78.3 and Tuna 74.3).

Relative performance against understanding-only LMMs. Tuna-2's GQA score of 65.0 falls below the 7B-scale understanding-only models Qwen2.5-VL (60.7) but represents a different model class; compared to the strongest 7B understanding-only model on each benchmark, Tuna-2 is generally competitive but not dominant on general VQA — Qwen2.5-VL achieves higher MMVet (61.7 vs. 51.7) and MMMU (58.6 vs. 50.7). This gap likely reflects the tradeoff of joint generation-understanding training: dedicating model capacity and training budget to generation reduces peak understanding performance compared to understanding-only systems, though the gap is not catastrophic.

Pixel-centric benchmarks — where Tuna-2 excels. On tasks specifically designed to test fine-grained visual perception (the pixel-centric benchmarks), Tuna-2 shows the clearest advantages: V* 59.2 (vs. Tuna-R 57.6, Tuna 52.4, and the best understanding-only model Qwen2.5-VL at 71.2 — but among UMMs, Tuna-2 leads all native UMMs and most composite UMMs); CountBench 81.7 (vs. Tuna-R 77.8, Tuna 73.5, and the best overall among all models compared at 7B scale); VisuLogic 28.8 (vs. Tuna-R 26.2, Tuna 22.4). The CountBench result of 81.7 is the highest among all models in Table 1 at any scale below 14B parameters. This pattern — Tuna-2 performing roughly comparably to encoder-based models on general VQA but showing clear superiority on fine-grained perception — directly supports the paper's claim that encoder-free pixel-space training preserves visual details that pretrained encoders discard.

Comparison with larger models. Several baselines in Table 1 exceed 7B parameters and are grayed out: BAGEL (14B) achieves higher scores on most benchmarks (GQA 66.4, MMVet 67.2, MMMU 55.3, MMVP 85.0) as expected given its 2× parameter advantage; Ming-UniVision (16B) shows mixed performance, with Tuna-2 surpassing it on 8 of 12 benchmarks despite Ming-UniVision's more than 2× larger parameter count. The fact that a 7B encoder-free model can match or exceed a 16B encoder-based UMM on most understanding benchmarks is a strong efficiency argument for the encoder-free design.

Image Generation Benchmarks

Headline result: Tuna-2 achieves competitive generation quality with state-of-the-art native UMMs on GenEval (0.87 overall) and DPG-Bench (86.54 overall), though it slightly trails the encoder-based Tuna-R (GenEval 0.88, DPG-Bench 86.35) and the latent-space Tuna (GenEval 0.90, DPG-Bench 86.76) — Table 2.

GenEval per-attribute breakdown. On GenEval's six attributes, Tuna-2 scores: single object 0.99, two objects 0.96, counting 0.80, colors 0.91, position 0.84, color attribute 0.76, overall 0.87. The counting attribute (0.80) and color attribute (0.76) are the weakest scores, which is consistent with these being the most challenging attributes for text-to-image alignment across all models in the benchmark. Compared to Tuna, Tuna-2 lags on position (0.84 vs. 0.88), color attribute (0.76 vs. 0.83), and counting (0.80 vs. 0.81), while matching on single object (0.99 vs. 1.00) and two objects (0.96 vs. 0.97). The pattern suggests that pixel-space generation is competitive for object presence but loses ground on precise attribute binding and spatial reasoning — tasks where the VAE's compressed latent space may provide beneficial regularization.

DPG-Bench per-category breakdown. Tuna-2 scores: global 89.50, entity 91.40, attribute 92.07, relation 91.91, other 88.81, overall 86.54. Compared to Tuna: global 89.50 vs. 90.42, entity 91.40 vs. 91.68, attribute 92.07 vs. 90.94 (Tuna-2 actually leads here), relation 91.91 vs. 91.87 (essentially tied), other 88.81 vs. 90.73. The overall score of 86.54 vs. Tuna's 86.76 is a negligible gap of 0.22 points — within what might be expected from evaluation noise. Compared to the strongest native UMM on GenEval (Mogao at 0.89), Tuna-2 trails by 0.02, and compared to the strongest on DPG-Bench (Tuna at 86.76), it trails by 0.22. These are small margins that support the claim that pixel-space generation is "competitive" with latent-space approaches.

Comparison with generation-only models. The best generation-only models (Qwen-Image at 0.87 GenEval, LongCat-Image at 0.87 GenEval, Seedream 3.0 at 0.84 GenEval and 88.27 DPG-Bench) outperform most UMMs including Tuna-2, which is expected since they dedicate their entire model capacity to generation without the competing understanding objective. The gap between Tuna-2 (0.87 GenEval) and the best generation-only model at similar scale (LongCat-Image at 0.87 GenEval) is essentially zero, which is remarkable given that Tuna-2 simultaneously supports multimodal understanding.

LLM-judge evaluation (Table 3). The LLM-judge evaluation compares image quality and diversity across Tuna, Tuna-R, and Tuna-2. Under GPT-5.4 judgment: Tuna-2 wins on quality 32.1% of the time (vs. Tuna-R 35.7%, Tuna 22.3%) and on diversity 48.4% of the time (vs. Tuna-R 30.9%, Tuna 20.6%). Under Claude Opus 4.7: quality 34.8% (vs. 37.2% for Tuna-R, 28.1% for Tuna), diversity 41.9% (vs. 29.9% for Tuna-R, 28.2% for Tuna). The diversity advantage is the most striking result — Tuna-2 wins on diversity roughly 1.5–2.4× more often than either encoder-based variant under both judges. The quality results are more nuanced: Tuna-R is consistently preferred on quality (35.7% vs. 32.1% under GPT-5.4; 37.2% vs. 34.8% under Claude), suggesting that the representation encoder does provide a modest quality advantage for individual image fidelity, while the encoder-free design provides a substantial diversity advantage. This supports the paper's narrative that the encoder's inductive biases help generation quality but may constrain the variety of outputs the model can produce, since the encoder's fixed feature space limits the range of visual variations that can be expressed.

Image Editing

Headline result: Tuna-2 achieves strong editing performance on ImgEdit (Table 4) with a total score of 4.09, outperforming earlier UMMs like OmniGen (2.96), BAGEL (3.20), UniWorld (3.26), and OmniGen2 (3.44), while trailing Tuna (4.31) and Tuna-R (4.18).

Per-editing-type breakdown. Tuna-2 scores: Add 4.34, Adjust 4.13, Extend 2.22, Replace 4.53, Remove 4.42, Background 4.36, Style 4.58, Hybrid 3.91, Action 4.28. The Extend score (2.22) is notably lower than other operations, which is consistent across all unified models (Tuna scores 2.46 on Extend, Tuna-R 2.38) — image extension (outpainting) appears to be a fundamentally harder task than inpainting-style editing. Compared to Tuna-R, Tuna-2's largest gaps are on Hybrid (3.91 vs. 4.06) and Action (4.28 vs. 4.43), while it is very close on Replace (4.53 vs. 4.61), Remove (4.42 vs. 4.48), and Background (4.36 vs. 4.44). The consistent ~0.1–0.2 point gap per category accumulates to a 0.09 total point difference, indicating that the encoder provides a small but systematic benefit across editing operations. Compared to the best generation-only editing model (GPT-Image at 4.20 total), Tuna-2 trails by only 0.11 points, which is notable for a unified model that also performs understanding.

Image Reconstruction

Headline result: Tuna-2 achieves strong reconstruction quality (rFID 0.15, PSNR 32.80, SSIM 0.93) that approaches specialized tokenizers and ranks first among unified tokenizers — Table 5.

Comparison with specialized tokenizers. Against the FLUX.1[dev]-VAE (rFID 0.06, PSNR 33.65, SSIM 0.93) — arguably the strongest specialized image tokenizer in the comparison — Tuna-2's reconstruction quality is very close: rFID 0.15 vs. 0.06 (gap of 0.09, small in absolute terms), PSNR 32.80 vs. 33.65 (gap of 0.85 dB, which is perceptually small), SSIM 0.93 vs. 0.93 (identical). Against other specialized tokenizers like SD-VAE (rFID 1.06, PSNR 28.62), DC-AE (rFID 0.22, PSNR 26.15), and VA-VAE (rFID 0.26, PSNR 28.59), Tuna-2 substantially outperforms all of them. The only specialized tokenizer that matches or exceeds Tuna-2 is the FLUX.1 VAE.

Comparison with unified tokenizers. Among unified tokenizers designed for both semantic understanding and reconstruction — UniTok (rFID 0.38), TokenFlow (rFID 0.63), X-Omni (rFID 8.30), MingTok (rFID 0.53), RAE (rFID 0.61), PS-VAE (rFID 0.20) — Tuna-2 and Tuna-R (rFID 0.12) are clear leaders. The gap to the next-best unified tokenizer (PS-VAE at rFID 0.20) is substantial (0.05 rFID improvement), and the gap to methods like RAE (rFID 0.61) or TokenFlow (rFID 0.63) is large. This establishes that Tuna-2's unified pixel-space representations preserve reconstruction-relevant information substantially better than most specialized unified tokenizers, which are explicitly designed and trained for reconstruction quality.

Qualitative results (Figure 4). The visual comparison in Figure 4 shows that Tuna-R and Tuna-2 produce reconstructions that are visually nearly indistinguishable from the FLUX-VAE reconstruction, while RAE (a non-KL-regularized VAE approach) shows noticeable artifacts. This is important because KL-regularized VAEs like FLUX-VAE incorporate a strong prior that stabilizes training but can limit reconstruction fidelity; the paper argues that Tuna-2 achieves comparable reconstruction without this regularization, suggesting that the end-to-end unified training provides sufficient regularization on its own.

Training Dynamics: Understanding vs. Generation Tradeoffs

Headline result: The generation-to-understanding data ratio significantly affects both training losses, with MSE (flow matching loss) being more sensitive to the ratio than CE (language modeling loss), and a 7:3 ratio providing the best tradeoff — Figure 5 (Section 3.3).

Loss curve analysis. Figure 5 plots both MSE and CE loss curves for different ratios (8g2u, 7g3u, 5g5u, 3g7u — where g = generation, u = understanding). Key observations: (1) increasing the proportion of generation data consistently reduces MSE loss, with a clear monotonic relationship (more generation data → lower generation loss); (2) CE loss varies within a "relatively smaller range" across ratios, suggesting that understanding performance is less data-hungry than generation performance at this scale; (3) the 7g3u ratio achieves the best tradeoff — it produces MSE loss close to the 8g2u setting while maintaining CE loss close to the 5g5u setting. The asymmetry in sensitivity (MSE more responsive than CE) aligns with the intuition that generation is the harder, higher-dimensional task requiring more training examples to reach good performance.

This result has an important practical implication that goes unstated in the paper: the 7:3 optimal ratio is likely specific to the model scale, training duration, and data quality used in these experiments. At different scales (larger models, longer training, different data distributions), the optimal ratio might shift — a larger model might benefit from more understanding data to fully utilize its capacity, or might saturate on generation more quickly. The paper does not ablate the ratio at multiple scales, which limits the generalizability of this finding.

Scaling Analysis: Encoder-Free vs. Encoder-Based Learning Curves

Headline result: Tuna-R (encoder-based) initially outperforms Tuna-2 on understanding benchmarks, but Tuna-2 catches up and eventually surpasses Tuna-R after sufficient training; on generation, Tuna-R maintains a consistent advantage that narrows with training scale — Figure 6 (Section 3.5).

Understanding curves (OCRBench, MMVP, V).* On OCRBench, Tuna-R starts ahead (higher accuracy at early checkpoints) but Tuna-2's curve has a steeper slope and crosses Tuna-R's curve approximately midway through training, ending with a clear advantage at the final checkpoint. On MMVP, the same pattern holds: Tuna-R starts higher, Tuna-2 catches up and surpasses. On V*, the crossing point appears later in training but Tuna-2 still achieves a higher final accuracy. These three benchmarks represent different aspects of visual understanding — OCRBench tests text reading, MMVP tests visual perception with challenging distractors, V* tests guided visual search for small objects — and the consistent crossover pattern across all three supports the robustness of the finding.

Generation curve (GenEval). On GenEval, Tuna-R consistently outperforms Tuna-2 throughout training, but the gap narrows: at the final pretraining checkpoint, the curves are close, and after SFT (marked on the plots), they achieve "nearly identical performance." This suggests that the encoder's benefit for generation quality — likely its provision of structured semantic features that help the flow matching head learn the image manifold — is real but diminishes as the encoder-free model accumulates enough data to learn comparable structure from scratch.

Interpretation of the crossover. The crossover pattern on understanding benchmarks is the paper's most important scaling result because it directly supports the central claim that "pretrained vision encoders are not necessary for multimodal modelling, and end-to-end pixel-space learning offers a scalable path toward stronger visual representations" (abstract). If Tuna-2 had remained below Tuna-R across all training scales, the encoder would be demonstrably necessary. If Tuna-2 had been ahead from the start, encoders would be demonstrably harmful. The crossover — helpful early, constraining late — is the most nuanced and interesting finding: it suggests that encoders provide a valuable initialization that accelerates early learning, but that this initialization is suboptimal as a final representation because it was optimized for a different objective (contrastive pretraining) and imposes constraints (fixed resolution, semantic invariance) that limit what the model can learn from end-to-end task-driven optimization.

Caveats about the scaling analysis. The scaling curves in Figure 6 show performance against training tokens consumed, but the maximum training scale is 300K steps of pretraining plus 50K steps of SFT. It is possible that with even more training, Tuna-2's advantage on understanding would continue to grow (the curves don't appear to have saturated), while the generation gap might close completely. Alternatively, Tuna-R might re-establish a lead at much larger scales if the encoder's semantic knowledge becomes more valuable with more data. The paper cannot distinguish these possibilities because it tests only one training budget. Additionally, the scaling analysis uses only three understanding benchmarks and one generation benchmark — a broader set of benchmarks at each checkpoint would provide a more complete picture of how the representation gap evolves across task types.

Ablation Studies and Robustness Checks

Masking-based feature learning (Table 6): Applying masking during the final 40% of pretraining improves performance across both understanding and generation benchmarks for both Tuna-R and Tuna-2. On Tuna-R with the 1.5B backbone, masking improves OCRBench from 58.3 to 59.2 (+0.9), MMVP from 56.7 to 58.0 (+1.3), CountBench from 57.2 to 58.2 (+1.0), and GenEval from 55.7 to 56.0 (+0.3). On Tuna-2 with the 1.5B backbone, masking improves OCRBench from 55.4 to 56.8 (+1.4), MMVP from 52.3 to 55.7 (+3.4), CountBench from 53.4 to 57.6 (+4.2), and GenEval from 47.6 to 48.2 (+0.6). Tuna-2 benefits substantially more from masking than Tuna-R does — the improvements are roughly 1.5–4× larger in absolute terms — which the paper attributes to Tuna-R's SigLIP 2 encoder already having been pretrained with a similar masked prediction objective, meaning Tuna-R starts with representations that have partially internalized the benefits of masking. This differential benefit is consistent with the paper's narrative that end-to-end models need (and can effectively use) stronger regularization to develop robust representations, while pretrained encoders provide some of this robustness out of the box.

VAE vs. no VAE for understanding (Table 6, Tuna vs. Tuna-R): Comparing Tuna (which uses VAE latents) and Tuna-R without masking (both use the same 1.5B backbone and the same SigLIP 2 encoder), Tuna-R outperforms Tuna on all three understanding benchmarks: OCRBench 58.3 vs. 56.9 (+1.4), MMVP 56.7 vs. 54.0 (+2.7), CountBench 57.2 vs. 55.6 (+1.6). This supports the paper's claim that VAE compression discards fine-grained visual information useful for understanding, since the only difference between Tuna and Tuna-R in this comparison is whether the representation encoder processes VAE latents or raw pixels. However, Tuna-R underperforms Tuna on GenEval (55.7 vs. 57.2, −1.5), suggesting that VAE latents are beneficial for generation quality in the early training regime (50K steps with the 1.5B backbone).

Generation-to-understanding data ratio (Figure 5): As described in the main results section above, this ablation establishes the 7g3u ratio as optimal and demonstrates asymmetric sensitivity of the two objectives to data mixture. An additional insight not highlighted earlier: the fact that CE loss varies within a relatively small range across extreme ratios (8g2u to 3g7u) suggests that the understanding objective can be learned effectively from relatively little data when the generation objective provides rich visual supervision — the model learns visual representations from the generation task that transfer to understanding. This is indirect evidence for positive transfer between the two tasks in the unified training paradigm.

Tuna vs. Tuna-R vs. Tuna-2 attention map analysis (Figure 7): This is a qualitative ablation rather than a quantitative one, but it provides mechanistic insight into why Tuna-2 outperforms on fine-grained understanding. The visualizations show four test cases: (1) "shining window" — Tuna-2 produces tight, accurate attention on the shining region, while Tuna-R, Tuna, and baseline LMMs produce more dispersed or mislocalized attention; (2) "purple object" — Tuna-2 cleanly highlights the target object, while other models show "dispersed attention or spurious activations in irrelevant regions"; (3) "dog cafe" misleading context — Tuna-2 attends to the actual dogs in the image rather than the misleading "cat cafe" text prompt or the "dog cafe" sign, while other models are pulled toward textual cues; (4) "football match" counterintuitive case — Tuna-2 correctly attends to the glass cup being kicked rather than the visually salient football or the football-related language cues. These attention maps provide face validity for the paper's claims about Tuna-2 learning more robust visual representations, but they are selected examples (not a systematic quantitative evaluation), so their evidential weight should be considered illustrative rather than conclusive.

LLM-judge inter-rater consistency (Table 3): Using both GPT-5.4 and Claude Opus 4.7 as judges provides a form of robustness check on the LLM-judge evaluation. The two judges agree on the qualitative ranking: Tuna-R preferred for quality (35.7% / 37.2%), Tuna-2 preferred for diversity (48.4% / 41.9%), Tuna least preferred on both. The quantitative differences between judges — Claude shows slightly higher preference rates for both Tuna-R quality (37.2% vs. 35.7%) and Tuna diversity (28.2% vs. 20.6%) — suggest some judge-specific biases but don't change the overall conclusions. Using two judges from different AI labs (OpenAI and Anthropic) provides some protection against single-judge systematic biases.

Reconstruction quality comparison with RAE (Figure 4): The qualitative comparison in Figure 4 places Tuna-R and Tuna-2 reconstructions alongside FLUX-VAE and RAE. RAE (Representation Autoencoder) is a non-KL-regularized VAE approach, and its reconstructions show visible degradation compared to the nearly perfect reconstructions from Tuna-R and Tuna-2. This is an informative negative result: it shows that simply removing KL regularization from a VAE (as RAE does) is not sufficient to achieve the reconstruction quality that Tuna-2 achieves through end-to-end unified training. The unified multimodal training objective apparently provides a form of implicit regularization that produces more faithful representations than non-regularized autoencoding alone.

Missing ablation — masking ratio and timing: The paper does not ablate the masking ratio range (0–50%), the probability of applying masking (50% of examples), or the timing of masking introduction (final 40% of pretraining). These hyperparameters are stated as design choices without empirical justification through ablation. It is possible that different settings — e.g., higher maximum masking ratio (75%), earlier masking introduction, or different application probabilities — would produce different results. The paper treats these choices as fixed based on prior work (MAE, SigLIP 2) and intuition, which is reasonable for a paper introducing many new components simultaneously, but readers should understand that the masking scheme's effectiveness may depend on these hyperparameters in ways the paper does not quantify.

Missing ablation — patch size: The paper uses a patch size of 16×16 pixels without ablating alternatives (e.g., 8×8 or 14×14). This choice affects the number of visual tokens per image and the granularity of visual information available to the model. Smaller patches would preserve more spatial detail at higher computational cost; larger patches would reduce cost but potentially lose fine-grained information needed for pixel-centric benchmarks. The paper does not explore this tradeoff.

Missing ablation — pretraining duration and the crossover point: The crucial finding that Tuna-2 eventually surpasses Tuna-R on understanding is demonstrated at a single training budget (300K pretraining steps). The paper does not show whether the crossover would occur earlier or later with different model sizes, different data scales, or different encoder choices. A larger encoder (e.g., InternViT-6B vs. SigLIP 2 So400M) might push the crossover point further out or change the asymptote entirely. The paper's claim that "end-to-end pixel-space learning offers a scalable path toward stronger visual representations" is supported at the tested scale but not proven at larger scales — the trend is suggestive but not conclusive.

Critical Assessment

Central claim: "Pretrained vision encoders are not necessary for multimodal modelling" (abstract). The paper provides strong evidence for this claim through the existence proof of Tuna-2: a fully encoder-free architecture that achieves competitive or superior performance across a comprehensive suite of benchmarks. The controlled comparison with Tuna-R demonstrates that removing the encoder does not cause catastrophic performance degradation — the model still learns effective visual representations. The claim is supported, with the important qualification that "not necessary" does not mean "never beneficial." The encoder-based Tuna-R maintains advantages on generation quality (Table 2, Table 3 quality metric) and image editing fidelity (Table 4), and provides faster early-training convergence (Figure 6). So pretrained encoders are not necessary for achieving competitive final performance, but they may be beneficial for generation quality, data efficiency, and training speed. The paper's abstract states this claim in its strongest form, while the body text provides appropriate nuance — readers should calibrate their takeaway to the latter.

Central claim: "End-to-end pixel-space learning offers a scalable path toward stronger visual representations for both generation and perception" (abstract). The evidence for "stronger visual representations for perception" is solid: Tuna-2 outperforms Tuna-R on 9 of 12 understanding benchmarks (Table 1), with particularly strong results on pixel-centric benchmarks, and the scaling curves (Figure 6) show Tuna-2's advantage emerging and growing with training scale. The evidence for "stronger visual representations for generation" is weaker: Tuna-2 trails Tuna-R on generation quality (GenEval 0.87 vs. 0.88, DPG-Bench 86.54 vs. 86.35, Table 3 quality metric), though the gap is small and narrows with training (Figure 6). The claim of a "scalable path" is supported by the trend lines in Figure 6 but is extrapolative — the paper tests only one training budget and one model scale. Whether the trends continue beyond 300K steps and whether they hold at larger model sizes or with more diverse training data are open questions. The paper provides evidence that the path exists and is promising, not that it has been fully traversed.

Central claim: "Tuna-2 achieves state-of-the-art performance in multimodal benchmarks" (abstract). This claim requires careful qualification. Among 7B-scale native UMMs, Tuna-2 clearly achieves state-of-the-art results on understanding (Table 1) and competitive results on generation (Table 2). However, Tuna-2 is not the best model overall on any single benchmark — it is the best 7B native UMM on understanding benchmarks, but understanding-only LMMs (Qwen2.5-VL) and larger UMMs (BAGEL at 14B) surpass it on multiple benchmarks. On generation, models like Mogao (0.89 GenEval) and Tuna (0.90 GenEval) slightly outperform Tuna-2. The claim is technically true within the paper's stated scope ("among all UMMs" in Table 1) but could be misread as claiming absolute state-of-the-art, which it does not achieve. The paper's careful table annotations (bolding best among UMMs, underlining second-best) provide appropriate context.

Claim: "The pretrained vision encoders' built-in inductive biases, such as fixed input resolutions and limited access to fine-grained low-level visual details" limit performance (Section 2.1). The paper provides strong indirect evidence through the pixel-centric benchmark results: Tuna-2 substantially outperforms encoder-based models on V*, CountBench, and VisuLogic, which directly test fine-grained visual perception (Table 1). The attention map visualizations (Figure 7) provide qualitative evidence for more precise visual grounding. However, the paper does not directly measure or ablate the specific mechanisms it identifies — it does not show that fixed resolution causes performance degradation (by testing Tuna-R at multiple resolutions), or that the encoder's semantic invariance specifically causes loss of low-level details (by probing the encoder's internal representations). The evidence is consistent with the claim but the causal chain is inferred rather than demonstrated. An experiment that modified SigLIP 2's output to preserve more fine-grained information and showed this closed the gap with Tuna-2 would strengthen the causal argument.

Claim: Masking-based feature learning "stabilize[s] training and encourage[s] the learning of more robust pixel-space representations" (Section 1). The ablation in Table 6 provides direct evidence: masking improves performance on all benchmarks for both Tuna-R and Tuna-2. The mechanism claim — that masking encourages robustness — is supported qualitatively by the attention map analysis (Figure 7), which shows Tuna-2 producing more stable attention under distracting conditions. However, the paper does not provide direct evidence of training stability (e.g., loss curve smoothness, gradient norm trajectories) or a formal measure of representation robustness (e.g., sensitivity to input perturbations). The claim is reasonably supported by the performance improvements, but the specific mechanism (stabilization, robustness) is inferred rather than measured.

Genuine weaknesses in the experimental design:

1. Single training run, no error bars. All results in Tables 1–5 are point estimates from single training runs. Without multiple runs or confidence intervals, the statistical significance of the performance differences between Tuna-R and Tuna-2 cannot be assessed. The gap on many benchmarks is small (e.g., GQA 65.0 vs. 63.5, MMMU 50.7 vs. 51.1) and could be within the noise range of training stochasticity. The authors mitigate this by reporting results across many benchmarks (consistency across 12 understanding benchmarks provides some protection against noise) and by showing learning curves (Figure 6) that demonstrate trends rather than single-point comparisons, but the lack of statistical rigor is a genuine limitation.

2. Single model scale (7B) for main results. The main results are all at the 7B scale. The scaling analysis (Figure 6) shows important trends but at only one scale. The claim that encoder-free designs are "a scalable path" would be much stronger with evidence at multiple scales — does the Tuna-2 vs. Tuna-R crossover happen earlier or later at 1B? At 70B? The smaller-scale ablation (Table 6, 1.5B backbone) provides some evidence that masking helps at smaller scale, but doesn't show the full comparison between Tuna-R and Tuna-2 at that scale. Without multi-scale evidence, the "scalable" claim is an extrapolation from a single data point.

3. Single encoder choice (SigLIP 2 So400M). Tuna-R uses exactly one representation encoder. The claim that encoder-free designs are better than encoder-based designs is tested only against this specific encoder. A different encoder — a larger one, one trained with a different objective (e.g., DINOv2 for dense features, InternViT for stronger semantics), or one with higher native resolution — might not show the same crossover pattern. The paper's finding that Tuna-2 benefits more from masking because SigLIP 2 already incorporates masking (Section 3.4) implies that the results are encoder-specific. A more comprehensive study would compare against multiple encoder choices to establish the generality of the finding.

4. Limited training scale for the crossover analysis. The 300K-step pretraining budget is substantial (64 nodes, 7B parameters) but may not be sufficient to observe the full trajectory of the Tuna-R vs. Tuna-2 gap. The generation curves in Figure 6 are still converging at the end of pretraining — with more training, would the gap close completely? Would Tuna-2 eventually surpass Tuna-R on generation as it did on understanding? The paper cannot answer this without longer training runs.

5. Missing ablation — training data scale. The paper uses 550M image-text pairs for pretraining. An important ablation would be to train Tuna-R and Tuna-2 at multiple data scales (e.g., 50M, 100M, 550M, 1B pairs) to test whether the encoder-free advantage emerges only above some data threshold. The paper's narrative implies that encoder-free models need sufficient scale to overcome the encoder's head start, but the threshold is not identified. If the crossover happens at 550M pairs, does it happen earlier with better data? Later with noisier data? This is unknown.

6. No systematic evaluation of encoder-free generation diversity. The LLM-judge diversity result (Table 3) is one of the paper's most interesting findings — Tuna-2 is substantially preferred for diversity. But this is measured for only 1.5K prompts with two judges, and diversity is a somewhat subjective metric. A more rigorous diversity evaluation (e.g., using LPIPS or FID-based diversity metrics, evaluating across multiple prompt categories, measuring mode collapse) would strengthen this claim. The paper presents diversity as a strength of the encoder-free design but the evidence, while suggestive, is limited.

7. The reconstruction experiment is after "lightweight finetuning." Table 5 reports reconstruction quality after "lightweight finetuning on an image reconstruction task." This means the reconstruction results do not purely reflect the quality of the representations learned during the main pretraining — they reflect the quality after additional task-specific training. This is a reasonable experimental design (it tests whether the representations support reconstruction, which is what matters), but readers should understand that the reported rFID/PSNR/SSIM are after this additional finetuning stage, not from the pretrained model directly.

Experiments that would have strengthened the paper:

1. Training Tuna-2 at multiple scales (1B, 3B, 7B, 13B) with matched Tuna-R baselines at each scale to map out how the crossover point shifts with model capacity.

2. Varying the pretraining data scale to identify the data threshold where Tuna-2 surpasses Tuna-R, which would provide practical guidance on when to prefer encoder-free designs.

3. Testing with alternative encoders (larger SigLIP variants, DINOv2, InternViT) to establish whether the encoder-free advantage is specific to SigLIP 2 So400M or generalizes across encoder types.

4. Directly measuring information preservation in the visual representations — e.g., training linear probes to decode fine-grained attributes (object count, spatial position, small object presence) from intermediate representations of Tuna-R vs. Tuna-2 — to provide causal evidence for the claim that encoders discard useful information.

5. Multiple training runs with error bars on key results (at least the Tuna-R vs. Tuna-2 comparisons on the main benchmarks) to assess statistical significance.

6. Systematic diversity evaluation using established metrics (LPIPS diversity, coverage, precision/recall for generative models) on a range of prompt types to quantify the diversity advantage suggested by Table 3.

7. Ablation of the SFT stage's contribution — the paper reports that after SFT, Tuna-2 and Tuna-R achieve "nearly identical" generation performance (Figure 6). Quantifying how much of the gap closure is due to SFT vs. the pretraining scale would clarify whether the encoder-free disadvantage is fundamentally a data-volume issue that SFT partially addresses.

6. Limitations and Trade-offs

Limitation 1: The Encoder-Free Design's Generation Quality Still Trails Latent-Space Methods, and the Gap May Not Close at Practical Training Scales

The assumption or constraint. Tuna-2 operates entirely in pixel space, which means it must learn to generate high-dimensional images (512×512×3 = 786,432 dimensions) directly rather than in a VAE-compressed latent space (typically ~16,384 dimensions). The paper implicitly assumes that with sufficient training data and compute, pixel-space generation will match or exceed latent-space generation quality. Across all generation evaluations, Tuna-2 consistently trails Tuna-R and the latent-space Tuna, even after the full training pipeline. The paper reports: Tuna-2 GenEval 0.87 vs. Tuna-R 0.88 vs. Tuna 0.90 (Table 2), DPG-Bench 86.54 vs. 86.35 vs. 86.76 (Table 2), and LLM-judge quality win rates of 32.1% for Tuna-2 vs. 35.7% for Tuna-R under GPT-5.4 (Table 3). On image editing, Tuna-2's total ImgEdit score of 4.09 lags Tuna-R's 4.18 and Tuna's 4.31 (Table 4). The paper acknowledges this pattern explicitly in Section 3.5: "Tuna-R consistently outperforms Tuna-2 throughout the entire training process" on generation, attributing it to "the semantic priors provided by the representation encoder."

The consequence. This is not a minor gap that obviously vanishes with more training. The scaling curves in Figure 6 show that Tuna-R and Tuna-2 achieve "nearly identical performance after SFT," but the curves are plotted at only one training budget and one model scale. A practitioner choosing between architectures faces a genuine tradeoff with no resolution from the paper's evidence: how much additional training would be required for Tuna-2 to match Tuna-R on generation, and does that training cost outweigh the architectural simplicity benefits? If pixel-space generation requires 2× or 3× more training compute to match latent-space quality, the end-to-end advantages may be economically outweighed by the generation deficit in applications where image quality is paramount (e.g., creative tools, design applications). The paper provides no scaling law that would let a practitioner estimate this tradeoff.

What evidence exists in the paper. The evidence is the consistent generation gap across all generation benchmarks (Tables 2, 3, 4) and the scaling curves in Figure 6 showing the gap narrowing but not fully closing at the tested training budget. The LLM-judge results in Table 3 provide a more nuanced picture: Tuna-2 leads substantially on diversity (48.4% win rate vs. 30.9% for Tuna-R under GPT-5.4) while trailing on quality (32.1% vs. 35.7%), suggesting the generation gap is specifically a fidelity gap rather than a variety gap. The paper does not measure whether the quality gap would close with longer training, different data, or larger model scale.

Mitigation status. The paper partially addresses this by demonstrating competitive performance (the gap to latent-space methods is small, e.g., 0.03 on GenEval overall), but does not provide evidence that the gap can be closed. The authors note the narrowing trend in Figure 6 and frame this as evidence that "the two model variants achieve nearly identical performance after SFT," but "nearly identical" still means Tuna-2 is consistently slightly behind. No ablation studies vary training duration to extrapolate the gap closure rate, and no experiments test larger model scales where pixel-space generation might be more (or less) disadvantaged relative to latent-space methods.

---

Limitation 2: The Difficulty Estimation Problem Is Completely Unaddressed — Tuna-2 Has No Mechanism for Deciding How Much Compute to Spend Per Example

The assumption or constraint. The paper's title and framing implicitly promise a practical system that can be deployed for multimodal tasks, but Tuna-2 treats all inputs uniformly during inference — the same architecture, same number of denoising steps, same generation resolution, and same computational budget are applied regardless of prompt complexity. The paper provides no mechanism for estimating how difficult a given generation or understanding task will be, and no adaptive allocation of compute based on that difficulty. This stands in contrast to the compute-optimal scaling paper in the reference example, where the central contribution was precisely a difficulty-conditioned policy for allocating inference compute. Tuna-2 inherits the standard inference paradigm of its generation framework (a fixed number of Euler solver steps for flow matching) without examining whether this uniform allocation is optimal.

The consequence. In deployment, uniform compute allocation is wasteful: an easy prompt ("a red circle on a white background") receives the same number of denoising steps as a complex prompt ("a bustling Renaissance marketplace with dozens of figures in period clothing, detailed architectural elements, and complex lighting"), and a simple visual question ("what color is this object?") uses the same forward pass budget as a challenging reasoning question requiring comparison of multiple fine-grained regions. For generation specifically, pixel-space flow matching operates on 786K-dimensional vectors, making each generation step substantially more expensive than latent-space methods — the cost of uniform over-allocation is proportionally higher. For understanding, the model processes all 1024 visual tokens (for a 512×512 image with 16×16 patches) regardless of whether the question could be answered from a small region of interest. A practitioner deploying Tuna-2 cannot trade off accuracy against latency or cost on a per-example basis, and has no principled way to set the number of inference steps other than using the fixed setting from the paper.

What evidence exists in the paper. The paper does not measure or discuss this limitation. The inference procedure is described only in general terms (Section 2.1: "During inference, we employ the Euler solver"), with no specification of the number of denoising steps used, no ablation of step count vs. generation quality, and no analysis of inference cost. The generation metrics (Table 2) are reported without any compute-per-example breakdown. The understanding benchmarks follow standard evaluation protocols that provide no signal about whether uniform processing is wasteful. This limitation is entirely unaddressed — the paper provides zero evidence about inference-time efficiency or the potential for adaptive compute allocation.

Mitigation status. Not addressed. The paper makes no claims about inference efficiency, does not include inference cost in any analysis, and does not propose a difficulty estimation mechanism. This is not an oversight specific to Tuna-2 — most multimodal model papers do not address this — but it is a consequential gap for a paper that claims to provide a "scalable path" for practical multimodal systems. Future work on adaptive denoising step counts, early-exit mechanisms for understanding, or difficulty-conditioned resolution selection would directly address this gap but is not suggested in the paper.

---

Limitation 3: Single Model Family, Single Model Scale — the Central "Encoder-Free Beats Encoder-Based" Finding Is Tested at Exactly One Configuration

The assumption or constraint. The paper's most important result — that Tuna-2 eventually surpasses Tuna-R on understanding — is demonstrated for exactly one model architecture (Qwen2.5-7B-Instruct decoder backbone), one encoder choice for Tuna-R (SigLIP 2 So400M), one pretraining data scale (550M image-text pairs, 300K steps), and one model size (7B parameters). The paper's abstract claims this shows "a scalable path toward stronger visual representations," but "scalable" implies that the advantage persists or grows across scales. The paper provides no evidence at other scales — no 1B-scale experiment, no 13B or 70B experiment, no experiment with a different LLM backbone (e.g., LLaMA-based), and no experiment with a different encoder for Tuna-R (e.g., a larger SigLIP variant, DINOv2, InternViT). The authors are transparent about the scope: all main experiments use Qwen2.5-7B-Instruct and SigLIP 2 So400M (Section 3.1), and the smaller-scale ablation (Section 3.4, Table 6) uses Qwen-2.5-Instruct-1.5B but only tests the masking scheme, not the full Tuna-R vs. Tuna-2 comparison.

The consequence. A practitioner cannot determine whether the Tuna-2 advantage generalizes to their setting. If they are using a different LLM backbone (e.g., LLaMA-3-8B), a different encoder (e.g., InternViT-300M), or a different model scale (e.g., 3B for edge deployment or 70B for maximum capability), the paper provides no guidance on whether the encoder-free advantage will appear, at what training scale the crossover occurs, or whether the advantage magnitude scales with model size. This is particularly acute for the generation tradeoff — if the encoder-based generation advantage widens at larger scales (because a bigger encoder provides richer semantic priors for the flow matching head), the paper's recommendation to prefer encoder-free designs could be wrong for large-scale deployments. Conversely, if the encoder-free understanding advantage grows with scale (because bigger models benefit more from end-to-end optimization), the paper understates the case for encoder-free designs at large scale. Neither hypothesis can be evaluated from the single-scale evidence.

What evidence exists in the paper. The only multi-scale evidence is indirect: the 1.5B backbone ablation (Table 6) tests masking effectiveness at a smaller scale, showing that masking helps Tuna-2 more than Tuna-R (consistent with the 7B finding), but does not compare full Tuna-R and Tuna-2 performance at 1.5B across the full benchmark suite. The attention map analysis (Figure 7) includes comparisons with understanding-only LMMs of different architectures (Qwen2.5-VL-7B, LLaVA-OV-1.5-8B, Penguin-VL-8B) suggesting some cross-architecture generality, but these are different model families with different training recipes, not controlled comparisons varying only the encoder presence. The paper explicitly acknowledges this scope limitation nowhere — the claims are stated in general terms without qualification about model specificity.

Mitigation status. Not addressed. The paper does not acknowledge the single-scale limitation, does not suggest that results might vary with model family or scale, and does not propose multi-scale experiments as future work. The "scalable path" language in the abstract is stated as a conclusion, not as a hypothesis requiring further testing. A more cautious framing — "at the 7B scale with Qwen2.5 and SigLIP 2, encoder-free designs match or exceed encoder-based designs after sufficient training" — would better reflect the evidence while acknowledging the scope limitation. Future work systematically varying model scale, encoder type, and LLM backbone would be needed to establish the claimed scalability.

---

Limitation 4: The Pixel-Space Design Imposes a Fundamental Latency-Throughput Tradeoff That Is Never Quantified or Discussed

The assumption or constraint. Tuna-2 generates images by iteratively applying the flow matching head and Euler solver to denoise random noise in pixel space (786K dimensions) rather than in a compact latent space (~16K dimensions). The paper uses this design without analyzing the inference cost implications: how many denoising steps are required, what is the wall-clock time per generation, how does this compare to latent-space methods, and how does the cost scale with image resolution. The paper completely omits any discussion of inference latency, throughput, or computational cost for generation. The only cost metric discussed anywhere is training data scale (number of tokens, Figure 6) and training hardware (64 nodes, Section 3.1). The user-facing cost — how long does it take to generate an image at inference time — is absent from the paper entirely.

The consequence. This is a critical omission for deployment decisions. Pixel-space generation operates on a space that is approximately 48× larger than a typical VAE latent space (786K / 16K ≈ 49×). While the flow matching head's architecture and the Euler solver's computational cost per step are not inherently tied to this dimensionality (the transformer processes token sequences, not raw pixels directly), the flow matching head must produce predictions in pixel space, and the iterative denoising process must operate on full-resolution pixel tensors. Without latency measurements, a practitioner cannot evaluate whether Tuna-2's architectural simplicity and diversity advantages (Table 3) are worth potentially higher inference costs. If pixel-space generation requires more denoising steps or more computation per step than latent-space alternatives, the practical advantage in deployment may be negative even if quality is comparable. Additionally, the computational cost scales with image resolution: a 1024×1024 image has 4× the pixels of a 512×512 image, making high-resolution generation quadratically more expensive in the output space. The paper provides no analysis of this scaling behavior.

What evidence exists in the paper. Zero. The paper does not report inference time, FLOPs per generation, memory usage, number of Euler solver steps, or any other computational cost metric for inference. The reconstruction experiment (Table 5) uses the ImageNet validation set (which has standardized resolutions) and reports quality metrics only. The generation benchmarks (GenEval, DPG-Bench) follow standard evaluation protocols that do not require reporting inference cost. The entire "Experimental Analysis" section contains no timing or efficiency measurements.

Mitigation status. Not addressed. Unlike the training dynamics analysis (Section 3.3), which carefully compares training losses and data mixtures, there is no analogous analysis for inference dynamics. The paper does not discuss the number of denoising steps, does not ablate step count vs. quality (a standard analysis in the diffusion model literature), and does not mention inference cost as a consideration or limitation. This omission is particularly notable because the paper explicitly promotes architectural simplicity as an advantage — but simplicity of design does not guarantee simplicity of execution. A simpler architecture that requires more inference computation is not obviously preferable to a more complex architecture that generates images faster. Future work quantifying the inference cost and comparing it against latent-space methods would be essential for practical deployment decisions.

---

Limitation 5: The Masking Scheme's Effectiveness Is Plausibly Encoder-Specific, and the Paper's Central Regularization Technique May Not Transfer to Settings Without SigLIP 2's Pretraining

The assumption or constraint. The masking-based feature learning scheme is the paper's proposed solution to the challenge of learning robust visual representations in high-dimensional pixel space. The paper presents it as a general technique, but the evidence for its effectiveness comes with a significant caveat that the authors themselves identify: Tuna-2 benefits more from masking than Tuna-R because "SigLIP 2 itself is pretrained with a similar masked prediction objective" (Section 3.4). This means the differential benefit — and thus a substantial portion of Tuna-2's ability to close the gap with the encoder-based variant — may be attributable to the fact that the specific encoder used in the comparison (SigLIP 2 So400M) has already internalized some of the benefits that masking provides. Against an encoder without masked pretraining (e.g., a standard CLIP model, or a supervised ViT), Tuna-R might benefit more from masking, and Tuna-2's relative advantage might shrink or disappear. Conversely, against an even stronger encoder with more sophisticated pretraining, Tuna-2's masking advantage might be even larger.

The consequence. The paper's central regularization technique — masking — is evaluated against a baseline that partially nullifies its effect through pretraining. A practitioner choosing between an encoder-free and encoder-based design needs to know: is masking a general-purpose technique that helps all pixel-space models, or does its effectiveness depend on the absence of pretrained features? If masking primarily compensates for the lack of pretrained semantic structure (i.e., it helps the model learn what an encoder would have provided), then its utility is specific to encoder-free training and the technique is valuable but not generalizable. If masking provides benefits orthogonal to pretrained features (e.g., robustness to occlusion, better spatial reasoning), it would also improve encoder-based models with non-masked encoders. The paper's evidence cannot distinguish these cases because the only encoder tested (SigLIP 2) already incorporates masking. More broadly, the paper's claim that encoder-free designs are competitive relies partially on masking closing the performance gap — if masking's effectiveness is contingent on the specific encoder comparison, the broader claim about encoder-free viability is weakened.

What evidence exists in the paper. Table 6 provides the core evidence: masking improves both Tuna-R and Tuna-2, but Tuna-2 benefits substantially more (e.g., +4.2 on CountBench for Tuna-2 vs. +1.0 for Tuna-R). The paper explicitly attributes this differential to SigLIP 2's masked pretraining: "We hypothesize that this difference is related to the SigLIP 2 representation encoder used in Tuna-R, since SigLIP 2 itself is pretrained with a similar masked prediction objective." This is a self-aware observation, but the paper treats it as an explanation rather than as a limitation that calls the generality of the findings into question. No experiment tests Tuna-R with a non-masked encoder (e.g., original CLIP, EVA-CLIP, DFN-CLIP) to determine whether the differential masking benefit is truly encoder-specific or reflects a deeper property of encoder-based vs. encoder-free training.

Mitigation status. Partially addressed through transparency. The authors are explicit about the mechanism they believe causes the differential benefit, which allows readers to calibrate their interpretation. However, the paper does not test this hypothesis, does not mention it as a limitation on the generality of the masking results, and does not suggest experiments with different encoder types as future work. A more complete analysis would include at least one alternative encoder (e.g., a standard CLIP model without masked pretraining) in the masking ablation to determine whether masking's benefits generalize or are encoder-contingent. Without this, the masking scheme's status as a general contribution to pixel-space multimodal training — rather than a technique that works specifically when the alternative is SigLIP 2 — remains uncertain.

---

Limitation 6: The Training Data Mixture and Scale Are Fixed, and the Optimal Ratio Between Generation and Understanding Data Is Likely Not Portable Across Model Scales or Training Budgets

The assumption or constraint. The paper ablates the generation-to-understanding data ratio (Section 3.3, Figure 5) and identifies 7g3u as optimal. However, this ablation is performed at exactly one training scale (presumably the full 300K-step pretraining, though the exact number of steps used for the ablation is not specified), one model scale (7B), and with one specific data mixture (in-house image-text pairs plus Nemotron text data). The paper does not investigate whether the optimal ratio shifts with model size, total training budget, or data composition. The finding that "the MSE loss is more sensitive to changes in the sampling ratio, while the CE loss varies within a relatively smaller range" (Section 3.3) is a point observation — it describes the loss landscape at the tested configuration but provides no evidence about whether this sensitivity relationship holds at other scales.

The consequence. This is a practical limitation for anyone attempting to reproduce or scale Tuna-2. If a practitioner trains a larger model (e.g., 70B), uses a different LLM backbone, or has a different data distribution (e.g., more high-quality captioning data, less generation data), the 7g3u ratio may not be optimal. The paper's narrative implies that generation is the harder, more data-hungry task — but this might invert at very large model scales where the transformer has more capacity to learn the image manifold from fewer examples, or where understanding tasks become more nuanced and require more training data. Without scaling the ratio experiment across model sizes, the practitioner cannot determine whether to invest in a broader sweep or to trust the 7g3u finding as robust. More subtly, the finding that CE loss is insensitive to the ratio suggests that understanding performance is data-efficient — but this could be an artifact of the specific understanding tasks in the pretraining data (mostly captioning), which are relatively simple compared to the generation task. If a practitioner adds more complex understanding data (multi-hop reasoning, long-form VQA), the optimal ratio might shift toward more understanding examples.

What evidence exists in the paper. The evidence is strictly limited to Figure 5, which compares four ratios (8g2u, 7g3u, 5g5u, 3g7u) at a single scale with a single data mixture. The paper does not report the number of training steps, the total data volume, or the absolute performance numbers (only relative loss curves) for this ablation. The loss curves show clear monotonic relationships (more generation data → lower MSE; more understanding data → lower CE) but do not reveal whether these relationships would hold at different scales. The scaling analysis in Figure 6 uses the fixed 7g3u ratio without testing whether a different ratio would change the Tuna-R vs. Tuna-2 crossover point.

Mitigation status. Not addressed. The paper treats the 7g3u ratio as a fixed hyperparameter determined by the ablation and applies it uniformly to all subsequent experiments without further analysis. There is no discussion of how this ratio might interact with model scale, training duration, or data quality, and no suggestion that future work should investigate data mixture scaling laws. Given that the paper's central argument is about scalability ("a scalable path toward stronger visual representations"), the absence of any analysis of how the training recipe itself scales is a notable gap. A scaling-aware practitioner would want to know whether the optimal generation-to-understanding ratio is a function of model size (analogous to how the optimal batch size scales with compute in pretraining scaling laws) before committing to a specific mixture for a different-scale deployment.

7. Implications and Future Directions

How This Work Changes the Landscape

The magnitude: a reframing, not a paradigm shift — yet. Tuna-2 does not introduce a fundamentally new learning algorithm or a novel architectural primitive. What it does is provide the first systematic, controlled empirical evidence that pretrained vision encoders, the near-universal foundation of multimodal models, transition from being helpful scaffolding to being constraints as training proceeds. This is not an existence proof that encoder-free models work — Mono-InternVL and NEO had already demonstrated viability for understanding-only settings — but rather a causal demonstration, through the Tuna-R vs. Tuna-2 comparison, that removing the encoder produces better final representations given sufficient training. The magnitude of the contribution is therefore a reframing: it shifts the default stance in multimodal architecture design from "you should use a pretrained encoder unless you have a specific reason not to" toward "you should consider whether the encoder's constraints are worth its early-training benefits given your training budget and target tasks."

The distinction between an existence proof and a causal superiority claim matters. Prior encoder-free models showed that encoders aren't necessary — Tuna-2's controlled comparison shows they may be suboptimal at scale. This changes the burden of proof: rather than encoder-free researchers needing to justify why they depart from the standard architecture, encoder-based researchers should be prepared to justify what the encoder is contributing that couldn't be learned end-to-end with the same compute budget. The paper doesn't settle this question definitively — the evidence is at one scale, with one encoder, on one class of tasks — but it shifts the Overton window of acceptable architectural choices in multimodal research.

Reconciling contradictions in prior work. The paper provides a coherent explanation for a tension that existed in the literature: why do some studies find that encoder-based models excel at visual understanding (the entire LLaVA/Qwen-VL/InternVL lineage) while recent monolithic models (NEO, Mono-InternVL) claim competitive performance without encoders? Tuna-2's scaling curves (Figure 6) resolve this apparent contradiction by showing that both claims are correct — at different training scales. Encoder-based models are better early in training, which is what most prior ablation studies capture (since they tend to train for limited budgets). Encoder-free models catch up and surpass them later, which only becomes visible with sustained large-scale training. The field's conflicting findings were not due to methodological errors on either side but rather due to studying different points on a training curve that crosses. This is a clarifying contribution: it transforms a "who is right" debate into a "when does the crossover happen" question, which is more productive for guiding practical decisions and future research.

Research directions that become more attractive. The paper's findings make several lines of inquiry significantly more promising:

• End-to-end multimodal pretraining from pixels becomes a first-class research direction rather than a curiosity. Before Tuna-2, a proposal to build a unified model without vision encoders would face the reasonable objection that encoders are known to be helpful and there's no evidence removing them is beneficial. The paper provides that evidence, lowering the barrier to proposing and funding encoder-free research.

• Scaling laws for multimodal architectures become an urgent priority. The crossover pattern in Figure 6 — one architecture starting ahead but being surpassed — is exactly the kind of phenomenon that scaling laws are designed to characterize. If we could predict, for a given model size and training budget, when the encoder-free architecture overtakes the encoder-based one, architecture selection becomes an engineering optimization problem rather than a matter of intuition. This paper provides the first data point on such a curve but cannot extrapolate; the field now has both the motivation and the template for building full multimodal scaling laws.

• The relationship between pretraining objectives and downstream task requirements gains new importance. The paper argues that SigLIP 2's contrastive pretraining discards fine-grained visual information that pixel-centric tasks need. This suggests that the encoder's objective, not just its architecture or scale, determines whether it helps or hinders unified models. Research into encoders pretrained with objectives that preserve spatial detail (e.g., DINOv2's self-supervised dense features, or encoders trained jointly with generation-aware losses) could produce encoder-based models that don't exhibit the crossover — or alternatively, research could pursue encoder-free designs that never suffer the early-training lag. Both paths are illuminated by the paper's finding that the encoder's objective, not just its presence, drives the dynamics.

Research directions that become less attractive. The paper also casts doubt on certain approaches:

• Incremental improvements to VAE-based unified representations — the paper shows that removing the VAE entirely (Tuna-R vs. Tuna, Table 1 pixel-centric benchmarks) improves fine-grained understanding, and that removing the encoder as well (Tuna-2) produces further gains. This suggests that the VAE bottleneck is a genuine limitation, not an engineering inconvenience, and that continued refinement of VAE architectures for unified models may be a diminishing-returns investment compared to exploring pixel-space alternatives.

• Larger vision encoders as a default scaling strategy. If an encoder-free 7B model can match or exceed a 16B encoder-based model (Tuna-2 vs. Ming-UniVision in Table 1), the strategy of scaling the vision encoder to improve multimodal performance — a common approach in the InternVL and Qwen-VL lineages — may be misallocating parameters. The encoder's parameters could potentially be more effectively deployed as additional decoder capacity in an end-to-end architecture. The paper doesn't prove this (it compares different architectures, not a controlled parameter-count sweep), but it shifts the burden of proof onto advocates of ever-larger vision encoders to demonstrate that those parameters couldn't be better used elsewhere.

• Multi-stage training pipelines with separate alignment phases. Tuna-2's fully end-to-end training (no connector alignment stage) is presented as a practical advantage, and the results show no penalty for this simplicity. The field's default assumption that encoder-LLM alignment requires a dedicated training stage — common since LLaVA introduced the connector-pretraining step — may be an artifact of modular architectures rather than a fundamental requirement of multimodal learning.

Follow-Up Research This Work Enables

Scaling laws for the encoder-free crossover point across model sizes and training budgets. The paper's central finding — that Tuna-2 surpasses Tuna-R on understanding after sufficient training — is demonstrated at exactly one scale (7B parameters, 300K pretraining steps). The most pressing open question is: how does the crossover point shift with model size and training budget? A strong follow-up would train Tuna-R and Tuna-2 at multiple scales (e.g., 1B, 3B, 7B, 13B, 34B) and multiple training durations, mapping accuracy versus total FLOPs for each architecture on a fixed benchmark suite (including at minimum OCRBench, MMVP, V*, and GenEval to capture both understanding and generation dynamics). The analysis would fit parametric scaling laws of the form Error(N, D) = a × N^(-α) + b × D^(-β) + c for each architecture, where N is parameter count and D is training tokens, and identify whether the encoder-free architecture has a steeper scaling exponent (better scaling at large compute) counterbalanced by a higher constant factor (worse at small compute). This would directly address the paper's "scalable path" claim by quantifying the scalability. The training cost would be substantial (roughly a 10× increase over the current paper's compute for a multi-scale study), but the resulting scaling laws would be the single most impactful piece of information for the multimodal architecture community.

Testing the encoder-dependence hypothesis: repetition with non-masked encoders. The paper's masking-based feature learning scheme shows differential benefits — Tuna-2 benefits more than Tuna-R, which the authors attribute to SigLIP 2 already having been trained with a similar masked prediction objective (Section 3.4). This raises a critical confound: is the Tuna-2 advantage genuinely about encoder-free vs. encoder-based, or is it about SigLIP 2 specifically? A direct follow-up would replicate the Tuna-R vs. Tuna-2 comparison using three alternative encoders for Tuna-R: (1) a standard CLIP ViT-L/14 (no masked pretraining, purely contrastive), (2) a DINOv2 ViT-L/14 (self-supervised, dense features, no text alignment), and (3) an InternViT-300M (larger, stronger, trained with multiple objectives). The prediction: against CLIP, Tuna-R should benefit more from masking (since CLIP lacks masked pretraining) and the crossover might occur later or not at all; against DINOv2, Tuna-R might perform better on pixel-centric benchmarks (since DINOv2 preserves spatial detail) and the crossover might not occur on those tasks; against InternViT, Tuna-R's head start might be larger and the crossover delayed. If the Tuna-2 advantage persists across all encoders on understanding benchmarks, the claim that encoder-free designs are fundamentally better at scale is strengthened. If it disappears with certain encoders, the finding is encoder-specific and the paper's conclusions should be appropriately narrowed. This experiment is crucial because the paper's abstract-level claim is stated in general terms, not qualified by encoder type.

Pixel-space generation scaling: does the quality gap close, and at what cost? The paper shows Tuna-2 trailing Tuna-R on generation quality (Tables 2 and 3, Figure 6 GenEval panel) but the gap narrowing with training. The unanswered practical question is: how much additional training would Tuna-2 need to match Tuna-R's generation quality, and does that training cost outweigh the architectural simplicity benefits? A follow-up would train Tuna-2 for 2-4× longer than the current 300K steps (i.e., 600K-1.2M steps) while keeping Tuna-R at the current budget, measuring generation quality (GenEval, DPG-Bench, LLM-judge quality) at regular intervals. If Tuna-2's generation quality asymptotically approaches Tuna-R's — reaching, say, within 0.5 points on GenEval overall — then pixel-space generation is genuinely competitive and the quality gap is just a data-volume issue that can be solved with more compute. If Tuna-2's quality plateaus below Tuna-R's, there may be a fundamental limitation to pixel-space generation (perhaps the higher dimensional output space requires an impractically large model to match latent-space quality, or the optimization landscape has local minima that pixel-space methods get stuck in). This experiment would also measure inference cost (FLOPs per generation, wall-clock time) to quantify the end-to-end tradeoff. A variant of this experiment would vary the number of flow matching inference steps to build a quality-vs-cost Pareto frontier for both Tuna-2 and latent-space methods, enabling practitioners to choose their operating point based on latency constraints.

Dynamic compute allocation for pixel-space generation. Tuna-2 applies a fixed number of denoising steps to all generation prompts, but the complexity of text-to-image generation varies enormously across prompts. A natural extension — inspired by the compute-optimal scaling paper in the reference example — would be to develop a difficulty-conditioned denoising schedule: use a lightweight predictor (perhaps a small probe on the transformer's early denoising steps, or a CLIP-score-based estimate of prompt complexity) to decide how many Euler solver steps to allocate per prompt. Easy prompts ("a red circle") might need only 5-10 steps, while complex prompts ("a bustling Renaissance marketplace...") might benefit from 50+ steps. A strong experiment would: (1) train a difficulty predictor on Tuna-2's intermediate denoising states, (2) evaluate the accuracy-vs-compute tradeoff curve on a held-out set of prompts with human-annotated complexity scores, and (3) compare against the uniform-allocation baseline, measuring whether adaptive allocation achieves equivalent generation quality with, say, 30% fewer total inference FLOPs. This would directly address the inference cost limitation identified in Section 6 (Limitation 2) and would make Tuna-2 more practical for deployment scenarios where generation latency matters.

Information-theoretic analysis of what pretrained encoders discard. The paper claims that pretrained encoders lose "fine-grained low-level visual details" (Section 2.1) and that this causes the performance gap on pixel-centric benchmarks (Table 1). This claim is inferred from benchmark results but never directly measured. A mechanistic follow-up would train linear probes to decode specific visual attributes from intermediate representations in Tuna-R (post-encoder, post-connector, and mid-decoder) and Tuna-2 (post-patchify and mid-decoder), measuring how much information about fine-grained attributes is preserved at each stage. The attributes to probe: object count (1-10 objects), spatial position (x,y coordinates of target objects, tested on synthetic data with controlled positions), small object presence (binary: is there an object smaller than 1% of image area?), color of specific regions, and text content (for OCR). The prediction: Tuna-R's encoder representations should show significantly lower mutual information with these attributes than Tuna-2's early representations, with the gap narrowing or inverting at later decoder layers as the model compensates. This would provide causal evidence for the paper's information-bottleneck hypothesis and would identify where in the architecture the information loss occurs, potentially enabling targeted interventions (e.g., skip connections that bypass the encoder's pooling layers for fine-grained tasks).

Unified model self-improvement via Tuna-2's own generations. The paper demonstrates that Tuna-2 performs both understanding and generation in a single architecture with a unified pixel-space representation. This enables a self-improvement loop that is architecturally impossible in decoupled models: use Tuna-2 to generate diverse training images for itself, then train on its own generations to improve understanding. Concretely: (1) take the Tuna-2 checkpoint, (2) use it to generate images from a diverse set of prompts (including prompts designed to test fine-grained visual skills — e.g., "a photo of exactly seven red apples and three green pears on a wooden table"), (3) automatically verify the generations using the model's own understanding capability (ask "how many red apples are in this image?" and check against the prompt specification), (4) filter to high-quality, verified image-text pairs, (5) continue training Tuna-2 on this self-generated data for additional steps, and (6) measure whether understanding benchmarks (particularly pixel-centric ones) improve from this self-supervised data. This experiment would test whether the unified representation enables a virtuous cycle where generation and understanding reinforce each other — a possibility that the paper's architecture enables but doesn't explore. The hypothesis is that generating diverse, challenging images and verifying them with the understanding head forces the model to refine its internal representations in ways that pure supervised training on static data doesn't. This is the multimodal analog of self-play in game-playing AI or self-training in semi-supervised learning.

Practical Applications and Downstream Use Cases

Fine-grained visual inspection and quality control. The paper's strongest and most consistent result is Tuna-2's superiority on pixel-centric benchmarks: CountBench (81.7, highest among all 7-9B models in Table 1), V* (59.2, requiring guided visual search for small objects), and VisuLogic (28.8, testing visual reasoning over fine details). These benchmarks directly map to industrial inspection tasks — counting items in a bin, detecting small defects on manufactured parts, verifying the presence and correct placement of components on a circuit board. A deployment scenario would use Tuna-2 in a visual inspection pipeline where: (1) a camera captures high-resolution images of products, (2) Tuna-2 is fine-tuned on a modest set of in-domain inspection examples (leveraging the strong pixel-space representations learned during pretraining, which should transfer well to fine-grained tasks), and (3) the model both answers inspection queries ("Is there a scratch on the left edge of the component?") and, when defects are found, generates annotated images highlighting the defect location (using the generation capability to produce visual explanations). The competitive advantage over encoder-based systems is the superior preservation of fine visual details — in an inspection setting where missing a small defect has high cost, Tuna-2's CountBench advantage (+4.3 points over the next-best 7B UMM, Show-o2 at 63.5) represents a meaningful reduction in missed detections.

On-device multimodal assistants with variable-resolution understanding. Tuna-2's encoder-free design removes the fixed-resolution constraint that limits encoder-based models. A SigLIP 2 encoder is pretrained at a specific resolution (typically 384×384), and while techniques like dynamic high-resolution adaptation exist (processing an image as multiple crops), they are band-aids that add complexity and compute. Tuna-2, operating directly on raw pixel patches, can natively process images at whatever resolution is provided (within sequence length limits), without architectural changes. This enables a deployment scenario on mobile devices where: (1) the base model runs on-device (the 7B parameter count is within the range of on-device deployment with quantization, especially as mobile NPUs improve), (2) for routine tasks (general VQA, captioning), the camera captures at moderate resolution (e.g., 512×512, producing 1024 visual tokens), and (3) for tasks requiring fine-grained perception (reading small text, identifying distant objects), the user zooms in or captures at higher resolution, and the same model processes the higher-resolution input directly (e.g., 1024×1024, producing 4096 visual tokens) — at higher computational cost, but with no architectural change, no additional training, and no loss of fidelity from downsampling to a fixed encoder resolution. The paper shows that Tuna-2 excels at precisely these fine-grained tasks (OCRBench 79.7, ChartQA 85.6), making the variable-resolution capability practically valuable rather than merely architecturally elegant.

Training data generation for specialist vision models. Tuna-2's combination of understanding and generation with strong diversity (Table 3: 48.4% diversity win rate under GPT-5.4) makes it a compelling data engine for training specialist vision models. The scenario: a team needs to train a specialized classifier or detector for a domain with limited real data (e.g., rare bird species identification, specific industrial defect categories, unusual traffic sign configurations). Using Tuna-2: (1) a domain expert writes a diverse set of text prompts describing the target concepts with controlled variation ("a photo of a Himalayan monal pheasant in snow, side view, natural lighting"), (2) Tuna-2 generates a large set of diverse training images (the diversity advantage means fewer mode-collapse issues than with typical generation models), (3) Tuna-2's understanding capability is used to automatically filter the generated images — the model answers verification questions about each generated image ("What species of bird is shown?" "Is the background snowy?" "Is the view from the side?") and discards images that don't match the specification, and (4) the filtered images are used to train the specialist model. The advantage over using a dedicated generation model (like FLUX or SD3) is the integrated verification step: a separate generation model plus a separate understanding model would require running two different systems, with potential representation mismatch between what the generator produces and what the verifier can reliably evaluate. Tuna-2's unified representation ensures that the same visual features used to generate the image are used to verify it, potentially improving the reliability of the filtering step.

When to Prefer This Method

The paper articulates a clear tradeoff between encoder-free (Tuna-2) and encoder-based (Tuna-R) designs, grounded in the scaling curves of Figure 6 and the benchmark results in Tables 1-4. The decision can be framed as a function of the primary task, training budget, and deployment constraints:

• Prefer Tuna-2's encoder-free design when: (1) fine-grained visual perception is the primary capability (CountBench, V*, VisuLogic tasks, or industrial inspection), since Tuna-2 achieves the strongest results on these benchmarks among all UMMs in Table 1; (2) training budgets are large enough to pass the crossover point — if you're planning sustained large-scale pretraining (hundreds of thousands of steps at 7B+ scale), the encoder's early-training advantage is transient, and the encoder-free design's superior asymptote on understanding will dominate; (3) architectural simplicity and end-to-end training are engineering priorities — Tuna-2 eliminates the connector alignment stage, the VAE, and the encoder, reducing the number of separately trained components from three (encoder, connector, VAE) to zero beyond the patch embedding; and (4) output diversity matters more than peak per-image quality — Table 3 shows Tuna-2 winning on diversity by substantial margins (48.4% vs. 30.9% for Tuna-R under GPT-5.4), making it preferable for applications like data augmentation or creative tools where varied outputs from the same prompt are valuable.

• Prefer Tuna-R's encoder-based design when: (1) per-image generation quality is the paramount concern — Tuna-R leads on GenEval (0.88 vs. 0.87), DPG-Bench (86.35 vs. 86.54), and LLM-judge quality metrics (35.7% vs. 32.1% win rate under GPT-5.4; Table 3), with the consistency of this advantage across metrics suggesting it's a real, if small, effect; (2) image editing fidelity is critical — Tuna-R achieves 4.18 vs. Tuna-2's 4.09 on ImgEdit total score (Table 4), with consistent advantages across most editing operations; (3) training budgets are limited — if you cannot afford the 300K-step pretraining budget where Tuna-2 catches up on understanding (Figure 6), the encoder's early-training advantage means Tuna-R will achieve better performance at lower total compute; and (4) you already have a strong pretrained encoder available and want to leverage that investment — Tuna-R's design reuses existing encoder infrastructure (SigLIP 2 in this case) without requiring the encoder to be retrained from scratch, which may be more practical if a high-quality encoder is already deployed in your organization.

These recommendations are bounded by the paper's evidence: they apply to 7B-scale models with Qwen2.5-style LLM decoders, SigLIP 2-style encoders (for Tuna-R), and training budgets in the range tested (300K pretraining steps on 550M image-text pairs). At substantially different scales, the crossover point may shift, and the relative advantages may invert — the paper provides the template for making this decision at any scale but not the data to instantiate it beyond the tested configuration.