Image Generators are Generalist Vision Learners

ArXiv: 2604.20329

🎯 Pitch

A text-to-image generator, instruction-tuned to output color-coded visualizations instead of pixels, matches or beats specialist models like SAM 3 and Depth Anything 3 on segmentation and depth estimation—without losing its ability to generate realistic images. This suggests that the representations learned through generative pretraining already contain state-of-the-art visual understanding, which can be unlocked by simply reformatting perception tasks as image-generation problems.

1. Executive Summary

This paper demonstrates that image generation pretraining serves as a foundational mechanism for learning general-purpose visual representations, introducing Vision Banana, a generalist vision model built by instruction-tuning the text-to-image generator Nano Banana Pro on a mixture of its original generation data alongside a small amount of vision task data formatted as invertible RGB visualizations (e.g., segmenting objects by coloring them with specified hex values, or encoding metric depth via a piecewise-linear traversal of the RGB cube). By parameterizing the output space of 2D and 3D perception tasks as images, the approach achieves state-of-the-art results across both visual understanding and generation from a single unified model—surpassing domain specialists like Segment Anything Model 3 on referring expression segmentation (0.738 cIoU vs. 0.734 on RefCOCOg), beating Depth Anything 3 on monocular metric depth estimation (average δ1 of 0.929 vs. 0.918 across four datasets), and matching Nano Banana Pro on text-to-image generation (53.5% win rate), establishing that the internal representations learned through generative pretraining already encode powerful understanding capabilities that require only lightweight instruction-tuning to unlock, without sacrificing the model's original generation fidelity.

2. Context and Motivation

The Core Problem: The Disconnect Between Generation and Understanding in Vision Models

The fundamental question this paper tackles cuts to the heart of how computer vision models are trained and what they learn: do models trained solely to generate images secretly develop rich, general-purpose visual understanding capabilities, and if so, can those capabilities be unlocked without sacrificing generation quality?

This question matters because the field of computer vision has historically treated visual generation and visual understanding as separate problems requiring fundamentally different approaches. Visual understanding—tasks like segmentation, depth estimation, and surface normal prediction—has been dominated by discriminative training paradigms: supervised learning with task-specific architectures (Dehghani et al., 2023; Dosovitskiy et al., 2020), contrastive learning (Radford et al., 2021; He et al., 2020; Chen et al., 2020b), self-supervised bootstrapping (Caron et al., 2021; Grill et al., 2020), and masked autoencoding (He et al., 2022; Bao et al., 2021). These approaches explicitly optimize models to extract features useful for downstream classification or dense prediction tasks. Meanwhile, visual generation—producing high-fidelity images from text prompts—has been developed and evaluated almost entirely independently, with metrics focused on photorealism, prompt adherence, and aesthetic quality rather than the model's implicit understanding of scene geometry, object identity, or spatial relationships.

This separation is both an intellectual and a practical inefficiency. It implies that building a state-of-the-art general vision system requires training separate models for generation and understanding, or at minimum requires a two-stage pipeline where a pretrained backbone (often discriminatively trained) is fine-tuned for generation or understanding. The paper challenges this assumption directly: if image generation pretraining naturally induces robust visual representations, then a single model could serve both purposes, dramatically simplifying the landscape of visual AI and potentially enabling a unified "omni" model analogous to how large language models unify diverse text-based tasks under a single generative interface.

Why This Problem Matters: Practical and Theoretical Significance

The paper's framing of this question carries implications on multiple levels:

Paradigm-level significance for the vision community. The natural language processing field underwent a dramatic paradigm shift when it was discovered that generative pretraining (next-token prediction) produces models with emergent reasoning, translation, summarization, and coding capabilities—capabilities that were not explicitly trained for but emerged as a byproduct of learning to model the distribution of text (Brown et al., 2020; Chowdhery et al., 2023). This insight reorganized the entire NLP field around a unified generative paradigm. The paper explicitly analogizes its work to this moment in NLP history, asking whether computer vision is poised for an equivalent transformation:

"In natural language processing (NLP), generative pretraining is performed to produce base models, often referred to as LLMs, that are good at generating text, whereas instruction-tuning guides them to follow specific tasks and produce text in requested formats and stay on the task. Analogously, we position a visual generative model as a 'base' model and perform instruction-tuning to align the model to produce visual output in desired formats, in accordance with the prompts." (Section 1)

If this analogy holds—if image generation pretraining is to vision what language modeling pretraining is to NLP—then the entire field's research agenda should reorient toward generative vision pretraining as the central mechanism for building foundational vision models. The paper's results represent evidence for or against this hypothesis.

Practical deployment implications. Currently, deploying a system that performs segmentation, depth estimation, and surface normal prediction requires integrating multiple specialist models, each with its own architecture, training pipeline, and inference stack. A single model that handles all these tasks—plus image generation and editing—dramatically reduces engineering complexity, maintenance burden, and deployment footprint. This is not merely an academic convenience; it has direct implications for resource-constrained environments (mobile devices, robotics platforms) where running multiple large models is infeasible.

Economic and compute-efficiency implications. Training a generalist model that excels at both generation and understanding means that the massive compute expenditure of pretraining a generative model (Nano Banana Pro, in this paper's case) yields returns on two previously separate categories of tasks. Rather than spending additional compute on discriminative pretraining for understanding, the generative pretraining compute is amortized across both generation and understanding. The paper quantifies this: Vision Banana achieves state-of-the-art understanding results using only "lightweight instruction-tuning" on "a small amount" of vision task data (Section 2), implying that the heavy lifting was already done by the generative pretraining.

Theoretical significance for representation learning. The central scientific question is whether learning to model $p(x)$ (the data distribution, via generation) implicitly requires learning useful features for predicting $p(y|x)$ (conditional task outputs, via discriminative modeling). This is a long-standing conjecture (Ranzato et al., 2011; Chen et al., 2016) that has been difficult to verify at scale because generative models historically produced lower-quality features than purpose-built discriminative models. The recent emergence of high-fidelity image generators (Black Forest Labs, 2025; ByteDance, 2026; Google, 2025a,b; OpenAI, 2026) makes it newly possible to test whether generative models that are genuinely good at their primary task (producing convincing images) have incidentally learned representations that rival or exceed those from discriminative pretraining. The paper's results provide strong evidence supporting this hypothesis at the scale of modern foundation models.

Prior Approaches and Where They Fall Short

The paper identifies a landscape of prior work that can be organized into several categories, each with specific limitations that Vision Banana addresses:

Category 1: Specialist Discriminative Models (The Dominant Paradigm)

The state of the art on nearly every visual understanding benchmark has been held by models explicitly designed and trained for that specific task. For segmentation, this includes the Segment Anything series (Kirillov et al., 2023; Ravi et al., 2024; Carion et al., 2025), DINO-X (Ren et al., 2024), and various methods evaluated in Tables 2a–2d. For monocular depth estimation, it includes Depth Anything V3 (Lin et al., 2025), Depth Pro (Bochkovskii et al., 2024), MoGe-2 (Wang et al., 2025c), UniK3D (Piccinelli et al., 2025a), and others compared in Table 3. For surface normal estimation, it includes Lotus-2 (He et al., 2025), StableNormal (Ye et al., 2024), DSINE (Bae and Davison, 2024), and Marigold (Ke et al., 2024).

What falls short: These models represent a fragmented landscape. Each achieves excellent performance on its target task but requires (1) a dedicated architecture, often incorporating task-specific inductive biases (e.g., camera intrinsic conditioning for depth models, mask decoding heads for segmentation models); (2) task-specific training losses (e.g., scale-invariant depth losses, per-pixel cross-entropy for segmentation); (3) large amounts of task-specific training data, often with expensive human annotations; and (4) separate deployment and maintenance. Critically, these models are not generalist—a model trained for segmentation cannot perform depth estimation, and vice versa. The paper notes that many of these methods also require camera intrinsic parameters during training or inference (DepthLM, Depth Anything V3, Depth Pro, UniK3D, MoGe-2 all require intrinsics at training time, per Table 3), which restricts their applicability to in-the-wild images where intrinsics are unknown.

Category 2: Early Generative Representation Learning (Promising But Sub-SOTA)

A body of work has investigated whether generative models learn useful features for understanding tasks. Early efforts include generative pretraining from pixels (Chen et al., 2020a), feature extraction from GANs (Bhattad et al., 2023; Baranchuk et al., 2021), zero-shot classification with diffusion models (Clark and Jaini, 2023; Li et al., 2023), and using diffusion features for semantic correspondence (Hedlin et al., 2023; Tang et al., 2023; Zhang et al., 2023b; Li et al., 2024b). Sequential modeling approaches like Bai et al. (2024) have shown promising scaling behavior.

What falls short: The paper acknowledges these efforts but notes that "their effectiveness has lagged behind non-generative models" (Section 1). These methods demonstrated that generative models possess some understanding capabilities, but did not achieve performance competitive with state-of-the-art discriminative specialists. The gap was substantial enough that the field continued to treat discriminative pretraining as the primary path to visual understanding.

Category 3: Emergent Zero-Shot Behaviors (Hinting at Capability But Not Benchmarked)

Most directly motivating this paper, recent work has observed that state-of-the-art image and video generators can be prompted to produce outputs that resemble computer vision visualizations—segmentation maps, depth maps, surface normal maps (Wiedemer et al., 2025; Zuo et al., 2025). For example, a user can prompt Nano Banana Pro with "generate a depth map of this image" and receive a plausible-looking depth visualization.

What falls short: Despite generating plausible-looking outputs, these models "do not provide state-of-the-art results on modern benchmarks. This is partially because these models do not strictly follow the prompts to produce vision outputs in the desired formats that can be decoded back to vision outputs for computing quantitative metrics" (Section 1). In other words, the zero-shot generation might produce a depth map that looks correct to a human eye but uses an unknown color-to-depth mapping, has inconsistent scale, omits fine details, or otherwise fails to align with the precise invertible format required for quantitative evaluation against ground truth. The capability is latent but not reliably extractable for benchmarking—exactly the gap that instruction-tuning aims to close.

Category 4: Fine-Tuning Generative Models for Understanding (Compromising Generality for Performance)

A more recent line of work adapts generative models for specific understanding tasks by adding specialized modules and performing full fine-tuning—approaches like InstructCV (Gan et al., 2023), Marigold (Ke et al., 2024), Diception (Zhao et al., 2025), and others (Garcia et al., 2025; Wang et al., 2026b). These methods take a pretrained generative model and modify it with task-specific heads, decoders, or other architectural additions, then fine-tune the entire model on task-specific data.

What falls short: While these methods successfully achieve near-SOTA or SOTA performance on their target tasks, they "sacrifice the model's generality across other understanding and generation tasks" (Section 1). The full fine-tuning process overwrites the original generative capabilities, meaning the resulting model can no longer perform image generation or other understanding tasks. The paper explicitly positions itself against this tradeoff:

"Such strategy differentiates our work from previous works that perform full finetuning to generative models without image generation data" (Section 2)

Vision Banana's key differentiator is that it achieves SOTA understanding results while retaining the base model's generation capabilities, verified through quantitative human evaluations (53.5% win rate on GenAI-Bench, 47.8% on ImgEdit).

How This Paper Positions Itself

Vision Banana positions itself at the intersection of two observations that, taken together, suggest a new paradigm:

Observation 1: Image generators have emerged to a quality level where their outputs are convincing enough for many practical applications. This emergence is recent—the paper cites generators from 2025–2026 (Black Forest Labs FLUX.2, ByteDance Seedance 2.0, Google Nano Banana Pro and Veo 3, OpenAI GPT-Image-1.5) as evidence that the generation quality threshold has been crossed.

Observation 2: These high-quality generators exhibit latent understanding capabilities when prompted appropriately (Wiedemer et al., 2025; Zuo et al., 2025), but these capabilities are not reliably extractable for formal evaluation.

The paper's thesis is that the gap between Observation 2 and SOTA benchmark performance is not a capability gap but an alignment gap. The model already possesses the visual understanding; it simply needs to be taught how to format that understanding as precise, invertible RGB visualizations that follow prescribed conventions. This is the "instruction-tuning" step—analogous to how LLMs are pretrained on next-token prediction and then instruction-tuned to follow specific task formats without fundamentally learning new capabilities, just learning to express existing capabilities in controlled ways.

The paper explicitly analogizes this to the LLM paradigm shift via the quote in Section 1 about generative pretraining followed by instruction-tuning. In this analogy:

Nano Banana Pro is the "base model" (analogous to a pretrained LLM like GPT-3)
Vision task data formatted as invertible RGB images is the "instruction tuning data" (analogous to the instruction-following datasets used for models like InstructGPT)
Vision Banana is the "instruction-tuned model" that can now perform vision tasks reliably while retaining its generation capabilities

The paper discriminates itself from prior work along several axes:

vs. Zero-shot prompting (Wiedemer et al., 2025; Zuo et al., 2025): Adds instruction-tuning to make the outputs precisely follow invertible color conventions, enabling quantitative benchmarking.
vs. Full fine-tuning with task-specific heads (Ke et al., 2024; Zhao et al., 2025; Gan et al., 2023): Avoids architectural modifications and full fine-tuning; preserves generation capabilities; uses the same RGB output space for all tasks.
vs. Training specialist models from scratch (Carion et al., 2025; Lin et al., 2025): Demonstrates that generative pretraining alone provides representations competitive with or superior to purpose-built discriminative training.
vs. Discriminative pretraining for vision (Radford et al., 2021; Oquab et al., 2023): Argues that generative pretraining, not contrastive or self-supervised learning, should be the central mechanism for building foundational vision models.

The paper's contribution is thus not a new architecture or a novel training algorithm, but rather a demonstration of a capability that was hypothesized but not proven: that a generative model trained without explicit discriminative objectives can, with minimal adaptation, match or exceed specialist models across diverse vision tasks. The significance lies in what this demonstration implies about the nature of visual representation learning—that learning to generate images is itself a sufficient training signal for acquiring general visual understanding.

The paper also positions itself as answering a specific form of the question "are generative models secretly generalist learners?" by establishing rigorous quantitative evidence. Prior work answered this question qualitatively ("yes, they show some understanding") or for narrow tasks. This paper answers it quantitatively across six standard benchmarks with metrics (mIoU, cIoU, gIoU, pmF1, δ1, AbsRel, mean/median angular error) that allow direct comparison against the best specialist models, under the strict zero-shot transfer setting where no training data from evaluation benchmarks is used—a protocol that tests genuine generalization rather than memorization.

3. Technical Approach

3.1 Reader Orientation

The system being built is Vision Banana, a single neural network that can accept a text prompt plus an input image and produce an output image, where that output image can be either a creative visual (like a text-to-image generation or an image edit) or a precise, invertible visualization of a computer vision task (like a segmentation mask or a depth map) that can be decoded back into a measurable prediction for benchmarking against ground truth. The core problem it solves is that standard image generators like Nano Banana Pro already possess latent visual understanding (they can generate plausible-looking depth maps if prompted), but they do not follow precise color-mapping conventions that would make those outputs quantifiably evaluable — the solution is a lightweight instruction-tuning procedure that teaches the model to produce vision-task outputs using strictly defined, invertible mappings from semantic or geometric values to RGB colors, enabling a single unified model to achieve state-of-the-art results on both visual understanding benchmarks and visual generation benchmarks without task-specific architectures or losses.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four conceptual components, although they are implemented within a single neural network:

Pretrained Image Generator (Nano Banana Pro): A large text-to-image diffusion model trained on massive web-scale data. It serves as the "base model," providing all prior knowledge about object appearance, scene geometry, spatial relationships, and semantic categories. It accepts a text prompt and optionally an input image, and produces an output RGB image.
Vision Task Data Constructor: A data preprocessing pipeline that converts ground-truth vision task outputs (segmentation masks, depth maps, surface normals) into RGB training target images using strictly invertible color-to-value mappings. Each task uses a different mapping designed to be both human-interpretable and mathematically invertible for quantitative evaluation.
Instruction-Tuning Mixture: The training procedure that mixes the vision task data into Nano Banana Pro's original training data at a low ratio. The model is fine-tuned on this mixture, learning to follow prompts that specify the desired vision task and the precise color mapping to use, without forgetting its original generation capabilities.
Task-Specific Decoders (Inference-Time): A set of lightweight post-processing routines that convert the model's generated RGB images back into task-specific outputs. For segmentation, this involves clustering pixels by color and mapping colors to class labels. For depth, it involves projecting RGB values back onto the edges of the RGB cube and inverting a power transform. For surface normals, it involves a direct linear mapping from RGB to vector components.

Information flows as follows: an input image and a task-specifying prompt enter the system → the instruction-tuned model generates an RGB image following the specified color convention → the task-specific decoder extracts the vision output → the output is compared against ground truth for evaluation.

3.3 Roadmap for the Deep Dive

First, the instruction-tuning strategy itself — how the training data mixture is constructed, why vision task data is mixed at a low ratio, and how this preserves generation capabilities — because this is the core mechanism that transforms a generator into a generalist vision model.
Second, the conceptual framework that unifies all tasks: encoding vision outputs as invertible RGB images, including the prompting strategy and the general approach to color mapping — because this is the universal interface that makes the system work across diverse tasks.
Third, the depth estimation encoding pipeline — the most mathematically involved mapping, involving a power transform and a piecewise-linear traversal of the RGB cube — because it exemplifies the careful design required to make generation outputs quantitatively evaluable.
Fourth, the segmentation encoding strategy, including both semantic (class-level) and instance-level coloring, and how unknown instance counts are handled — because segmentation presents unique challenges (open-vocabulary classes, variable numbers of instances) that require task-specific prompting strategies.
Fifth, the surface normal encoding, which is the simplest mapping because surface normals are already 3D vectors naturally aligned with RGB channels — this rounds out the 3D understanding tasks.
Sixth, the post-processing decoders that recover vision outputs from generated RGB images, including the critical inversion of the depth mapping — because without these decoders, the generated outputs cannot be quantitatively evaluated.
Seventh, the training data sources, evaluation protocol, and how the zero-shot transfer setting is maintained — because these implementation details determine whether the results reflect genuine generalization.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a methodology and empirical demonstration paper whose core idea is that state-of-the-art image generators already possess latent visual understanding capabilities, and that these capabilities can be unlocked for quantitative benchmarking by instruction-tuning the model to produce precisely formatted, invertible RGB visualizations of vision task outputs, without modifying the architecture or sacrificing generation quality.

Instruction-Tuning Strategy: Mixing Vision Data with Generation Data

The central training procedure is straightforward in concept but carries specific design choices that are critical to the paper's claims. The goal is to teach Nano Banana Pro to perform vision tasks while preserving its original image generation capabilities. The paper describes this as a "lightweight instruction-tuning" that "mixes vision task data into Nano Banana Pro's own training mixture at a very low ratio" (Section 2).

The training data mixture. Rather than fine-tuning the model exclusively on vision task data — which would cause catastrophic forgetting of generation capabilities — the authors combine two data sources during training:

Nano Banana Pro's original image generation data: This includes the original text-to-image pairs and any other data used to train the base model. By keeping this data in the training mixture, the model continues to see examples that require creative generation, preserving the learned weights responsible for photorealistic synthesis, compositional reasoning, and text-to-image alignment.
Vision task data formatted as RGB images: This is new data constructed by the authors, where each example consists of an input image, a text prompt specifying the vision task and color mapping, and a target RGB image that is the ground-truth vision output rendered using the specified invertible color scheme. The target image is what the model should generate; during training, the model's output is compared against this target.

The "very low ratio" of vision task data is not given as a specific percentage in the paper, but the concept is that the vision data serves as an instruction signal — it teaches the model how to format its outputs, not what visual understanding is. Since the base model already possesses the understanding capability (the paper's central hypothesis), the instruction-tuning merely aligns the model's generation behavior to follow precise color conventions. This is analogous to how LLM instruction-tuning uses relatively small datasets (compared to pretraining) to teach the model to follow specific output formats without teaching new facts or reasoning capabilities.

Why mixing is essential (preserving generation). The paper validates that generation capabilities are preserved through quantitative human evaluation: on text-to-image generation (GenAI-Bench), Vision Banana achieves a 53.5% win rate against Nano Banana Pro, and on image editing (ImgEdit), it achieves a 47.8% win rate (Section 2). A win rate near 50% indicates that human evaluators cannot reliably distinguish the instruction-tuned model from the base model on generation tasks — exactly what would be expected if the instruction-tuning did not degrade the model's original capabilities. The paper also shows qualitative comparisons in Figures 9 and 10 (Appendix), where Vision Banana and Nano Banana Pro produce highly similar outputs for the same text-to-image and image-editing prompts.

This contrasts with prior work that performs "full finetuning to generative models without image generation data" (Section 2), such as InstructCV (Gan et al., 2023), Marigold (Ke et al., 2024), and Diception (Zhao et al., 2025). Those methods achieve strong performance on their target vision tasks but lose the ability to generate creative images. Vision Banana's simultaneous achievement of SOTA understanding and preserved generation is the paper's key methodological contribution — it demonstrates that the understanding is emergent from the pretraining, not learned from scratch during fine-tuning.

Training data sources. The vision task training data is constructed from two categories:

For 2D tasks (segmentation): The paper uses "in-house model annotations for web-crawled 2D images" (Section 2). This means the authors used existing model-based annotation pipelines (likely segmentation models similar to SAM) to produce pseudo-ground-truth masks for a large collection of web images, rather than relying on expensive human annotations. The specific models and annotation procedures are not detailed, but the key claim is that these annotations provide sufficient training signal for instruction-tuning without requiring human-labeled data.
For 3D tasks (depth, surface normals): The paper uses "synthetic data from rendering engines" (Section 2). This is a critical design choice because synthetic data provides perfect, noise-free ground truth for 3D geometry — every pixel's depth and surface normal is known exactly in a rendering engine, eliminating annotation error. The use of synthetic data also makes it straightforward to exclude any real-world benchmark datasets from the training data, maintaining the zero-shot transfer protocol.

Crucially, no training data from any evaluation benchmark is included in the instruction-tuning mixture. The paper states this explicitly: "Crucially, no training data from our evaluation benchmarks is included in the instruction-tuning mixture, ensuring that our results reflect true generalist capability" (Section 2). This zero-shot transfer protocol — the same one used by Segment Anything (Kirillov et al., 2023) and CLIP (Radford et al., 2021) — means the model has never seen the specific images or annotation styles of the test datasets during training. Any strong performance on these benchmarks must therefore come from generalization, not memorization.

The Universal Interface: Vision Task Outputs as Invertible RGB Images

The paper's single most important design decision is to parameterize the output space of all vision tasks as RGB images rather than as structured outputs (e.g., class probability vectors, coordinate lists, or depth value arrays). This has several consequences that collectively enable the approach:

Consequence 1: No architectural modifications. Because both creative generation and vision task outputs are RGB images, the model architecture does not need to be changed. There is no task-specific decoder head, no separate output branch, no additional layers. The exact same network that generates a photograph of a cat also generates a segmentation mask — the only difference is the prompt. This contrasts with methods like Diception (Zhao et al., 2025) or Marigold (Ke et al., 2024), which add specialized modules to diffusion models for vision tasks.

Consequence 2: Shared representation learning. During training, the model's internal representations are optimized simultaneously for tasks as diverse as "generate a realistic image of a sunset" and "produce a depth map of this indoor scene using this specific color mapping." The gradients from both types of tasks flow through the same weights, potentially creating synergistic representations where geometric understanding learned for depth estimation benefits creative generation (e.g., generating images with more realistic spatial layouts) and semantic understanding learned for segmentation benefits generation (e.g., generating images where objects have coherent boundaries and identities).

Consequence 3: The prompting interface is natural language. The model receives a text prompt that specifies both (a) what task to perform and (b) the precise color mapping to use. For example, a semantic segmentation prompt might be:

"Generate a semantic segmentation visualization image, using this color mapping: {"cat": "red", "lock": "pink", "exit sign": "light purple", "background": yellow}."

This is just natural language with some structured notation embedded in it. The model, pretrained on text-to-image generation, already understands how to follow natural language instructions about image content. The instruction-tuning teaches it to additionally follow instructions about color conventions — but the fundamental mechanism (text-conditional image generation) is unchanged.

Consequence 4: The output is human-interpretable. A generated segmentation mask or depth map is directly viewable as an image, which means human annotators can qualitatively assess the model's outputs without any decoding step. This is valuable for debugging and for the qualitative examples throughout the paper. The model is generating images that look like visualizations a human would produce.

Consequence 5: Evaluation requires decoding. For quantitative benchmarking, the generated RGB image must be converted back into a task-specific output (pixel-wise class labels, metric depth values, or surface normal vectors). This requires a task-specific post-processing decoder that inverts the color mapping defined in the prompt. The decoder is not learned; it is a deterministic mathematical procedure based on the known encoding scheme. If the model perfectly follows the prompt's color convention, the decoding is exact. If the model generates colors that deviate from the specification (e.g., slightly wrong RGB values for a class), the decoding must handle this noise — typically through nearest-neighbor matching or thresholding.

This creates an inherent tension: the prompting must specify a color mapping that is simultaneously (a) easy for the model to follow (colors are distinct and natural to generate), (b) invertible without ambiguity (each vision output value maps to a unique color, and the inverse mapping is robust to small generation errors), and (c) covering the full range of possible output values (for continuous quantities like depth). Different tasks require different tradeoffs along these dimensions, which is why the paper develops task-specific encoding schemes rather than a single universal mapping.

Metric Depth Estimation Encoding: The Power Transform + RGB Cube Walk

This is the most mathematically involved encoding scheme in the paper, and it deserves detailed treatment because it exemplifies the careful design required to make generative outputs quantitatively evaluable.

The problem. Metric depth estimation requires predicting, for each pixel, a physical distance in meters. These values are unbounded non-negative real numbers: depth ranges from 0 (at the camera plane) to arbitrarily large values (for distant objects like the sky or far-away buildings). The model's output space is RGB images, where each color channel is bounded in [0, 1] (or equivalently, integer values in [0, 255]). The challenge is to define an invertible mapping from unbounded depth values to bounded RGB colors such that (a) nearby depths (which matter more for most applications) receive more discriminative colors than distant depths, (b) the mapping is smooth enough that the model can learn to generate it, and (c) the mapping can be exactly inverted for evaluation.

Step 1: Curving metric depth with a power transform. Raw metric depth values can span several orders of magnitude — a desk might be at 0.5 meters while a mountain might be at 5000 meters. If linearly mapped to [0, 1], almost all RGB resolution would be spent on distant objects, and nearby objects (which are more important for robotics, grasping, and most depth estimation metrics) would be compressed into a tiny fraction of the color space. To address this, the paper applies a power transform from Barron (2025) to "curve" the depth values, allocating more of the normalized range to nearby depths:

$f(d, \lambda, c) = 1 - (1 - d / \lambda c)^{\lambda + 1}$

where $d \in [0, \infty)$ is the raw metric depth in meters, $\lambda$ is a shape parameter controlling the degree of curvature, and $c$ is a scale parameter controlling the overall normalization.

What it computes: This function maps an unbounded depth value $d$ to a normalized value in $[0, 1)$ . For the chosen parameters ( $\lambda = -3$ , $c = 10/3$ ), the function compresses large depths into a small region near 1 while spreading out small depths across most of the [0, 1) range. The exponent $\lambda + 1 = -2$ means the function grows quadratically near zero (providing high discriminability for nearby depths) and flattens asymptotically toward 1 (grouping all distant depths together).

Why this form: The paper explicitly states the motivation: "Because the utility of accurate metric depth for nearby image content is generally higher than that of distant content (e.g., graspable objects matter more for robotics tasks, stereo/monodepth benchmarks usually measure accuracy terms of disparity or relative/log-depth) we 'curve' metric depth prior to RGB encoding" (Section 3.2). In other words, standard depth estimation benchmarks (and practical applications) already emphasize accuracy at close range — metrics like $\delta_1$ threshold accuracy penalize relative errors equally across depths, and robots care about distances to graspable objects. The power transform aligns the color-space allocation with this practical priority, giving nearby depths more RGB resolution.

The specific parameters ( $\lambda = -3$ , $c = 10/3$ ) are set heuristically and held constant across all experiments. The constraint $\lambda < -1$ ensures the transform is concave (compressing distant depths), which is what allocates resolution to nearby objects. The authors do not ablate these parameters, so their optimality is not established, but the principle is clear.

Step 2: Mapping normalized depth to RGB via an RGB cube walk. The normalized depth value $f(d, \lambda, c) \in [0, 1)$ must now be mapped to an RGB color. The paper uses a mapping that "interpolates along a piecewise-linear function that follows the edges of the RGB cube, traversing along its edges from black to white, similarly to the first iteration of a 3D Hilbert curve" (Section 3.2). A visualization of this mapping is provided in Figure 5, which shows the color bar with depth values (in meters) overlaid at various colors.

In essence, this is a space-filling curve traversal of the RGB cube: starting at black (0, 0, 0) for depth 0, the color progresses along the edges of the cube through red, yellow, green, cyan, blue, magenta, and finally white (1, 1, 1) for the maximum normalized distance. This traversal visits the extreme vertices of the RGB cube, producing highly saturated, maximally distinct colors that are easy for the model to generate (since these are "canonical" colors at the corners of RGB space) and easy for the decoder to identify.

Why this mapping: Several alternatives are possible but suboptimal:

A linear RGB mapping (e.g., $R = d$ , $G = d$ , $B = d$ producing a grayscale depth map) would only use 256 distinguishable values (8-bit grayscale), severely limiting depth resolution. It would also be ambiguous under lighting variations (a "medium gray" pixel might be 0.45 or 0.55 depth, hard to distinguish after generation noise).
A standard colormap (e.g., viridis, plasma, inferno) uses smoothly varying colors that are perceptually uniform but not maximally distinct in RGB space. The model might struggle to generate the exact continuous gradient, and small generation errors could shift a pixel's color to a region that maps to a very different depth value.
The RGB cube edge walk uses the most saturated, extreme colors in the RGB space. These are the "easiest" colors for the model to generate (they correspond to pure activations in the output channels) and the easiest to decode (each color corresponds to a unique parameter along the cube's edges, and the mapping is piecewise-linear and analytically invertible). The use of a Hilbert-curve-like traversal ensures that small changes in depth produce visually noticeable changes in color (the curve avoids staying in similar color regions for extended portions of the depth range).

Step 3: Inverting the mapping for evaluation. At inference time, the model generates an RGB image; each pixel's color must be converted back to a metric depth value. The decoding proceeds in reverse:

For each pixel's RGB value, project it onto the nearest line segment of the RGB cube's edge traversal. This handles the case where the model's generated color is not exactly on the cube edge (due to generation noise or minor inaccuracies) — the pixel is snapped to the closest point on the predefined color trajectory.
Invert the linear interpolation along that edge segment to recover the normalized depth value $f_{\text{pred}} \in [0, 1)$ .
Invert the power transform to recover the metric depth:

$d_{\text{pred}} = \lambda c \cdot \left(1 - (1 - f_{\text{pred}})^{1/(\lambda + 1)}\right)$

This inversion is possible because the power transform (Equation 1) is a bijection between $[0, \infty)$ and $[0, 1)$ for $\lambda < -1$ — every depth maps to a unique normalized value, and every normalized value maps back to a unique depth. This is an essential property: if the mapping were not bijective, the decoding would be ambiguous and quantitative evaluation would be impossible.

Training augmentation with alternative colormaps. The paper notes an additional practical detail: "To enhance the model's robustness across diverse color representations, we augment our training data with alternative color maps, such as Plasma, Inferno, Viridis, and grayscale" (Section 3.2). This means that during training, some examples use the RGB cube edge walk while others use standard colormaps. The model is prompted with the specific colormap to use (e.g., "Generate a depth map using the plasma colormap"). At evaluation time, the paper uses the RGB cube edge walk because it is the most precisely invertible. The augmentation ensures the model does not overfit to a single color scheme and can generalize its depth estimation capability across different visual representations. It also means the model learns that depth is about the spatial pattern of colors, not the specific colors themselves — a form of invariance learning.

Important constraint: no camera intrinsics. The paper emphasizes that "this result is achieved without relying on camera parameters (neither intrinsics nor extrinsics) during both training or inference" (Section 3.2). Camera intrinsics (focal length, principal point) are necessary to convert pixel coordinates to metric depth in classical geometric computer vision because a pixel's depth depends on both its disparity and the camera's focal length. Many specialist depth models (DepthLM, Depth Anything V3, Depth Pro, UniK3D, MoGe-2 per Table 3) require intrinsics during training or inference to resolve this ambiguity. Vision Banana does not use intrinsics, meaning it must infer absolute scale purely from visual cues — the known sizes of objects, perspective cues, and learned priors about the physical world. This is a harder problem but demonstrates that the generative pretraining has endowed the model with rich metric knowledge about object scales.

Surface Normal Estimation Encoding: Direct RGB Mapping

Surface normal estimation is the simplest encoding task because surface normals are already 3D vectors — and RGB is also a 3D vector space. The mapping is nearly direct.

The representation. A surface normal at a pixel is a unit vector $(x, y, z)$ where each component ranges from $-1$ to $1$ . The paper uses a camera-space normal formulation with a standard right-handed coordinate system: $+x$ points right, $+y$ points up, and $+z$ points out of the image plane toward the viewer. Unit-vector normalization means $x^2 + y^2 + z^2 = 1$ .

The encoding to RGB. The vector components are linearly mapped to RGB channels using a simple rescaling:

$R = \text{trunc}((1 - x) / 2, \text{min} = 0, \text{max} = 1) \times 255$
$G = \text{trunc}((1 + y) / 2, \text{min} = 0, \text{max} = 1) \times 255$
$B = \text{trunc}((1 + z) / 2, \text{min} = 0, \text{max} = 1) \times 255$

The "trunc" notation means values are clamped to [0, 1] before scaling to [0, 255]. Since $x, y, z \in [-1, 1]$ , the linear mapping $(1 \pm \text{component}) / 2$ transforms each component to [0, 1]. The sign conventions differ per channel because the normal directions have different zero points:

$x = -1$ (facing left) → $R = (1 - (-1)) / 2 = 1$ (full red); $x = +1$ (facing right) → $R = (1 - 1) / 2 = 0$ (no red). The sign convention is flipped ( $1 - x$ instead of $1 + x$ ) so that left-facing surfaces are red and right-facing surfaces are dark.
$y = -1$ (facing down) → $G = (1 + (-1)) / 2 = 0$ (no green); $y = +1$ (facing up) → $G = (1 + 1) / 2 = 1$ (full green). The convention is $1 + y$ so that upward-facing surfaces are green.
$z = -1$ (facing away from camera) → $B = 0$ (no blue); $z = +1$ (facing toward camera) → $B = 1$ (full blue). The convention is $1 + z$ so that surfaces directly facing the camera are blue.

The paper gives specific color descriptions: "Facing Left $(-1, 0, 0)$ : Encoded as Pinkish Red. Facing Up $(0, 1, 0)$ : Encoded as Light Green. Facing the Camera $(0, 0, 1)$ : Encoded as Light Blue/Purple" (Section 3.2). These are the intuitive "canonical" colors that a human would expect for a surface normal map, and they emerge naturally from the linear RGB mapping.

Why this mapping: Unlike depth estimation, surface normal estimation does not require a complex mapping because the output space (3D unit vectors) is naturally isomorphic to normalized RGB colors — both are elements of the unit sphere embedded in 3D Euclidean space (with a trivial rescaling). The mapping is a simple affine transformation from $[-1, 1]^3$ to $[0, 1]^3$ , which is both exactly invertible and visually intuitive. The model can learn this mapping trivially because it is just a color transformation of a natural 3D signal. The main challenge is not the encoding but the prediction itself — the model must infer 3D orientation from a 2D image, which requires understanding of shading, texture gradients, perspective, and object geometry.

Evaluation decoding. At inference time, the generated RGB values are converted back to normal vectors by inverting the affine transformation for each pixel:

$x = 1 - 2 \times (R / 255)$
$y = 2 \times (G / 255) - 1$
$z = 2 \times (B / 255) - 1$

The resulting vector $(x, y, z)$ may not be exactly unit length due to generation noise; the paper does not specify a renormalization step, but standard practice in surface normal evaluation is to either renormalize or compute angular errors using the raw vectors. The evaluation metric is the mean or median angular error between the predicted normal vector and the ground-truth normal vector, computed as $\arccos(\hat{n} \cdot n_{\text{gt}})$ where $\hat{n}$ and $n_{\text{gt}}$ are the predicted and ground-truth unit vectors respectively. This metric is measured in degrees; lower values indicate better performance.

Semantic Segmentation Encoding: Class-to-Color Mapping

Semantic segmentation requires labeling each pixel with one of a predefined set of class names (e.g., "road," "person," "sky"). The challenge is to encode a categorical variable (the class identity) as an RGB color in a way that is (a) easy for the model to generate, (b) robustly decodable, and (c) flexible enough to handle open-vocabulary classes.

The prompting strategy. The prompt explicitly specifies the mapping from class names to colors. The paper demonstrates several prompt formats in Figure 2, showing that the model handles diverse natural language descriptions:

JSON-style structured mapping: "Generate a semantic segmentation visualization image, using this color mapping: {"cat": "red", "lock": "pink", "exit sign": "light purple", "background": yellow}."
RGB tuple notation: "Generate a visualization image of semantic segmentation, using this color mapping: {"cat ears": <255, 165, 0>, "exit sign": <0, 0, 255>, "background":<125,0, 125>}"
Descriptive natural language: "This image is a per-pixel class labeling of the input. The macaron cakes are represented by (255, 255, 0). The round plates are represented by (255, 192, 128). The slice cakes are depicted in (64, 192, 64). The flowers are shown in (128, 0, 64). The tongs are (255, 0, 192)."
Hex code notation: "Generate a semantic segmentation visualization of the input. The menu is #80C000. The dessert is #800000. The patterns on the wall is #40FFC0"

This flexibility is important: it shows that the model has genuinely learned the concept of color-coded segmentation rather than memorizing a specific output format. It can follow arbitrary textually-specified class-to-color mappings, including for class names that were not in the training data (open-vocabulary segmentation).

The generation target. During training, for each input image and its associated class labels, the target output image is constructed by painting each pixel with the RGB color corresponding to its ground-truth class, as specified in the prompt. Pixels belonging to unlisted classes are colored as "background" (typically a specified background color or black).

Interpreting the model's ability. A subtle but important point: the model must perform two distinct sub-tasks simultaneously. First, it must identify which pixels belong to each named class — this requires visual recognition, object boundary detection, and understanding of class semantics. Second, it must render those pixels in the exact color specified by the prompt — this requires precise generation control. The fact that Vision Banana succeeds at SOTA levels (Table 2a, 0.699 mIoU on Cityscapes) suggests that both sub-tasks are well-supported by the generative pretraining: the model already knows what a "cat" looks like and where its boundaries are (from generating images of cats), and it already knows how to produce specific colors in specific regions (from text-to-image generation with color specifications).

Decoding for evaluation. At inference time, the generated RGB image is decoded back to a per-pixel class label. The paper describes this as "we assign pixels to classes by matching its color according to the prompt" (Section 3.1). The simplest approach is nearest-neighbor classification: for each pixel, find the class whose specified color (from the prompt) is closest to the pixel's generated color in RGB space, and assign that class. If the model perfectly generates the specified colors, this is exact. If generation noise produces slight color variations, the nearest-neighbor rule is robust as long as the specified class colors are sufficiently far apart in RGB space.

A critical design detail — the classes are not a fixed set. Unlike traditional semantic segmentation models that are trained to output probabilities over a fixed vocabulary of class labels (e.g., 19 Cityscapes classes), Vision Banana's class vocabulary is specified entirely in the prompt. This means the same model can segment any set of classes — the Cityscapes 19 classes, a custom set of 3 classes defined by the user, or even classes defined by descriptive phrases ("cat ears," "patterns on the wall") rather than single words. This open-vocabulary capability is inherited from the text-to-image generator's understanding of natural language and is a significant advantage over fixed-class specialist models.

The paper shows evidence of this generalization: "Vision Banana can understand nuanced concept in language as well" (Section 3.1), demonstrated in Figure 3b where the model segments "crescent-shaped croissant instances" versus other croissant variations, and Figure 4 where it segments "the chef's names in both Chinese and English" — multilingual text content that requires understanding of both language and text layout.

Instance Segmentation Encoding: Dynamic Color Assignment

Instance segmentation poses a unique challenge for the RGB encoding approach: unlike semantic segmentation, where each class has a single pre-specified color, instance segmentation requires a different color for each individual instance of a class, and the number of instances is unknown in advance. The prompt cannot pre-specify "instance 1: red, instance 2: green, ..." because the model does not know how many instances exist until it analyzes the image.

The per-class inference strategy. The paper's solution is a per-class inference strategy: "For each inference, we instruct Vision Banana to produce segmentation masks for only one class, allowing the model to dynamically assign colors to different instances" (Section 3.1). For example, given an image containing five dogs and three cats, the model is first prompted with "Generate an instance segmentation visualization of this image. Each dog is colored differently" and then separately prompted with "Generate an instance segmentation visualization of this image. Each cat is colored differently." Each inference produces a single-class instance segmentation map where the background is a specified color (e.g., black or a given RGB value) and each instance of the target class is a unique, automatically chosen color.

How the model assigns instance colors. The prompts shown in Figure 3 use phrasings like "Each piece of garlic is colored differently," "Each price tag is colored differently," "Each basketball instance is represented by a solid circular mask, and a different color is used for each mask." The model has autonomy in choosing which colors to assign to instances; the only requirement is that different instances receive visually distinguishable colors. This leverages the model's generative capability to produce coherent, varied colors — something it already knows how to do from text-to-image generation where prompts like "multicolored" or "different colors" are common.

Decoding for evaluation. The decoding step is more complex than for semantic segmentation because the colors are not specified in advance. The paper describes: "During evaluation, we simply cluster pixels that have similar colors by thresholding" (Section 3.1). This is a connected-components-style approach: pixels that are spatially connected and share approximately the same color (within some threshold) are grouped into the same instance mask. The background pixels (which have a specified color, per the prompt) are excluded.

This decoding strategy has inherent limitations:

Color collisions: If the model accidentally assigns very similar colors to two adjacent instances, the threshold-based clustering may merge them into a single instance, reducing the instance count accuracy.
Non-contiguous instances: If an instance is partially occluded and appears as two disconnected regions, they will be assigned different instance IDs unless color similarity across gaps is used (which would risk merging genuinely different instances).
Color bleeding: Slight color variations within a single instance (due to generation noise or shading-like artifacts) could fragment it into multiple instances if the clustering threshold is too tight.

These limitations likely explain why Vision Banana's instance segmentation performance lags slightly behind the best specialist model (DINO-X achieves 0.552 $pmF_1$ on SA-Co/Gold; Vision Banana achieves 0.540*, where the asterisk indicates evaluation on only 500 randomly sampled queries from SA-Co/Gold to save compute, per Table 2b). Instance segmentation is the one task where the RGB encoding format may be inherently limiting compared to specialized mask prediction architectures that output per-pixel instance embeddings or directly predict bounding boxes.

The model's understanding of nuanced language. Despite these limitations, the qualitative examples in Figure 3 demonstrate impressive capabilities: distinguishing "crescent-shaped" croissants from other shapes, identifying price tags as distinct instances, and recognizing basketballs versus other balls. These require understanding both object identity and fine-grained attribute descriptions, further evidence that the generative pretraining has produced rich semantic representations.

Referring Expression Segmentation: Language-Conditioned Instance Selection

Referring expression segmentation is a variant of instance segmentation where the target instance is specified not by a class name but by a free-form natural language description. The task is to produce a mask for the specific object(s) that match the referring expression, such as "the man in pink t-shirt" or "the cat that is cleaning itself."

The prompting strategy. The prompts for referring expression segmentation are straightforward: they ask the model to generate a segmentation mask where the target object is rendered in a specified color (typically solid white or a bright color) and everything else is black or a specified background color. For example, Figure 4a shows the prompt: "A segmentation map image. The area that corresponds to the man in pink t shirt is rendered solid white; the other man is rendered in green." This is a binary segmentation task (target vs. background), which makes decoding trivial: threshold the image to find pixels matching the target color.

Why this task is a natural fit for generative models. The paper claims that "Vision Banana is a natural fit for this task" (Section 3.1) because referring expression segmentation requires:

Language understanding: Comprehending free-form natural language descriptions including attributes ("pink t-shirt"), actions ("stretching," "cleaning itself"), spatial relationships, and even multimodal reasoning about unusual object functionality (Figure 4c: "toaster as a game controller"). Text-to-image generators are explicitly trained to map natural language to visual content, so they inherently possess this capability.
Fine-grained visual recognition: Identifying specific objects based on detailed visual attributes and distinguishing them from similar but non-matching objects. This requires the same kind of discriminative visual understanding that the model uses when generating images from detailed prompts (e.g., generating "a man in a pink t-shirt next to a man in a green t-shirt" requires knowing what distinguishes the two).
Reasoning about relationships: Some referring expressions require reasoning about object relationships or scene context (e.g., "the game control device" in Figure 4c, where a toaster is being used as a game controller). This kind of contextual reasoning is beyond simple category recognition and requires an understanding of object affordances and scene semantics.

The paper also notes an emergent capability: "Vision Banana also demonstrates a similar mastery of referring expressions on semantic and instance segmentation, despite not being explicitly trained to condition these specific tasks on free-form text queries" (Section 3.1). For example, in semantic segmentation (Figure 2b, right), the model understands what "patterns on the wall" refers to, segmenting the decorative pattern as a distinct class separate from the wall itself. This cross-task transfer — where referring expression capability learned for one task spontaneously transfers to others — is evidence of genuine generalization rather than task-specific memorization, and is attributed to the shared generative interface.

Decoding for evaluation. For referring expression segmentation, the standard metric is cIoU (cumulative Intersection-over-Union) for RefCOCOg or gIoU (generalized Intersection-over-Union) for ReasonSeg. The decoding is straightforward: identify pixels matching the target color (within a threshold) as the predicted mask, then compute IoU with the ground-truth mask. There is no need for instance separation or class-to-color mapping because only one target region is specified per prompt.

ReasonSeg and multimodal reasoning. The ReasonSeg benchmark (Lai et al., 2024) is particularly challenging because the referring expressions often require complex reasoning — understanding implicit relationships, resolving ambiguity, or performing multi-step inference. For this benchmark, Vision Banana is "paired with Gemini 2.5 Pro" (Table 2d, footnote), meaning that the reasoning step is outsourced to a large language model: Gemini 2.5 Pro processes the image and the complex referring expression, produces a simplified segmentation prompt (e.g., specifying the target object in clear language), and then Vision Banana performs the actual segmentation. The same setup is used for SAM 3 Agent (the previous SOTA), making the comparison fair. The fact that Vision Banana + Gemini 2.5 Pro achieves 0.793 gIoU, surpassing all prior methods including those trained on in-domain data, suggests that Vision Banana's segmentation quality is high enough that even when paired with the same reasoning engine, it outperforms a specialized segmentation model (SAM 3).

Post-Processing Decoders: Recovering Vision Outputs from RGB Images

Each vision task requires a different decoding procedure to convert the generated RGB image back into a structured prediction. The decoders are not learned — they are deterministic mathematical procedures that invert the encoding schemes defined during training.

Semantic segmentation decoder. For each pixel in the generated image, compute the Euclidean distance in RGB space to each class's specified color (from the prompt). Assign the pixel to the class with the minimum distance. If the minimum distance exceeds a threshold (not specified in the paper), the pixel may be assigned to an "unknown" or "background" class. This is effectively a nearest-neighbor classifier in RGB space, where the class prototypes are the colors specified in the prompt.

Instance segmentation decoder. Since instance colors are not pre-specified, the decoder must discover them from the generated image. The paper describes a threshold-based clustering approach: group spatially connected pixels that have similar colors (within a color-distance threshold). Each connected component becomes a candidate instance mask. The background color (specified in the prompt, e.g., "background is black" or "background is set to white color") is excluded. This is essentially a color-based connected-components algorithm. The per-class inference strategy means the decoder only needs to separate instances of a single class; it does not need to simultaneously distinguish between instances and between classes.

Referring expression decoder. This is the simplest: identify all pixels whose color matches the specified target color (e.g., solid white, green, cyan) within a tolerance threshold. The resulting binary mask is the prediction. No class assignment or instance separation is needed.

Depth estimation decoder. This is the most mathematically involved decoder, described in detail in the depth encoding section above. The steps are:

For each pixel's RGB value, project it onto the nearest line segment of the RGB cube's edge traversal. This finds the closest point on the predefined piecewise-linear color trajectory.
Determine the normalized distance parameter along that edge segment (this is a linear interpolation parameter between the segment's start and end colors).
Convert that normalized parameter to a full normalized depth value $f_{\text{pred}}$ by accounting for the cumulative lengths of all previous edge segments.
Invert the power transform to recover the metric depth: $d_{\text{pred}} = \lambda c \cdot (1 - (1 - f_{\text{pred}})^{1/(\lambda + 1)})$ .

The inversion is exact (up to the projection error in step 1) because both the RGB cube walk and the power transform are bijections. The projection step handles the case where the model's generated colors deviate slightly from the exact cube edges — this is a form of error correction that makes the decoding robust to generation noise.

Surface normal decoder. For each pixel, invert the affine transformation:

$x_{\text{pred}} = 1 - 2 \times (R / 255)$
$y_{\text{pred}} = 2 \times (G / 255) - 1$
$z_{\text{pred}} = 2 \times (B / 255) - 1$

The resulting vector may not be exactly unit length if the model's generation introduces deviations. The paper does not specify a renormalization step, but standard practice is to compute angular error either with or without renormalization depending on the benchmark's convention. The metrics reported (mean and median angular error) are computed by comparing the predicted normal direction to the ground-truth normal direction using the angular distance formula.

Training and Evaluation Protocol: Zero-Shot Transfer

The paper's experimental protocol is designed to test genuine generalization rather than dataset-specific memorization, following the zero-shot transfer standard established by CLIP (Radford et al., 2021) and Segment Anything (Kirillov et al., 2023).

Training data exclusion. The paper states: "no training data from our evaluation benchmarks is included in the instruction-tuning mixture" (Section 2). This means:

For Cityscapes (semantic segmentation), the model never sees Cityscapes training images or annotations during instruction-tuning. It learns semantic segmentation from web-crawled images annotated with in-house models and from synthetic data, then generalizes to the Cityscapes domain (urban street scenes from German cities) at test time.
For NYUv2, iBims-1, ETH3D, DIODE, KITTI, and nuScenes (depth estimation), the model is "trained entirely on synthetic depth data created from simulation engines — we use zero real-world depth data" (Section 3.2). All real-world depth evaluation is therefore zero-shot — the model has never seen these environments, camera types, or depth distributions during training.
For ScanNet, NYUv2, DIODE, and VKitti (surface normal estimation), similarly, the model has not seen the training splits of these datasets.

This zero-shot protocol is stringent. It means that any gap between Vision Banana's performance and specialist models that are trained on in-domain data cannot be attributed solely to data access — the specialist models have an inherent advantage of domain-specific training, making Vision Banana's competitive or superior results more impressive.

In-house annotation models for 2D tasks. The paper's training data for segmentation comes from "in-house model annotations for web-crawled 2D images" (Section 2). The specific annotation models are not named, but they are likely large, capable segmentation models similar to SAM that produce pseudo-ground-truth masks automatically. The quality of these pseudo-labels affects the instruction-tuning quality: if the annotation model makes systematic errors on certain object types or boundary regions, Vision Banana may learn these biases. The paper does not analyze the impact of annotation quality on final performance, so this remains an unknown variable. However, the strong zero-shot results on multiple benchmarks suggest the annotations are of sufficient quality.

Synthetic data for 3D tasks. For depth and surface normal estimation, the training data comes from "rendering engines" — presumably 3D simulation platforms that can generate photorealistic or semi-realistic scenes with perfect geometric ground truth. Common rendering engines in this context include Blender, Unreal Engine, Unity, or specialized datasets like Hypersim or SceneNet. The advantage is perfectly accurate depth and surface normal annotations at unlimited scale; the disadvantage is a domain gap between synthetic rendered scenes and real photographs. The paper's strong results on real-world depth benchmarks (Table 3, achieving 0.948 δ1 on NYUv2, a real indoor dataset) suggest that the generative pretraining on real images helps bridge this domain gap — the model has seen enough real-world visual diversity during pretraining that it can apply geometric reasoning learned from synthetic data to real photographs.

Evaluation metrics per task. The paper uses standard metrics for each benchmark, which enables direct comparison against prior work using identical evaluation protocols:

Semantic segmentation: mIoU (mean Intersection-over-Union) on Cityscapes, computed over 19 classes.
Instance segmentation: $pmF_1$ on SA-Co/Gold, a metric that balances precision and recall for instance-level mask prediction.
Referring expression segmentation: cIoU (cumulative IoU) on RefCOCOg and gIoU (generalized IoU) on ReasonSeg.
Metric depth estimation: δ1 (the fraction of pixels where the ratio between predicted and ground-truth depth is less than 1.25) and AbsRel (absolute relative error, computed as $|d_{\text{pred}} - d_{\text{gt}}| / d_{\text{gt}}$ averaged over pixels). Higher δ1 is better; lower AbsRel is better.
Surface normal estimation: Mean and median angular error in degrees, computed between predicted and ground-truth normal vectors. Lower is better.

Model pairing for reasoning tasks. For ReasonSeg, which requires complex language reasoning about the image, Vision Banana is paired with Gemini 2.5 Pro. The paper notes this in the table footnote: "On ReasonSeg, methods are paired with multimodal LLMs for reasoning. We use Gemini 2.5 Pro in our case" (Table 2). This is standard for the benchmark — SAM 3 Agent, the previous best method, uses the same pairing. The reasoning model handles the interpretation of complex referring expressions (e.g., resolving "the toaster being used as a game controller" into a concrete segmentation target), and Vision Banana performs the actual mask generation. This division of labor — reasoning in language, segmentation in vision — is a natural fit for the generative interface paradigm and demonstrates how Vision Banana can be integrated into larger multimodal pipelines.

Design Choice Summary: Why RGB Encoding Over Alternatives

The paper's approach rests on several deliberate design choices that distinguish it from alternative ways of using generative models for vision tasks:

Why not extract features from the generative model and train a downstream head? This is the approach taken by much prior work (Baranchuk et al., 2021; Bhattad et al., 2023; Zhao et al., 2023) — freeze the generative model, extract its internal feature representations at some layer, and train a task-specific prediction head (e.g., a segmentation decoder or a depth regression head) on top. The paper argues that this approach requires architectural modifications (the prediction head), task-specific training losses, and typically full fine-tuning that degrades generation capabilities. Vision Banana avoids all of these by keeping the output space as RGB images and the training objective as the original generation loss.

Why not use standard vision task output formats (e.g., JSON, coordinate arrays)? Some vision-language models (e.g., Gemini 2.5, GPT-4V) output segmentation masks as polygon coordinates or bounding boxes in structured text formats. The paper opts for RGB images instead because the base model is an image generator — its native output modality is images, and forcing it to output text would require either major architectural changes (adding a text decoder) or training the model to generate images of text (which would be noisy and difficult to parse). By keeping outputs as images, the model operates in its native modality and leverages its full generation capability.

Why not train a single mapping across all tasks (e.g., grayscale for everything)? Using a single color scheme for all tasks would lose the benefits of task-specific encodings. Segmentation needs discrete colors for distinct classes, depth needs a continuous mapping with emphasis on nearby distances, and surface normals need a 3D vector mapping. A one-size-fits-all approach would either be ambiguous (a gray pixel could be a segmentation class, a depth value, or a normal component) or require task-specific prompts to resolve the ambiguity — which is exactly what the paper does, but with task-optimized rather than generic color schemes.

Why use a low mixing ratio for vision data? The paper does not specify the exact ratio, but the principle is clear from the context: if the vision data dominated the training mixture, the model would experience catastrophic forgetting of its generation capabilities. The low ratio ensures that the model's primary training signal remains its original generation objective, with the vision data serving as a "nudge" to follow specific color conventions. This is analogous to how LLM instruction-tuning typically uses a small fraction of instruction data relative to pretraining data size — the goal is alignment, not retraining.

Why not train on the evaluation benchmarks' training sets? The paper's zero-shot protocol is deliberate: by excluding evaluation benchmarks from training, the results demonstrate generalization rather than dataset-specific fine-tuning. This is important for the paper's central claim that generative pretraining produces general visual understanding. If the model were fine-tuned on Cityscapes training data, its strong Cityscapes performance would not be evidence of general segmentation ability — it could simply be memorizing the Cityscapes domain. The zero-shot results prove that the model's segmentation ability transfers across domains, camera types, and class vocabularies.

4. Key Insights and Innovations

Innovation 1: Generative Vision Pretraining as a Sufficient Foundational Training Paradigm

This paper's most intellectually significant contribution is not its instruction-tuning method or its RGB encoding scheme — it is the empirical demonstration that generative pretraining alone, without any discriminative objectives, produces visual representations competitive with or superior to the best purpose-built discriminative models in the field. This is a paradigm-level claim about how computer vision should be done.

What the field believed before this work. The computer vision community has operated under the implicit assumption that understanding and generation require fundamentally different training paradigms. Visual understanding has been pursued through a progression of explicitly discriminative methods: supervised classification (Krizhevsky et al., 2012; Dosovitskiy et al., 2020), contrastive learning (Radford et al., 2021; He et al., 2020; Chen et al., 2020b), self-supervised bootstrapping (Caron et al., 2021; Grill et al., 2020), and masked autoencoding (He et al., 2022; Bao et al., 2021). Each of these paradigms optimizes the model to produce features useful for distinguishing categories, matching views, or reconstructing missing patches — objectives designed to force the model to learn what things are. Meanwhile, image generation has been pursued through GANs, diffusion models, and autoregressive transformers, optimized to produce convincing pixels, with evaluation focused on photorealism and prompt adherence. These two traditions have remained largely separate, with occasional attempts to repurpose generative features for understanding (Baranchuk et al., 2021; Bhattad et al., 2023; Li et al., 2023) yielding results that consistently lagged behind discriminative specialists.

The paper's results systematically overturn this assumption. Vision Banana, built solely by instruction-tuning a generative model on a small amount of vision-formatted data, achieves:

0.699 mIoU on Cityscapes semantic segmentation, surpassing SAM 3 (0.652) — a model purpose-built for segmentation with extensive human-annotated mask data and specialized architectural choices.
0.738 cIoU on RefCOCOg referring expression segmentation, exceeding SAM 3 Agent (0.734).
Average δ1 of 0.929 on metric depth estimation across four datasets, beating Depth Anything V3 (0.918) — the state-of-the-art specialist trained with domain-specific losses and camera intrinsic conditioning.
Lowest mean angular error on indoor surface normal estimation (Tables 1, 4), surpassing Lotus-2, a model explicitly fine-tuned on surface normal tasks with specialized training.

These are not marginal improvements on a subset of tasks — they represent state-of-the-art across both 2D semantic understanding and 3D geometric inference, achieved by a single model with no task-specific architecture.

Why this is fundamental, not incremental. This finding is not a methodological refinement; it is a scientific result about what representations emerge from generative training. It demonstrates that the objective "learn to generate realistic images from text descriptions" is sufficiently rich to induce representations that capture object identity, semantic category boundaries, spatial relationships, 3D geometry, metric scale, and surface orientation — precisely the properties that decades of discriminative representation learning research has attempted to extract via specialized objectives. This parallels the discovery in NLP that next-token prediction induces reasoning and understanding capabilities (Brown et al., 2020), but for vision, the claim is arguably stronger: visual understanding from generative pretraining not only works but actually surpasses the best discriminative alternatives, something that has not been consistently true in language (where discriminatively trained models like BERT-family architectures remain competitive on many understanding benchmarks).

The paper's framing as a paradigm shift is supported by the evidence. The authors explicitly position this as "a major paradigm shift for computer vision, where generative vision pretraining takes a central role in building Foundational Vision Models for both generation and understanding" (Section 4). This framing would be empty rhetoric if the results were merely competitive; the fact that Vision Banana surpasses specialist models on multiple benchmarks makes the claim credible. The paper is not arguing that generative pretraining is "also good" — it is arguing that generative pretraining is better, and that the field's continued investment in discriminative pretraining for visual understanding may be path-dependent rather than optimal.

A critical nuance: the capability is latent, not created by instruction-tuning. The paper's instruction-tuning procedure uses a "low ratio" of vision data in the training mixture, and the model retains its generation capabilities (53.5% win rate vs. base model on GenAI-Bench). This means the understanding capability was already present in Nano Banana Pro's weights before instruction-tuning; the tuning merely taught the model to express that understanding in a precisely decodable format. This is the core of the LLM analogy: just as GPT-3 could perform translation or summarization before instruction-tuning but needed alignment to produce reliably formatted outputs, Nano Banana Pro could "understand" scene geometry and object identity before tuning but needed alignment to produce benchmark-compatible visualizations. The success of lightweight instruction-tuning is evidence for the pre-existence of the capability, not evidence that the capability was learned during tuning.

Anchored to evidence. This innovation is demonstrated across Tables 1–4 and Figures 2–8. The most compelling single piece of evidence is Table 3, where Vision Banana achieves a higher average δ1 than Depth Anything V3 on the four overlapping datasets (0.929 vs. 0.918) without using camera intrinsics — a piece of information that every competing specialist model uses during training or inference. This means the generative pretraining has endowed the model with metric-scale priors that eliminate the need for the geometric information that the field has assumed is essential for monocular depth estimation.

Innovation 2: The "Generation as Universal Interface" Principle — RGB as a Unifying Output Space

The paper's second major conceptual contribution is the demonstration that RGB image generation can serve as a universal output interface for vision tasks, analogous to how text generation serves as the universal output interface for NLP tasks. This is a design principle rather than a training innovation, but its implications are far-reaching because it eliminates the need for task-specific architectures, loss functions, and decoding heads.

What the field did before. Prior work on unified vision models attempted to handle multiple tasks through two main strategies: (1) shared backbones with task-specific heads (e.g., a shared encoder feeding separate decoders for segmentation, depth, and normals, each with their own loss function), or (2) text-based output formats where the model generates structured text describing the vision output (e.g., polygon coordinates for segmentation masks, or "the depth at pixel (x,y) is z meters"). The first approach requires architectural modifications and task-specific training objectives for each new task, making it difficult to add tasks without modifying the model. The second approach forces the model to output text, which is inefficient for dense prediction tasks (describing every pixel in a 1024×1024 image as text would require millions of output tokens) and does not leverage the native generation capabilities of image generators.

The paper's approach is fundamentally different: the model's output is always an RGB image, regardless of the task. For semantic segmentation, it generates a color-coded class map. For depth estimation, it generates a false-color depth visualization. For surface normal estimation, it generates a normal map. For creative generation, it generates a photograph or artwork. The prompt specifies both the task and the precise color convention to use, and the model's output is always in its native modality. This means:

No architecture changes. The exact same model weights process every task. There is no segmentation head to train, no depth regression layer, no normal decoder. The model is a text-to-image generator, and it remains exactly that — it just receives different text prompts.

No task-specific losses. All tasks use the same generation loss (the diffusion model's denoising objective or equivalent). The model does not need to learn to optimize scale-invariant depth losses, cross-entropy for segmentation, or angular losses for normals — it just learns to generate the target RGB image, and the quality of the understanding output is determined by how well that generated image matches the target visualization.

Open-vocabulary by construction. Because tasks are specified via natural language prompts, the model can handle arbitrary class vocabularies (for segmentation), arbitrary color mappings, and even tasks it was not explicitly trained on — the paper shows the model performing referring expression segmentation on semantic and instance segmentation tasks "despite not being explicitly trained to condition these specific tasks on free-form text queries" (Section 3.1).

Why this is a conceptual advance, not just an engineering convenience. This design principle matters beyond its implementation simplicity. It suggests that vision tasks are not fundamentally different from each other — they are all instances of conditional image generation, where the condition is the input image plus a task specification, and the output is a visualization of the desired information. Semantic segmentation is "generate an image where each pixel's color indicates its class." Depth estimation is "generate an image where each pixel's color indicates its distance." Surface normal estimation is "generate an image where each pixel's color indicates its orientation." By unifying these under the generation framework, the paper dissolves the artificial distinction between "generation" and "perception" that has structured the field for decades.

The paper explicitly makes this connection to NLP's universal text interface in Section 4: "image generation can serve as the universal interface for computer vision, analogous to how text generation acts as the unifying interface for many tasks embedded in natural language." In NLP, the discovery that text generation could serve as a universal output format enabled models like GPT-3 to perform translation, summarization, question answering, code generation, and reasoning all through the same next-token prediction mechanism. Vision Banana demonstrates the visual analog: that RGB image generation can serve as the universal output format for segmentation, depth estimation, normal estimation, image editing, and creative generation all through the same denoising mechanism.

The hard part is making the outputs evaluable. The conceptual insight is elegant, but the execution requires careful engineering of invertible color mappings — the power transform for depth, the RGB cube walk for color-to-value encoding, the per-class inference strategy for instance segmentation. These task-specific encoding schemes are not the intellectual contribution; they are the necessary engineering to make the universal interface quantifiably testable. The intellectual contribution is the principle that such encodings are sufficient to unify diverse vision tasks under a single generation framework.

The generative handling of ambiguity is a bonus insight. The paper notes in Section 4 that "generative models inherently learn the full data distribution, gracefully managing ambiguity by design." This contrasts with discriminative models, which "usually resort to custom architectures and training losses" to handle ambiguous inputs where a single image could have multiple valid depth interpretations or segmentation boundaries. For example, Segment Anything models generate multiple mask candidates and apply the loss to only one, requiring architectural support for multi-output prediction. Generative models avoid this by sampling from the learned distribution — if there are multiple valid segmentations, the model can generate any of them, and the ambiguity is naturally represented in the output distribution rather than requiring explicit handling.

Anchored to evidence. The universality claim is supported by the breadth of tasks demonstrated: semantic segmentation (Table 2a, Figure 2), instance segmentation (Table 2b, Figure 3), referring expression segmentation (Table 2c–2d, Figure 4), metric depth estimation (Table 3, Figures 6–7), surface normal estimation (Table 4, Figure 8), text-to-image generation (Figure 9), and image editing (Figure 10) — all using the same model with only prompt changes. The open-vocabulary generalization is demonstrated in Figure 2b (segmenting "patterns on the wall" and "cat ears" — phrases unlikely to appear in segmentation training data) and Figure 4d (segmenting multilingual text content).

Innovation 3: The Diagnostic Result That Generative Representations Already Encode Metric 3D Geometry and Object Semantics

Beyond the paradigm-shift argument, the paper provides a specific and surprising diagnostic finding: generative models trained on 2D images learn internal representations that encode metric-scale 3D geometry — not just relative depth ordering, but actual physical distances in meters — without ever being trained on metric depth data during pretraining. This finding has implications for our understanding of what generative models learn and for the feasibility of 3D vision from 2D supervision.

What makes this finding surprising. Monocular metric depth estimation is fundamentally ill-posed in classical computer vision: a single 2D image can correspond to infinitely many 3D scenes differing by an overall scale factor, and recovering absolute scale requires knowledge of camera intrinsics or scene priors (e.g., known object sizes). Most specialist depth models require camera intrinsics during training or inference to resolve this ambiguity — DepthLM, Depth Anything V3, Depth Pro, UniK3D, and MoGe-2 all use intrinsics at training time (Table 3). The assumption has been that without intrinsics, a model can at best estimate relative depth (up to an unknown scale), and that the scale must come from explicit geometric information.

Vision Banana achieves state-of-the-art metric depth estimation without using camera intrinsics during either training or inference (Table 3). This means the model is inferring absolute scale purely from visual cues — it knows approximately how large a chair is in meters, how far away a wall typically is in an indoor scene, how tall a person appears at a given distance. This knowledge cannot come from the depth training data (which is synthetic and could contain any scale), so it must come from the generative pretraining: from having seen millions of real photographs of chairs, walls, and people, the model has internalized statistical regularities about object sizes and scene layouts in physical units.

The "vibe test" in Figure 7 is particularly revealing. An author takes a casual smartphone photograph near Kinkaku-Ji temple, Vision Banana estimates the depth at a marked point as 13.71 meters, and the actual distance measured on Google Maps is 12.87 meters — an AbsRel error of 0.065. This is a single anecdotal example, but it's compelling because the photo is not from any benchmark dataset, the camera is a consumer cell phone, and the model has never seen this location. The model's ability to estimate near-correct metric depth on an arbitrary in-the-wild photograph suggests that its internal representation of object scales and scene geometry is genuinely general, not dataset-specific.

Why this changes our understanding of generative pretraining. The standard narrative about generative models is that they learn to model the appearance of the world — color distributions, texture patterns, object shapes, and how they compose into scenes. The depth estimation results suggest they learn much more: they learn a quantitative, metric understanding of physical space. This is not an obvious consequence of training to generate images. A model could learn to generate a realistic image of a room with a desk and a chair without knowing the exact distance between them in meters — relative positioning would suffice for visual plausibility. The fact that metric depth emerges suggests that the training data contains sufficient statistical signal about absolute scales (from diverse camera types, known object sizes, EXIF metadata potentially including focal length) that the model internalizes a "measuring tape" for the visual world.

The surface normal results provide convergent evidence. Surface normal estimation requires understanding local 3D orientation — which way each surface is facing. Vision Banana achieves the lowest mean angular error on indoor surface normal benchmarks (Table 4), surpassing Lotus-2, a model specifically designed for geometric prediction. This is not metric in the same way depth is (normals are unit vectors, scale-invariant), but it confirms that the generative model has learned detailed 3D geometry, not just coarse scene layout.

The segmentation results demonstrate semantic grounding. The model's ability to segment arbitrary open-vocabulary classes with fine-grained precision — "cat ears" and "crescent-shaped croissant instances" (Figures 2–3) — demonstrates that the generative representations encode detailed object identity, part-level structure, and attribute-level distinctions. The model does not just know that an object is a "cat"; it knows what cat ears look like and can segment them separately from the cat's body. This level of semantic granularity is typically associated with models trained on densely annotated instance segmentation or part segmentation data, not models trained only on image captions.

This innovation is a diagnostic result, not a method contribution. The paper does not propose a new technique for extracting depth or semantics from generative models. The contribution is the demonstration that the information exists in the model and can be accessed through instruction-tuning. This reframes generative models from "image synthesizers" to "visual world models" — systems that have learned a rich, queryable internal representation of the physical and semantic structure of the visual world. This has implications for how we evaluate generative models (should we test their geometric understanding even if we only care about generation?) and for how we build visual AI systems (perhaps the most compute-efficient path to general visual intelligence is to train the best possible generative model, then extract whatever capabilities are needed).

Anchored to evidence. The metric depth claim is supported by Table 3, specifically the average δ1 across four datasets (0.929 vs. 0.918 for Depth Anything V3) and the qualitative examples in Figures 6–7. The semantic grounding claim is supported by the segmentation results in Table 2 and the qualitative examples in Figures 2–4. The convergent evidence from both 2D semantics and 3D geometry strengthens the case that these are not isolated capabilities but reflect a genuinely general-purpose visual understanding.

Innovation 4: Instruction-Tuning Without Catastrophic Forgetting as Evidence for Latent Capability

The paper makes a methodological contribution that doubles as a diagnostic discovery: the demonstration that vision task capabilities can be unlocked through instruction-tuning with a low mixing ratio of vision data, while completely preserving the base model's generation capabilities, is strong evidence that the capabilities were already latent in the pretrained weights rather than learned from scratch during fine-tuning. This finding has implications for how the field should approach model adaptation and for our understanding of what large-scale pretraining produces.

What the field typically observes with fine-tuning. When a pretrained model is fine-tuned on a new task without including data from the original task, catastrophic forgetting is the norm — performance on the original task degrades significantly because the model's weights are overwritten to optimize the new objective. Prior work that adapts generative models for vision tasks through full fine-tuning (Ke et al., 2024; Zhao et al., 2025; Gan et al., 2023) achieves strong vision task performance but loses generation capability — the models can segment or estimate depth, but can no longer generate creative images. This is exactly what standard transfer learning theory predicts: the model's capacity is reallocated from the source task to the target task.

Vision Banana breaks this pattern. With instruction-tuning that mixes vision task data into Nano Banana Pro's training data "at a very low ratio" (Section 2), the model achieves state-of-the-art on vision understanding tasks while scoring a 53.5% win rate against its untuned base model on GenAI-Bench (text-to-image) and 47.8% on ImgEdit (image editing) — both near 50%, indicating that human evaluators cannot reliably distinguish the instruction-tuned model from the original on generation tasks (Section 2, Table 1). The qualitative comparisons in Figures 9 and 10 (Appendix) confirm that the generation outputs are nearly indistinguishable.

Why this is evidence for latent capability. If the vision task capabilities had to be learned from scratch during instruction-tuning, they would require substantial changes to the model's weights — enough that the generation capabilities should degrade, especially given that the vision data is "at a very low ratio" and thus provides limited training signal. The fact that (1) strong vision performance emerges and (2) generation performance is preserved simultaneously suggests that the bulk of the learning happened during pretraining, and the instruction-tuning is merely aligning existing knowledge to the desired output format — the distinction the paper draws with LLM instruction-tuning.

This claim is reinforced by the observation that zero-shot prompting of the base model (without instruction-tuning) already produces plausible-looking vision outputs, as documented by Wiedemer et al. (2025) and Zuo et al. (2025). Those outputs are not precisely formatted enough for benchmarking, but they demonstrate that the base model already "knows" what a depth map or segmentation mask looks like. The instruction-tuning is teaching format compliance, not visual understanding.

The cross-task transfer provides additional evidence. The paper notes that Vision Banana demonstrates referring expression capabilities on semantic and instance segmentation "despite not being explicitly trained to condition these specific tasks on free-form text queries" (Section 3.1). If the model were simply memorizing task-specific behaviors during instruction-tuning, this transfer would not occur — the model would only follow referring expressions on the referring expression task. The fact that the capability spontaneously generalizes to other segmentation tasks suggests it is a property of the underlying representations, not a learned behavior tied to specific training prompts.

This innovation is a diagnostic discovery dressed as a methodology contribution. The instruction-tuning approach is methodologically novel (mixing vision data at low ratio, preserving generation), but its deeper significance is what it tells us about the pretrained model: that visual understanding in generative models is not a separate capability that must be added post-hoc, but rather a naturally emergent property of large-scale generative training. This diagnostic finding has implications for model evaluation (we should test pretrained generators for understanding even if they weren't trained for it), model architecture (perhaps the best way to build a vision understanding model is to build a better image generator), and training efficiency (if understanding emerges from generation, we don't need to spend separate compute budgets on discriminative pretraining).

Anchored to evidence. The preservation of generation capabilities is documented in Section 2 (win rates on GenAI-Bench and ImgEdit) and the qualitative Figures 9–10. The cross-task transfer is demonstrated by the referring expression examples in Figures 2b and 3b, and the associated text noting that this capability was not explicitly trained. The comparison to full-fine-tuning methods that lose generation capabilities is made explicitly in Section 2 ("Such strategy differentiates our work from previous works that perform full finetuning to generative models without image generation data").

5. Experimental Analysis

Evaluation Methodology

Datasets and benchmarks. The paper evaluates on a diverse set of standard academic benchmarks spanning 2D semantic understanding and 3D geometric inference. For 2D segmentation: Cityscapes val (Cordts et al., 2016) for semantic segmentation (19 classes, mIoU), SA-Co/Gold (Carion et al., 2025) for instance segmentation (pmF1, evaluated on a randomly sampled subset of 500 queries "to save compute"), RefCOCOg UMD val (Kazemzadeh et al., 2014) for referring expression segmentation (cIoU), and ReasonSeg val (Lai et al., 2024) for reasoning-based referring expression segmentation (gIoU, paired with Gemini 2.5 Pro for reasoning). For 3D understanding: NYU v2 (Silberman et al., 2012), iBims-1 (Koch et al., 2018), ETH3D (Schops et al., 2019), DIODE-Indoor (Vasiljevic et al., 2019), KITTI (Uhrig et al., 2017), and nuScenes (Caesar et al., 2020) for metric depth estimation (δ1, AbsRel); NYUv2, DIODE-indoor, ScanNet (Dai et al., 2017), and Virtual KITTI 2 (Cabon et al., 2020) for surface normal estimation (mean and median angular error in degrees). For generation preservation, GenAI-Bench (Li et al., 2024a) for text-to-image and ImgEdit (Ye et al., 2025) for image editing, both evaluated via human preference win rates.
Base model. All experiments use Nano Banana Pro (Google, 2025a) as the pretrained image generator. The paper does not disclose the exact parameter count, architecture details, or training data scale of Nano Banana Pro — it is referred to as "a leading image generator" and the model's capabilities are taken as given. The choice is justified by the paper's thesis: Nano Banana Pro is representative of the current generation of high-fidelity text-to-image models (cited alongside FLUX.2, Seedance 2.0, Veo 3, and GPT-Image-1.5 in Section 1) whose generation quality has crossed a threshold where latent understanding capabilities become extractable. The paper does not experiment with alternative base models, so the generalizability of findings to other generator architectures (diffusion vs. autoregressive, different scales, different training data) is not established.
Metrics. For semantic segmentation, mIoU (mean Intersection-over-Union) computed over all classes in the dataset (19 for Cityscapes), where IoU for each class is the ratio of intersection to union between predicted and ground-truth pixel masks. For instance segmentation, pmF1 — the paper does not define this metric explicitly, but it is a standard measure combining precision and recall for instance-level mask prediction, where matched prediction-ground-truth instance pairs are scored based on mask IoU. For referring expression segmentation, cIoU (cumulative IoU) on RefCOCOg and gIoU (generalized IoU) on ReasonSeg — both measure the overlap between the predicted mask and the single ground-truth mask for the referred object, with gIoU being a variant that accounts for multiple valid referring targets. For metric depth estimation, δ1 (the fraction of pixels where max(d_pred/d_gt, d_gt/d_pred) < 1.25 — i.e., the prediction is within 25% of ground truth) and AbsRel (mean over pixels of |d_pred − d_gt| / d_gt). Higher δ1 is better; lower AbsRel is better. For surface normal estimation, mean and median angular error in degrees, computed as arccos(n_pred · n_gt) per pixel, where both vectors are assumed unit length. For generation, win rate from human side-by-side evaluation (Vision Banana vs. Nano Banana Pro).
Baselines. The paper compares against a comprehensive set of prior work, organized by task. For semantic segmentation (Table 2a): SegMan-L (Fu et al., 2025) as a non-zero-shot upper bound (0.842 mIoU, trained on Cityscapes); zero-shot methods include APE-D (Shen et al., 2024, 0.442), OpenSeeD (Zhang et al., 2023a, 0.478), X-Decoder (Zou et al., 2023, 0.520), and SAM 3 (Carion et al., 2025, 0.652). For instance segmentation (Table 2b): SAM 3 (0.661 pmF1, non-zero-shot); zero-shot methods include APE-D (0.369), OWLv2 (Minderer et al., 2023, 0.420), Gemini 2.5 (Gemini Team, 2025, 0.461), and DINO-X (Ren et al., 2024, 0.552). For referring expression segmentation on RefCOCOg (Table 2c): HyperSeg + Phi2 2.7B (Wei et al., 2024, 0.794) and X-SAM + Phi3 3.8B (Wang et al., 2026a, 0.838) as non-zero-shot upper bounds; zero-shot baselines include HybridGL (Liu and Li, 2025, 0.513), Kang et al. + LLaVA-1.5 13B (Kang et al., 2025, 0.677), and SAM 3 Agent + Gemini 2.5 Pro (0.734). For ReasonSeg (Table 2d): X-SAM (0.566) and LISA-13B (Lai et al., 2024, 0.650) as non-zero-shot; zero-shot baselines include SegZero (Liu et al., 2025, 0.626), RSVP (Lu et al., 2025, 0.647), and SAM 3 Agent + Gemini 2.5 Pro (0.770). For metric depth (Table 3): DepthLM-7B (Cai et al., 2025), Depth Anything V3 (Lin et al., 2025), Depth Pro (Bochkovskii et al., 2024), UniK3D (Piccinelli et al., 2025a), and MoGe-2 (Wang et al., 2025c) — all specialist models, most requiring camera intrinsics at training or inference time. For surface normals (Table 4): Marigold (Ke et al., 2024), DSINE (Bae and Davison, 2024), StableNormal (Ye et al., 2024), and Lotus-2-Normal (He et al., 2025). For generation, the baseline is the untuned Nano Banana Pro itself.
Generation budget and compute accounting. The paper does not report generation budgets or inference FLOPs for vision tasks — there is no controlled comparison of "Vision Banana at N denoising steps vs. specialist model at equivalent FLOPs." This is a notable omission for a paper arguing about the efficiency of generative pretraining. The only compute-related note is that Vision Banana is evaluated on a 500-query subset of SA-Co/Gold "to save compute" (Table 2b footnote). For the generation preservation experiments, compute is implicitly controlled by comparing outputs from the same architecture at the same resolution. The paper acknowledges in Section 4 that "utilizing image generators like Nano Banana Pro currently incurs a significantly higher computational overhead than running lightweight specialist models" and lists this as a topic for future work.
Training data and zero-shot protocol. The critical experimental design choice is the zero-shot transfer protocol: "no training data from our evaluation benchmarks is included in the instruction-tuning mixture" (Section 2). For 2D tasks, training data comes from "in-house model annotations for web-crawled 2D images"; for 3D tasks, from "synthetic data from rendering engines." This means all evaluation results reflect cross-domain generalization — the model is tested on datasets (Cityscapes, NYUv2, KITTI, etc.) whose training splits were excluded from instruction-tuning. The paper does not report the exact size of the instruction-tuning dataset or the precise mixing ratio of vision data to generation data. There is no cross-validation or statistical significance testing reported — all results appear to be single-run evaluations on the standard test/validation splits of each benchmark.

Main Quantitative Results

2D Semantic Understanding: Segmentation

Semantic segmentation on Cityscapes (Table 2a). Vision Banana achieves 0.699 mIoU on Cityscapes val, establishing new state-of-the-art among zero-shot transfer methods. This surpasses SAM 3 by 4.7 points (0.652 → 0.699), X-Decoder by 17.9 points (0.520 → 0.699), and OpenSeeD by 22.1 points (0.478 → 0.699). The gap to the non-zero-shot upper bound SegMan-L (0.842, trained on Cityscapes) is 14.3 points — substantial but notable given Vision Banana has never seen Cityscapes-style urban street scenes during training. The paper does not report per-class IoU breakdown, so it is unclear whether the improvement over SAM 3 is uniform across classes or concentrated on specific categories.

Instance segmentation on SA-Co/Gold (Table 2b). Vision Banana achieves 0.540 pmF1 on a randomly sampled 500-query subset. Under the zero-shot transfer setting, this surpasses APE-D (0.369), OWLv2 (0.420), and Gemini 2.5 (0.461), but trails DINO-X (0.552) by 1.2 points. Compared to SAM 3 in the non-zero-shot setting (0.661), Vision Banana lags by 12.1 points, though SAM 3 was trained on SA-Co data while Vision Banana was not. This is the only segmentation task where Vision Banana is not the top zero-shot method, and the paper acknowledges the challenge: instance segmentation requires dynamic color assignment for an unknown number of instances, a harder constraint under the RGB encoding approach than structured mask prediction.

Referring expression segmentation on RefCOCOg (Table 2c). Vision Banana achieves 0.738 cIoU on RefCOCOg UMD val, beating SAM 3 Agent + Gemini 2.5 Pro (0.734) by a narrow 0.4 point margin and substantially outperforming other zero-shot methods (HybridGL at 0.513, Kang et al. at 0.677). The gap to non-zero-shot methods that train on RefCOCOg is larger: HyperSeg + Phi2 2.7B achieves 0.794, and X-SAM + Phi3 3.8B achieves 0.838. These non-zero-shot methods also use smaller language models (Phi2 2.7B, Phi3 3.8B) compared to Gemini 2.5 Pro, suggesting their advantage comes from in-domain training rather than reasoning capability.

Referring expression segmentation on ReasonSeg (Table 2d). Paired with Gemini 2.5 Pro for reasoning, Vision Banana achieves 0.793 gIoU, setting a new state-of-the-art that surpasses all prior methods including those trained on in-domain data. The previous best zero-shot method, SAM 3 Agent + Gemini 2.5 Pro, achieves 0.770 (a 2.3 point gap). The previous best non-zero-shot method, LISA-13B, achieves 0.650 (a 14.3 point gap). This is the paper's strongest single segmentation result — Vision Banana not only dominates zero-shot methods but exceeds models explicitly trained on ReasonSeg training data. The success on ReasonSeg, which requires complex reasoning about language and visual relationships, suggests the generative pretraining is particularly valuable for tasks requiring cross-modal understanding.

Qualitative assessment of segmentation generality. Figures 2–4 demonstrate capabilities beyond what the quantitative benchmarks measure. Figure 2 shows the model handling diverse prompting styles (JSON, RGB tuples, natural language descriptions, hex codes) and segmenting fine-grained concepts ("cat ears," "patterns on the wall," "whiskers"). Figure 3 shows instance segmentation with nuanced language understanding ("crescent-shaped croissant instances," "each piece of garlic is colored differently"). Figure 4 shows referring expression capabilities including action descriptions ("the stretching cat"), unusual object uses ("toaster as a game controller"), and multilingual text recognition. The paper claims that referring expression capabilities transfer to semantic and instance segmentation "despite not being explicitly trained to condition these specific tasks on free-form text queries" (Section 3.1) — this cross-task generalization is demonstrated qualitatively rather than through a controlled quantitative experiment.

3D Understanding: Metric Depth Estimation

Aggregate performance across datasets (Table 3, summary rows). The paper reports two aggregate comparisons to handle the fact that different specialist methods evaluate on different subsets of datasets. When compared on the four datasets that Depth Anything V3 evaluates on (NYU, ETH3D, DIODE, KITTI), Vision Banana achieves an average δ1 of 0.929 vs. 0.918 for Depth Anything V3 — a 1.1 point advantage. When compared on the four datasets that DepthLM-7B evaluates on (NYU, iBims1, ETH3D, nuScenes), Vision Banana achieves an average δ1 of 0.865 vs. 0.855 for DepthLM-7B — a narrower 1.0 point advantage. Across all six evaluated datasets, Vision Banana achieves an unweighted average δ1 of 0.882 and an average AbsRel that is "20% lower" (0.127 vs. 0.156, specific baseline not named in the main text but inferable from Table 3 as compared to MoGe-2 at 0.156 AbsRel on the overlapping datasets).

Per-dataset analysis reveals strengths and weaknesses. On NYU v2 (indoor, near-field), Vision Banana achieves δ1 = 0.948, AbsRel = 0.116. This is competitive with the best specialists (Depth Anything V3: δ1 = 0.963; Depth Pro: δ1 = 0.961; UniK3D: δ1 = 0.965) but not the absolute best — DepthLM-7B achieves δ1 = 0.915. The AbsRel of 0.116 is comparable to the best performers (UniK3D: 0.081; Depth Pro: 0.0733). On ETH3D (outdoor, varied scales), Vision Banana achieves δ1 = 0.935, the highest reported value, surpassing Depth Anything V3 (0.908) and UniK3D (0.687 — a surprisingly low value that may indicate UniK3D's difficulty with this benchmark). On DIODE-Indoor, Vision Banana achieves δ1 = 0.917, substantially outperforming all compared methods (Depth Anything V3: 0.838; UniK3D: 0.713; Depth Pro: 0.671; MoGe-2: 0.664). This is the dataset where Vision Banana shows the largest relative advantage, suggesting particular strength in indoor geometry estimation. On KITTI (autonomous driving, distant scenes), Vision Banana achieves δ1 = 0.915, surpassing Depth Anything V3 (0.843 — but note the ‡ footnote indicating this number was "reported by Depth-Anything V3" rather than independently reproduced) and all other compared methods. On iBims-1, Vision Banana achieves δ1 = 0.934, slightly below DepthLM-7B (0.920) and Depth Anything V3 (0.913 — Figure 5 in the paper suggests this dataset was partially evaluated). On nuScenes, Vision Banana achieves δ1 = 0.643, which is notably lower than DepthLM-7B (0.865 — but DepthLM was trained on nuScenes, making the comparison not zero-shot) and Depth Anything V3 (0.840).

The KITTI result requires scrutiny. The Depth Anything V3 δ1 of 0.843 on KITTI is footnoted as "‡ Numbers reported by Depth-Anything V3" (Table 3), meaning the authors did not independently evaluate Depth Anything V3 on KITTI but cite the numbers from the Depth Anything V3 paper. Similarly, the DepthLM-7B numbers include the caveat that DepthLM "is trained on nuScenes so it's not zero-shot." These inconsistencies in evaluation protocol across baselines make direct numerical comparisons less clean than the table suggests.

The no-intrinsics result. Vision Banana achieves these results "without relying on camera parameters (neither intrinsics nor extrinsics) during both training or inference" (Section 3.2, Table 3 header). All compared specialist models require camera intrinsics — DepthLM-7B, Depth Anything V3, Depth Pro, UniK3D, and MoGe-2 all use intrinsics at training time, and some also at inference (Depth Pro: inference ✓; DepthLM-7B: training ✓; Depth Anything V3: training ✓; UniK3D: training ✓; MoGe-2: training ✓, per Table 3 header). This makes Vision Banana's results more impressive: it achieves SOTA without access to information that the field has assumed is necessary for metric depth estimation. However, the paper does not ablate whether the base model might be implicitly inferring intrinsics from visual cues (focal length affects perspective distortion, which is visible in images), so "without using intrinsics" means "without explicit intrinsic inputs" rather than "without intrinsic information."

Qualitative depth results (Figures 6–7). Figure 6 shows depth maps and 3D reconstructions on NYU v2 and ETH3D samples, demonstrating that the predicted depths can be unprojected into geometrically consistent 3D point clouds — the paper shows two novel views of each reconstructed scene. Figure 7 presents the "vibe test": a smartphone photo at Kinkaku-Ji temple where Vision Banana estimates 13.71 meters at a marked point, compared to 12.87 meters from Google Maps (AbsRel ≈ 0.065). This is a single anecdotal example with no statistical power, but it demonstrates generalization to completely out-of-distribution consumer photographs.

3D Understanding: Surface Normal Estimation

Indoor surface normal estimation (Table 4, indoor columns). Vision Banana achieves the lowest mean and median angular errors averaged across the three indoor datasets (NYUv2, DIODE-indoor, ScanNet). On NYUv2: mean angle error = 15.549°, median = 9.300° — this is the best reported result, surpassing Lotus-2 (mean 16.558°, median not reported for Lotus-2 on NYUv2), DSINE (17.017° / 10.190°), and StableNormal (17.168° / 10.028°). On DIODE-indoor: mean = 17.778°, median = 8.876°, compared to Lotus-2 at 16.9° mean (median "N/A"), DSINE at 16.4° / 8.4°, and StableNormal at 19.707° / 10.527°. On ScanNet: mean = 13.818°, median = 11.556°, best among compared methods (Lotus-2: 18.575° mean; DSINE: 18.453° / 13.871°; StableNormal: 13.701° / 9.46°).

Outdoor surface normal estimation (Virtual KITTI 2). On the outdoor dataset VKitti, Vision Banana achieves mean = 29.063°, median = 10.699°, which is slightly worse than Lotus-2 (mean 28.894°, median 9.677° — but Lotus-2 "is trained on Virtual KITTI 2 for surface normal estimation" per the paper's note, making the comparison not zero-shot). Vision Banana also trails DSINE (16.2° / 8.3°) by a substantial margin on this outdoor benchmark. The paper notes that despite higher quantitative error on VKitti, Vision Banana produces "demonstrably superior visual quality" with "higher fidelity and finer granular details" (Section 3.2, Figure 8), highlighting a tension between quantitative metrics and qualitative assessment that is common in surface normal evaluation (mean angular error can be dominated by a few large errors while median and visual quality remain good).

The visual quality vs. quantitative metrics tension (Figure 8). The bottom row of Figure 8 shows a VKitti sample where Vision Banana's surface normal map appears visually sharper and more detailed than Lotus-2's — individual objects have crisper boundaries and the normal directions appear more consistent within object surfaces. This is a recurring phenomenon in generative dense prediction: diffusion-based methods tend to produce visually pleasing, sharp outputs that may not perfectly match ground-truth pixel values, while regression-based methods may achieve lower per-pixel error at the cost of blurrier outputs. The paper does not deeply investigate this tradeoff, but it is an important caveat for interpreting the quantitative metrics.

Visual Generation Preservation

Text-to-image generation (GenAI-Bench). In human side-by-side evaluations comparing Vision Banana against Nano Banana Pro on GenAI-Bench prompts, Vision Banana achieves a 53.5% win rate — meaning human raters preferred Vision Banana's output slightly more often than the base model's output. A win rate near 50% indicates the two models are essentially indistinguishable in generation quality. Figure 9 (Appendix) shows four qualitative examples where Vision Banana (left) and Nano Banana Pro (right) produce highly similar images for prompts like "A ghostly ship sailing on a fog-shrouded, moonlit sea" and "A samurai with a silk sash in a cherry blossom garden" — the outputs are compositionally and stylistically consistent between the two models.

Image editing (ImgEdit). On the ImgEdit benchmark, Vision Banana achieves a 47.8% win rate against Nano Banana Pro — again near 50%, indicating near-indistinguishable editing quality. Figure 10 (Appendix) shows four examples: "Change the grassy hills in the picture to a beach with ocean waves," "Remove the plant from the shelf, and resize the picture frame to be larger," "Change the vehicle's color to red," and "Change the background of the suit from a blank wall to a luxurious office setting." In each case, both models produce plausible edits that follow the instruction, with similar visual quality.

Interpretation of generation preservation. These results are the paper's key evidence that instruction-tuning does not cause catastrophic forgetting. However, several caveats apply: (1) the sample sizes for human evaluation are not reported — we don't know how many comparisons were made or whether the 53.5% / 47.8% win rates are statistically distinguishable from 50%; (2) the evaluation is only against the base model, not against other state-of-the-art generators — even if Vision Banana equals Nano Banana Pro, both could have been superseded by more recent models at the time of publication; (3) the comparisons are qualitative (human preference) rather than quantitative (FID, CLIP score, etc.), making them harder to reproduce and compare across papers; (4) there is no evaluation of generation capabilities on tasks that might conflict with vision understanding — for example, whether the model's improved geometric understanding makes it worse at generating surreal or physically impossible scenes.

Ablation Studies and Robustness Checks

The paper includes remarkably few formal ablation studies. This is a significant weakness relative to what would be expected for a paper making paradigm-level claims. Below are the design choices that are discussed, even if not systematically ablated:

Low mixing ratio of vision data to generation data. The paper states that vision task data is mixed "at a very low ratio" (Section 2) and argues this preserves generation capabilities, but the exact ratio is never specified, and no alternative ratios are tested. The paper does not show what happens if the mixing ratio is higher (does generation quality degrade? at what point? does understanding performance improve?) or lower (is there a minimum threshold below which instruction-tuning fails?). This is a critical missing ablation because the central claim — that lightweight instruction-tuning unlocks latent capabilities — depends on the ratio being small. Without varying the ratio, we cannot distinguish between "unlocking pre-existing capabilities" and "efficiently learning new capabilities from a small dataset."

Colormap augmentation for depth estimation. The paper mentions augmenting depth training data with alternative colormaps (Plasma, Inferno, Viridis, grayscale) to improve robustness (Section 3.2), but does not report an ablation comparing performance with and without this augmentation. It is unclear whether the augmentation is necessary, helpful, or irrelevant to the final depth estimation performance. Since the evaluation uses only the RGB cube edge walk colormap, the augmentation's value would be as a regularizer — but this hypothesis is untested.

Choice of depth encoding parameters (λ = −3, c = 10/3). The power transform parameters are set heuristically and held constant. No ablation varies λ or c to determine sensitivity to these choices. Would performance degrade if a different curvature were used? Is the specific RGB cube edge traversal superior to alternative invertible color mappings (e.g., a direct hue-based encoding, or a learned color-to-depth mapping)? These questions are unexplored.

PRM/ORM training details and verifier quality. Note: This paper does not use PRMs or ORMs — those were concepts from the reference example. For this paper, the equivalent would be ablating the quality of instruction-tuning data (e.g., synthetic vs. real annotations for segmentation, different rendering engines for 3D data). No such ablation exists. The paper uses in-house model annotations for 2D tasks and synthetic data for 3D tasks but does not compare against alternative data sources or analyze how annotation quality affects downstream performance.

Per-class vs. all-class instance segmentation inference. The paper adopts a per-class inference strategy for instance segmentation to handle unknown instance counts. No comparison is provided against alternative strategies such as single-pass inference where the model assigns colors to all instances across all classes simultaneously, or iterative refinement approaches where the model segments one instance at a time and masks it out for subsequent inferences. The per-class strategy's cost (N inferences for N classes) and its impact on performance are not analyzed.

Reasoning model pairing for ReasonSeg. Vision Banana relies on Gemini 2.5 Pro for reasoning on ReasonSeg, but no ablation compares different reasoning models or measures Vision Banana's standalone reasoning capability. If the same Gemini 2.5 Pro is paired with a weaker segmentation model, does performance drop significantly? Or is the reasoning model doing most of the work? The paper cannot disentangle these factors without an ablation.

Transfer of referring expression capability to other segmentation tasks. The paper claims that referring expression capability transfers to semantic and instance segmentation without explicit training (Section 3.1), but this claim is supported only by qualitative examples (Figures 2b, 3b), not by a quantitative experiment. A proper ablation would evaluate referring-expression-conditioned semantic/instance segmentation on a benchmark with quantitative metrics and compare against a model explicitly trained for this capability.

Training data scale and diversity. The paper provides no details about the size of the instruction-tuning dataset (number of images, number of tasks, distribution across tasks) or the diversity of the web-crawled and synthetic data. No experiment varies the quantity of instruction-tuning data to determine whether Vision Banana's performance is near saturation or would continue to improve with more data.

Full fine-tuning baseline for generation preservation. The paper argues that lightweight instruction-tuning preserves generation while full fine-tuning does not, citing prior work (Ke et al., 2024; Zhao et al., 2025; Gan et al., 2023). However, the paper does not include its own full fine-tuning baseline on the same base model and tasks to directly quantify the tradeoff between understanding performance and generation preservation. Without this, the claim that "lightweight instruction-tuning is necessary and sufficient" is supported by cross-paper comparisons rather than controlled experiment.

Critical Assessment

Central Claim: Image generators are generalist vision learners — generative pretraining serves a role similar to LLM pretraining, enabling state-of-the-art visual understanding.

What the experiments demonstrate. The paper shows that a specific image generator (Nano Banana Pro), when instruction-tuned on a mixture of vision and generation data, can match or exceed specialist models on segmentation, depth estimation, and surface normal estimation benchmarks under a zero-shot transfer protocol. The results on ReasonSeg (0.793 gIoU, SOTA), metric depth (average δ1 of 0.929 vs. 0.918 for Depth Anything V3 on four datasets), and semantic segmentation (0.699 mIoU on Cityscapes, best zero-shot) are genuinely strong and demonstrate that the model has acquired substantial visual understanding.

What the experiments do not demonstrate. The paper does not establish causality — it does not prove that the understanding came from generative pretraining as opposed to some other aspect of the model's training. Nano Banana Pro is a proprietary model whose training recipe (architecture, data, objectives, scale) is not disclosed. It may use auxiliary losses, multimodal training objectives, or data sources that contribute to visual understanding independently of the generation objective. The paper's central claim requires that generation be the cause of understanding, but the experiments only show correlation: a model trained (partially) for generation exhibits understanding after instruction-tuning.

A stronger experiment would compare Vision Banana against a model of identical architecture and data scale trained with a non-generative objective (e.g., contrastive learning, masked autoencoding) and instruction-tuned identically. If the generative model substantially outperforms the discriminative model, that would support the causal claim. Without this comparison, alternative explanations remain viable: perhaps the understanding comes from the scale of data and compute rather than the generative objective, or from auxiliary training signals applied alongside generation.

Additionally, the paper experiments with only one base model (Nano Banana Pro). The claim that "image generators are generalist vision learners" is a claim about a class of models, but only one instance is tested. Different generators (FLUX.2, Seedance, GPT-Image-1.5) may exhibit different understanding capabilities depending on their architecture, training data, and scale. The paper's title and conclusions implicitly generalize from a single model to the entire category, which the experiments do not support.

Central Claim: Vision Banana surpasses or rivals domain specialists including SAM 3 on segmentation and Depth Anything 3 on depth estimation.

What the experiments demonstrate. On semantic segmentation (Cityscapes), Vision Banana (0.699 mIoU) surpasses SAM 3 (0.652) by a clear margin. On referring expression segmentation (RefCOCOg), Vision Banana (0.738 cIoU) edges out SAM 3 Agent (0.734) by a narrow 0.4 points. On ReasonSeg, Vision Banana (0.793 gIoU) beats SAM 3 Agent (0.770) by 2.3 points. On metric depth, Vision Banana (average δ1 0.929) beats Depth Anything V3 (0.918) by 1.1 points across the four overlapping datasets.

What the experiments do not demonstrate — the SAM 3 comparison caveats. SAM 3 is evaluated under different conditions across the segmentation benchmarks. On Cityscapes (Table 2a), SAM 3 achieves 0.652 mIoU as a "Zero-Shot Transfer" method — meaning it was not trained on Cityscapes. On SA-Co/Gold (Table 2b), SAM 3 achieves 0.661 pmF1 but is listed as "Non Zero-Shot Transfer" — meaning it was trained on SA-Co data, while Vision Banana (0.540 pmF1) is zero-shot. The comparison is therefore not apples-to-apples: SAM 3 had access to in-domain training data for instance segmentation but not for semantic or referring segmentation. Vision Banana's superior performance on semantic and referring tasks is achieved without this in-domain advantage, but its inferior performance on instance segmentation may partly reflect the data disparity.

Furthermore, SAM 3 Agent on ReasonSeg and RefCOCOg is a composite system (SAM 3 + Gemini 2.5 Pro), just as Vision Banana is a composite system (Vision Banana + Gemini 2.5 Pro) on ReasonSeg. The paper does not isolate whether the improvement comes from better segmentation or from the Gemini 2.5 Pro reasoning component. If SAM 3's segmentation masks were paired with Vision Banana's prompts (or vice versa), would the results change? The composability of these systems makes direct model-to-model comparisons ambiguous.

What the experiments do not demonstrate — the Depth Anything V3 comparison caveats. The depth comparison uses numbers reported by Depth Anything V3's paper rather than independent re-evaluation (footnoted in Table 3 for the KITTI result). The Depth Anything V3 model may have been evaluated under different conditions (different image preprocessing, different depth range normalization, different handling of invalid pixels). Without running both models through identical evaluation pipelines, the comparison is between numbers from different papers rather than a controlled benchmark. This is standard practice in the field but limits the reliability of fine-grained comparisons (1.1 δ1 point difference across four datasets is a small margin that could be affected by evaluation protocol differences).

Additionally, Depth Anything V3 and the other depth specialists use camera intrinsics at training time, while Vision Banana does not. This means Vision Banana is solving a harder problem (metric depth from visual cues alone) yet achieves better results — which supports the paper's claim. But it also means the specialist models may not be optimized for the no-intrinsics setting. If Depth Anything V3 were retrained without intrinsics (or fine-tuned to infer them implicitly), might it close the gap? The paper cannot answer this without controlled experiments varying the intrinsics condition.

Central Claim: Lightweight instruction-tuning unlocks latent capabilities without sacrificing generation quality.

What the experiments demonstrate. The GenAI-Bench (53.5% win rate) and ImgEdit (47.8% win rate) evaluations suggest that human raters cannot reliably distinguish Vision Banana's generation outputs from Nano Banana Pro's. This is moderately strong evidence that generation capabilities are preserved.

What the experiments do not demonstrate. The sample sizes and statistical significance of the human evaluations are not reported. A 53.5% win rate based on 100 comparisons could easily be within sampling noise of 50%. Without confidence intervals or sample sizes, we cannot assess whether the observed win rates are statistically distinguishable from 50/50.

More importantly, the paper tests generation preservation only on the specific tasks evaluated (text-to-image on GenAI-Bench, image editing on ImgEdit). It does not test whether more complex generation capabilities — compositional generation, long-form coherent image sequences, style-consistent character generation, generation of specific rare concepts — are preserved. The generation benchmarks used are standard but limited in scope. It's possible that the model has suffered subtle degradation in generation dimensions not captured by these benchmarks (e.g., reduced diversity, over-stylization toward the vision data's visual characteristics, or reduced ability to generate certain object categories that were heavily represented in the vision training data).

Furthermore, the claim that the capabilities were "latent" (pre-existing in the base model) rather than learned during instruction-tuning is not directly tested. The paper argues this based on (a) the low mixing ratio of vision data, (b) the preservation of generation capabilities, and (c) prior observations of zero-shot vision behaviors in base models (Wiedemer et al., 2025; Zuo et al., 2025). More direct evidence would include: evaluating the base model's understanding via probing (linear classifiers or other lightweight decoders on frozen features), measuring how quickly understanding performance improves during instruction-tuning (fast improvement suggests latent capability; slow improvement suggests new learning), or ablating the amount of instruction-tuning data to see if performance saturates quickly (suggesting format alignment rather than capability learning). None of these experiments are performed.

Central Claim: Image generation serves as a universal interface for vision tasks, analogous to text generation's role in language.

What the experiments demonstrate. The paper shows that a single model can perform seven distinct tasks (three segmentation variants, depth estimation, surface normal estimation, text-to-image generation, image editing) with a unified RGB-image-in, RGB-image-out interface. The model's ability to handle diverse color specifications (JSON, hex codes, RGB tuples, natural language) within its prompts is demonstrated qualitatively.

What the experiments do not demonstrate. The "universality" claim would be strengthened by showing that new tasks can be added without architectural changes or additional training of the base model — for example, demonstrating zero-shot or few-shot transfer to a held-out task (e.g., edge detection, keypoint estimation, optical flow) that was not included in instruction-tuning. The paper does not test this. All demonstrated tasks were included in the instruction-tuning mixture, so we cannot distinguish between "the model learned to perform these specific tasks" and "the model learned a general task-specification capability that can handle arbitrary new vision tasks."

Additionally, some tasks may be fundamentally awkward to represent as RGB images. Instance segmentation with hundreds of overlapping instances across dozens of classes would require either a massive number of colors (exceeding the distinguishable capacity of 24-bit RGB) or per-class inference (which scales linearly with the number of classes, as the paper's instance segmentation approach does). Tasks requiring non-spatial outputs — object counting, visual question answering, scene graph generation — do not naturally map to per-pixel RGB images. The "universal interface" claim applies specifically to dense prediction tasks with spatial outputs; the paper's framing as a general replacement for vision task interfaces overstates the demonstrated scope.

Critical Weaknesses in Experimental Design

No controlled comparison of pretraining objectives. The paper's key scientific claim — that generative pretraining specifically is responsible for the understanding capabilities — is untested. A minimal experiment to support this claim would train two models with identical architectures and data but different objectives (e.g., diffusion generation vs. MAE vs. contrastive), then compare their instruction-tuned understanding performance. Without this, the paper shows that a large model trained on massive data (with an undisclosed mix of objectives) performs well after instruction-tuning — which is consistent with many possible explanations beyond the specific role of generation.

Single model family, undisclosed training details. All experiments use Nano Banana Pro, a proprietary model whose architecture, training data composition, training objectives, parameter count, and compute budget are not disclosed. This makes the results impossible to reproduce independently and limits the scientific conclusions to "this specific Google model works this way." The paper's claims about generative pretraining in general would require replication across multiple model families, scales, and training paradigms — none of which is attempted.

Missing ablations for critical design choices. As detailed in the Ablation Studies section above, the paper does not systematically vary: the mixing ratio of vision to generation data, the choice of depth encoding parameters, the use of colormap augmentation, the per-class inference strategy for instance segmentation, the quantity of instruction-tuning data, or the choice of base model. This makes it difficult to assess which design choices are essential and which are incidental.

Small or absent sample sizes for key evaluations. The instance segmentation evaluation uses only 500 randomly sampled queries from SA-Co/Gold (Table 2b footnote) "to save compute." The human evaluations for generation preservation report win rates without sample sizes or confidence intervals. No statistical significance tests are reported for any comparison. In a paper making claims about state-of-the-art performance, the absence of statistical rigor makes it difficult to assess whether apparent improvements (e.g., 0.738 vs. 0.734 cIoU on RefCOCOg, a 0.4 point difference) are reliable or within sampling noise.

Inconsistent zero-shot protocols across baselines. The paper's comparisons mix methods that are zero-shot, non-zero-shot, and partially-zero-shot without always clearly flagging the differences. SAM 3 is zero-shot on Cityscapes but trained on SA-Co. DepthLM-7B is trained on nuScenes. Lotus-2 is trained on Virtual KITTI 2. The paper notes these in footnotes and gray shading but does not control for them systematically. The strongest claim — that Vision Banana surpasses specialists — should ideally be demonstrated against specialists that are also zero-shot on the target benchmark, or against the specialists' performance when both have equivalent data access.

No compute-efficiency comparison. Vision Banana is built on Nano Banana Pro, a large image generator that likely requires orders of magnitude more inference compute than lightweight specialist models (e.g., Depth Anything V3 is designed for efficient inference; SAM 3 can process images in real-time). The paper acknowledges this in Section 4 ("utilizing image generators like Nano Banana Pro currently incurs a significantly higher computational overhead than running lightweight specialist models") but provides no quantitative FLOP or latency comparison. For practitioners, the question is not just "does Vision Banana achieve higher accuracy?" but "is the accuracy gain worth the additional compute cost?" The paper does not answer this.

Summary of What the Experiments Genuinely Establish

The experiments convincingly demonstrate that a specific, large-scale image generator can be instruction-tuned to perform diverse vision understanding tasks at state-of-the-art levels while retaining its generation capabilities. This is a significant engineering and empirical result — it shows that the gap between generative and discriminative models for visual understanding has been closed for at least some tasks under some conditions.

The experiments provide suggestive but not conclusive evidence for the paper's broader scientific claims: that generative pretraining causes visual understanding to emerge, that this emergence is a general property of generative models rather than a specific property of Nano Banana Pro, and that RGB image generation is a universal interface for vision tasks. These claims are consistent with the presented results but are not uniquely supported by them — alternative explanations (scale, data diversity, auxiliary objectives, specific architectural choices) cannot be ruled out.

The most solidly established result is the performance on metric depth estimation without camera intrinsics — this is a clear demonstration that generative pretraining has equipped the model with metric-scale priors that the field previously assumed required explicit geometric information. The most clearly oversold result is the "universal interface" claim, which is demonstrated on a modest set of seven tasks, all of which were included in training, with no test of generalization to held-out tasks.

6. Limitations and Trade-offs

The Cost of Difficulty Estimation and Instruction-Tuning Data Curation Is Not Accounted For

The assumption or constraint. The paper's headline results — SOTA performance across seven tasks with a single model — depend on access to a pretrained image generator that already possesses the requisite visual understanding capabilities. This capability is taken as given; the paper does not account for the computational or data cost of producing the base model (Nano Banana Pro), nor does it include the cost of curating the instruction-tuning data. The paper acknowledges this implicitly in Section 4:

"utilizing image generators like Nano Banana Pro currently incurs a significantly higher computational overhead than running lightweight specialist models. Developing acceleration and cost-reduction strategies will be an essential hurdle to overcome for the deployment of generative vision framework."

However, the cost of producing the instruction-tuning data itself — web-crawled images annotated by in-house models for 2D tasks, synthetic data rendered from simulation engines for 3D tasks — is never quantified. The paper provides no information about the scale of the instruction-tuning dataset (number of images, number of task examples, distribution across tasks), so the cost of assembling this data relative to the cost of simply training a specialist model from scratch on existing public datasets is unknown.

The consequence. A practitioner considering this approach must weigh the total cost of (1) pretraining or licensing a large image generator, (2) curating and rendering instruction-tuning data for each desired vision task, and (3) running inference with a model that may require orders of magnitude more FLOPs than a purpose-built lightweight specialist. The paper's claim that "generative pretraining is a sufficient foundational training paradigm" must be understood in the context of these costs: it is sufficient given access to a sufficiently capable generator, but the cost of producing that generator is externalized from the paper's analysis.

For any new vision task not already demonstrated (e.g., optical flow, keypoint estimation, edge detection), a practitioner would need to design a new invertible RGB encoding scheme, curate training data with the appropriate ground-truth-to-RGB mappings, and instruction-tune the model — all without knowing in advance whether the base model's pretrained representations support the task well enough for instruction-tuning to succeed. The paper provides no guidance on how to determine which tasks are "within the model's capability range" before investing in data curation.

What evidence exists in the paper. The paper provides no FLOP comparison, no latency comparison, no training cost estimates, and no data scale numbers. Section 4 acknowledges the inference cost issue qualitatively but does not quantify it. The paper does not compare the total cost (pretraining + instruction-tuning data + instruction-tuning compute) against the cost of training a specialist model from scratch on the same tasks. The instance segmentation evaluation on SA-Co/Gold uses only 500 randomly sampled queries "to save compute" (Table 2b footnote), which hints at the computational burden but provides no systematic accounting.

Mitigation status. The paper does not attempt to mitigate this limitation. It lists "developing acceleration and cost-reduction strategies" as future work (Section 4) but provides no concrete proposals, baselines, or cost estimates. This limitation is fundamental to the generative pretraining paradigm: if the base model is expensive to train and expensive to run, the approach may be practically viable only for organizations that already have access to large pretrained generators, rather than as a general strategy for building vision systems.

The Method Fails on Instance Segmentation and Shows Weaker Performance on Tasks Requiring Unknown-Quantity Outputs

The assumption or constraint. The RGB encoding approach assumes that the mapping between vision outputs and colors can be specified in the prompt before inference. For tasks where the output structure is known in advance — semantic segmentation (known classes), depth estimation (known per-pixel continuous mapping), surface normals (known per-pixel vector mapping), referring expression segmentation (binary target vs. background) — this works because the prompt can specify exactly which colors should appear and what they mean. Instance segmentation breaks this assumption: the number of instances per class is unknown and variable, so the prompt cannot pre-specify which colors correspond to which instances.

The paper's workaround is a per-class inference strategy: run a separate forward pass for each class of interest, letting the model dynamically assign distinct colors to instances within that class, and then cluster pixels by color similarity for evaluation. The paper does not report the computational cost of this strategy (number of forward passes per image) or analyze how performance degrades as the number of classes increases.

The consequence. On SA-Co/Gold, the only instance segmentation benchmark evaluated, Vision Banana achieves 0.540 pmF1 under the zero-shot setting — trailing DINO-X (0.552) and substantially behind SAM 3 (0.661, non-zero-shot). This is the one segmentation task where Vision Banana is not the top-performing method in its category (Table 2b). The gap is larger than on any other segmentation benchmark: Vision Banana's 0.540 is below the zero-shot best DINO-X (0.552) by ~2.2%, while on semantic segmentation it leads the next best zero-shot method (0.699 vs. 0.652, a +7.2% advantage) and on referring expression segmentation it leads (0.738 vs. 0.734, narrowly; 0.793 vs. 0.770 on ReasonSeg).

The per-class inference strategy also introduces a tradeoff between completeness and cost: segmenting all instances in a scene with many object categories requires one forward pass per category, making the approach scale linearly with the number of classes. For open-vocabulary instance segmentation across an unbounded set of categories (e.g., "segment every distinct object in this image"), the cost is potentially unbounded. The paper does not discuss this scaling limitation.

What evidence exists in the paper. Table 2b shows the quantitative gap: 0.540 pmF1 vs. 0.552 (DINO-X, zero-shot) and 0.661 (SAM 3, non-zero-shot). The qualitative examples in Figure 3 demonstrate that the approach can produce reasonable instance segmentations for simple scenes with a small number of instances per class, but there is no evaluation on scenes with densely overlapping instances, high instance counts, or fine-grained distinctions (e.g., segmenting individual leaves on a tree, individual bricks in a wall). The paper implicitly acknowledges the challenge of instance segmentation by describing the per-class inference strategy as a workaround rather than a natural fit.

Mitigation status. The paper does not attempt to mitigate this limitation. No alternative decoding strategies are proposed (e.g., letting the model output numeric instance IDs as text overlaid on the image, or using iterative masking where the model segments one instance at a time), and no experiments explore whether performance improves with more sophisticated color-clustering or post-processing. The limitation is specific to tasks where the output structure cannot be fully specified in the prompt — a class that likely includes not just instance segmentation but also panoptic segmentation, object detection (where bounding boxes or polygons are needed), and counting tasks. The paper does not discuss the boundaries of the RGB encoding approach or identify which vision tasks are inherently incompatible with it.

The Central Causal Claim — That Generative Pretraining Causes Visual Understanding — Is Not Tested

The assumption or constraint. The paper's thesis is that "image generation pretraining serves a role similar to LLM pretraining" and that "the ability to create visual content implies an ability to understand it." This is a causal claim: generative training objectives cause the emergence of visual understanding capabilities. To test this claim, one would need to compare models with identical architectures, data scales, and compute budgets that differ only in their training objective (e.g., generative vs. discriminative vs. contrastive), and demonstrate that the generatively trained model develops superior understanding capabilities.

The paper does not perform this comparison. All experiments use Nano Banana Pro, a proprietary model whose full training recipe (objectives, data composition, auxiliary losses, training stages) is not disclosed. It may include multiple training phases with different objectives, auxiliary discriminative losses, or data sources that contribute to visual understanding independently of the generation objective. The paper's experiments demonstrate correlation between generation training and understanding capability in one model, not causation.

The consequence. The paper's most ambitious claim — that the computer vision field is "witnessing a major paradigm shift for computer vision, where generative vision pretraining takes a central role in building Foundational Vision Models for both generation and understanding" (Section 4) — rests on an untested causal premise. If the understanding capabilities observed in Nano Banana Pro arise from the scale of training data, the architecture design, or auxiliary objectives rather than from generation specifically, then the prescription ("train generative models, not discriminative ones") may be incorrect. A model trained with the same architecture and data but a discriminative objective (e.g., contrastive learning, masked autoencoding) might develop equally strong or stronger understanding capabilities. Without controlled comparisons, practitioners cannot determine whether to invest in generative pretraining, discriminative pretraining, or some hybrid.

The reliance on a single proprietary model also means the results may not generalize. Other image generators (FLUX.2, Seedance, GPT-Image-1.5) may exhibit different understanding profiles depending on their training recipes. A practitioner using a different base model cannot assume that instruction-tuning will yield comparable results. The paper's title — "Image Generators are Generalist Vision Learners" — implies a property of the class of models, but the evidence comes from a single instance.

What evidence exists in the paper. The paper provides no controlled comparison of pretraining objectives. All referenced prior work on generative understanding (Baranchuk et al., 2021; Bhattad et al., 2023; Li et al., 2023; Wiedemer et al., 2025; Zuo et al., 2025) demonstrates that generative models possess some understanding, but does not establish that they possess more understanding than equivalently-resourced discriminative models. The paper reports no probing experiments (e.g., linear classifier performance on frozen features) that would allow comparison against discriminatively pretrained feature extractors. The instruction-tuning results demonstrate that the base model can be adapted to vision tasks, but a model with different pretraining might adapt equally well or better.

Mitigation status. The paper does not address this limitation. The discussion in Section 4 treats the causal link between generation and understanding as established: "Models pretrained on large-scale image generation naturally acquire robust visual understanding capabilities. These generative priors surpass the specialized architectures and dedicated training paradigms traditionally employed by specialist vision models." The first sentence assumes what needs to be proved; the second makes an empirical claim (surpassing specialist paradigms) that is demonstrated for the specific model tested but not established as a general property of generation. No future work is proposed to disentangle the effects of pretraining objective, scale, data diversity, and architecture.

Surface Normal and Depth Estimation Metrics Are Inconsistent, and Visual Quality Assessment Is Subjective

The assumption or constraint. The paper evaluates 3D understanding tasks using standard quantitative metrics (δ1, AbsRel for depth; mean/median angular error for normals) and supplements these with qualitative visual comparisons. For surface normal estimation specifically, the paper claims that despite slightly worse mean angular error on Virtual KITTI 2 (29.063° vs. 28.894° for Lotus-2), Vision Banana produces "demonstrably superior visual quality" with "higher fidelity and finer granular details" (Section 3.2).

This creates a tension: the quantitative metrics suggest one ranking, while qualitative visual inspection suggests another. The paper does not resolve this tension by introducing a metric that captures the visual quality dimensions it claims are superior, nor does it explain why the quantitative metrics fail to capture the perceived quality. This is a known issue in generative dense prediction (diffusion-based methods tend to produce sharp, visually plausible outputs that may have high-frequency details deviating from ground-truth pixel values), but the paper treats visual quality as a valid counterweight to quantitative metrics without establishing it rigorously.

The consequence. The surface normal comparison against Lotus-2 on VKitti is ambiguous: Vision Banana loses on the quantitative metric by a small margin (29.063° vs. 28.894° mean error) but "wins" on visual quality according to the authors' qualitative judgment. A practitioner comparing methods cannot objectively weigh these claims — is Vision Banana better or worse at surface normal estimation on outdoor scenes? The paper's claim of "state-of-the-art" for surface normal estimation (Table 1) is based on the indoor average (where Vision Banana does lead quantitatively), but the outdoor performance is less clear.

For depth estimation, there is a similar subtlety: the Depth Anything V3 numbers for KITTI are footnoted as "‡ Numbers reported by Depth-Anything V3" (Table 3), meaning the authors did not independently evaluate this baseline. The paper also notes that "the average δ1 of Depth-Anything V3 on the 4 datasets it evaluated on (NYU + ETH3D + DIODE + KITTI) is 0.918; our average δ1 on the same 4 datasets is 0.929." This is a 1.1 point difference computed across papers with potentially different evaluation pipelines. Without running both models through identical evaluation code, the comparison may be affected by subtle differences in depth range clipping, invalid pixel handling, or metric computation.

What evidence exists in the paper. The surface normal visual quality claim is supported by Figure 8, which shows side-by-side comparisons on four examples. The Lotus-2 outputs appear somewhat blurrier and less detailed than Vision Banana's, consistent with the authors' claim. However, this is qualitative and based on a small number of cherry-picked examples — the paper does not report a systematic visual quality study (e.g., user study with multiple raters, or a perceptually-motivated metric). The depth evaluation inconsistencies are documented in the Table 3 footnotes, which acknowledge that DepthLM is trained on nuScenes (not zero-shot) and that KITTI numbers for Depth Anything V3 are cited from that paper rather than reproduced.

Mitigation status. Partial. The paper is transparent about the footnoted evaluation inconsistencies (Table 3 footnotes explicitly flag non-zero-shot baselines and cited numbers), but it does not attempt to resolve the tension between quantitative and qualitative evaluation for surface normals. A user study validating the visual quality claim, or a perceptual metric (e.g., LPIPS, or a task-specific metric that is less sensitive to high-frequency noise than per-pixel angular error), would strengthen the claim. The paper does not propose such metrics or studies.

No Generalization Evidence Beyond the Instruction-Tuned Tasks or the Single Base Model

The assumption or constraint. The paper's "universal interface" claim — "image generation can serve as the universal interface for computer vision, analogous to how text generation acts as the unifying interface for many tasks embedded in natural language" (Section 4) — implies that the approach can handle vision tasks beyond those explicitly included in instruction-tuning. In NLP, instruction-tuned LLMs can perform tasks they were never explicitly trained on (e.g., a model instruction-tuned on summarization and translation can also perform sentiment analysis when prompted appropriately). For the vision analog to hold, Vision Banana would need to demonstrate similar cross-task generalization.

The paper does not test this. All seven tasks evaluated (semantic segmentation, instance segmentation, referring expression segmentation, depth estimation, surface normal estimation, text-to-image generation, image editing) were included in the instruction-tuning mixture. There is no evaluation on a held-out vision task (e.g., edge detection, keypoint estimation, optical flow, image matting, intrinsic image decomposition) to test whether the model has learned a general "vision-task-following" capability rather than a set of task-specific skills.

The consequence. A practitioner cannot assume that adding a new vision task to the system will require only prompt engineering. If the model has not learned a general task-specification capability, each new task may require its own instruction-tuning data and training, eroding the claimed benefit of a "universal interface." The paper's analogy to NLP may be premature: LLMs can perform new tasks zero-shot after instruction-tuning because the task specification and answer are both in the text modality, which the model was trained to manipulate. For Vision Banana, a new task requires (1) an appropriate invertible RGB encoding scheme (which may not exist or may require careful design, as with the depth power transform), (2) training data in that format, and (3) potentially additional instruction-tuning. The "universal interface" is universal in output format (always RGB) but not in the sense that arbitrary new tasks can be added without training.

Furthermore, all experiments use a single base model (Nano Banana Pro). The paper provides no evidence that the approach works with other image generators (diffusion-based, autoregressive, GAN-based), at different model scales, or with different architectures. A practitioner using a different base model has no guarantee that instruction-tuning will unlock comparable understanding capabilities — the approach may depend on specific properties of Nano Banana Pro's training (e.g., the scale of its training data, the use of natural language captions, the presence of images with depth or segmentation-like visualizations in its training corpus).

What evidence exists in the paper. The paper demonstrates zero-shot transfer to new datasets within the trained tasks (e.g., Cityscapes for semantic segmentation, KITTI for depth), which is evidence of generalization within a task type, but provides no evidence of zero-shot transfer to new tasks. The qualitative examples in Figures 2b and 3b show referring-expression-like behavior on semantic and instance segmentation, which is mild cross-task transfer (language-conditioned segmentation applied to tasks where only class-conditioned segmentation was trained), but this is a narrow form of generalization within the segmentation family, not a test of generalization to entirely new output formats. The paper acknowledges as future work "scaling the diversity of instruction-tuned tasks may unlock further emergent cross-task generalization" (Section 4), which implies the current model does not yet exhibit this capability.

Mitigation status. Not addressed. The paper does not test generalization to held-out tasks, does not evaluate on alternative base models, and does not ablate which properties of the base model are necessary for the approach to work. The future work section mentions task diversity scaling and extension to multi-view and video inputs, but does not propose systematic cross-task generalization experiments. This limitation is significant because it bounds the "universal interface" claim: the evidence supports universality of output format (RGB images can represent many task outputs) but not universality of task learning (new tasks may require new training, not just new prompts).

7. Implications and Future Directions

How This Work Changes the Landscape

This paper instigates a genuine reorientation of the computer vision field's foundational assumptions. The dominant paradigm for decades has treated visual generation and visual understanding as separate problems requiring separate solutions: generation uses generative models (GANs, diffusion, autoregressive), understanding uses discriminative objectives (supervised classification, contrastive learning, masked autoencoding). The two communities share little; a researcher building segmentation models rarely considers whether their model could generate, and vice versa.

Vision Banana challenges this separation at the root. The paper's central empirical result — that a single model achieves state-of-the-art performance on both creative image generation (53.5% win rate vs. its untuned base on GenAI-Bench) and quantitative visual understanding benchmarks (0.699 mIoU on Cityscapes, 0.929 average δ1 on metric depth, 0.793 gIoU on ReasonSeg) without task-specific architectures — demonstrates something the field did not previously know: generative pretraining at scale produces visual representations that are not merely "also useful" for understanding, but are competitive with or superior to the best purpose-built discriminative representations.

This is not an incremental improvement over prior generative understanding work (Baranchuk et al., 2021; Bhattad et al., 2023; Li et al., 2023; Wiedemer et al., 2025; Zuo et al., 2025), all of which showed that generative models possess some understanding but never closed the gap with discriminative specialists. Vision Banana closes this gap on multiple benchmarks simultaneously. The leap from "diffusion models know something about segmentation" to "a generatively pretrained model is the best zero-shot segmenter and depth estimator available" is a qualitative change in the evidence, not a quantitative refinement.

Reconciling prior contradictions. The paper provides a resolution to the long-standing tension between the observation that generative models exhibit emergent understanding behaviors (zero-shot depth maps that look plausible, Wiedemer et al., 2025; Zuo et al., 2025) and the observation that these behaviors do not hold up under rigorous quantitative evaluation. The explanation is an alignment gap, not a capability gap: the base model already knows about geometry and semantics, but it doesn't know which precise color conventions to use so that its outputs can be decoded into evaluable predictions. Instruction-tuning bridges this gap by teaching format compliance — and the fact that a "low ratio" of vision data suffices (Section 2) while preserving generation quality is strong evidence that the underlying capability was already there.

This reframes the research question from "do generative models understand?" to "how do we extract the understanding they already have?" — a shift that redirects research energy from architecting new discriminative objectives toward designing better probing and instruction-tuning strategies for generative models.

A paradigm shift, with boundary conditions. The paper explicitly claims a "major paradigm shift for computer vision" (Section 4). This framing is partially supported but requires qualification. The evidence for a paradigm shift would be: (1) generative pretraining consistently yields better understanding features than discriminative pretraining across tasks, models, and scales; (2) the approach generalizes to new tasks without additional training; (3) practitioners can abandon discriminative pretraining in favor of generative models without penalty. The paper provides evidence for (1) on a single model and seven tasks; does not test (2); and leaves (3) uncertain given the undisclosed training recipe of the base model and the acknowledged inference cost.

What is clearly demonstrated is that the ceiling for generative understanding has been dramatically raised. Prior to this work, a reasonable practitioner could argue that generative models were interesting but practically inferior for understanding tasks — the gap was too large. After this work, a practitioner evaluating options for a visual understanding system must at least consider whether a generatively pretrained foundation model could replace or augment their specialist models. The burden of proof has shifted: it is no longer "prove that generative models can understand" but "prove that your discriminative specialist is worth the additional cost of training and maintaining a separate model."

Research directions that become more attractive. The obvious direction is scaling up: larger generative models, trained on more diverse data, instruction-tuned on more tasks. The paper's results on ReasonSeg (where adding Gemini 2.5 Pro for reasoning pushed performance to 0.793 gIoU, SOTA across all methods including non-zero-shot) suggest that compounding capabilities — generation + reasoning — is a high-leverage path. Research on better instruction-tuning strategies, invertible encodings for new task types, and cross-task generalization becomes directly relevant to improving a single unified system rather than a collection of specialists.

Research directions that become less attractive. The paper implicitly argues against the continued development of task-specific architectures for standard dense prediction tasks. If a generic image generator can match or beat a purpose-built depth estimation model by simply being prompted appropriately, the marginal value of developing yet another depth-specific architectural innovation diminishes. Research effort might better be spent on improving the generative base model, since improvements there flow through to all downstream tasks simultaneously. This does not mean specialist architectures are obsolete — they will remain relevant for efficiency-constrained deployment and for tasks where the RGB encoding approach is inherently awkward (e.g., instance segmentation, where Vision Banana trails DINO-X at 0.540 vs. 0.552 pmF1 under zero-shot). But it does challenge the implicit assumption that each vision task warrants its own dedicated model architecture.

Similarly, the paper's demonstration that metric depth can be estimated without camera intrinsics (Table 3, Vision Banana achieves 0.935 δ1 on ETH3D without intrinsics, while competing methods all require them at training time) suggests that research on intrinsic-conditioned depth models may be solving a problem that large-scale generative pretraining renders unnecessary. The model learns metric scale from visual statistics alone — an unexpected capability that, if reproduced across other generative models, would reduce the need for intrinsics-aware architectures.

The magnitude of the shift depends on reproducibility. The paper's impact hinges on whether its results replicate across base models and whether the approach can be adopted by researchers without access to proprietary generators. If open-weight generative models (e.g., Stable Diffusion variants) can be instruction-tuned to achieve comparable understanding performance, the paradigm shift claim strengthens considerably. If the results depend on undisclosed properties of Nano Banana Pro — its specific architecture, training data composition, or scale — then the contribution is an important existence proof rather than a field-reorienting discovery. The paper's lack of experiments on alternative base models makes this uncertainty the most critical open question for assessing its long-term impact.

Follow-Up Research This Work Enables

Zero-shot transfer to held-out vision tasks as a test of "universal interface." The paper claims image generation serves as a "universal interface" for vision (Section 4) but tests only tasks that were included in instruction-tuning. The most direct test of this claim would be to instruction-tune Vision Banana on tasks A, B, C (e.g., depth, segmentation, normals) and then evaluate zero-shot on a held-out task D that was never in the training mixture — for example, intrinsic image decomposition (albedo vs. shading), edge detection, 2D keypoint estimation, or optical flow. The evaluator would design an invertible RGB encoding for task D (e.g., optical flow as an HSV image where hue encodes direction and value encodes magnitude, with a known conversion formula), craft zero-shot prompts following the same pattern as the trained tasks, and measure whether the model produces decodable outputs that are above random-chance performance. Success would validate the "universal interface" claim; failure would bound it, revealing that the model has learned task-specific formatting rather than a general task-following capability. This experiment directly extends the paper's paradigm analogy to NLP: LLMs that are instruction-tuned on a subset of tasks can perform held-out tasks zero-shot (Wei et al., 2021); does Vision Banana show the same property?

Controlled comparison of pretraining objectives at matched scale. The paper's central causal claim — that generative pretraining specifically causes understanding to emerge — is untested. A minimal experiment to isolate the effect of the pretraining objective would train two Vision Transformer-based models with identical architecture and identical training data (e.g., a fixed corpus of image-text pairs) but different objectives: one as a diffusion model (generative), one as a contrastive SigLIP-style model (discriminative), and one as a masked autoencoder (reconstructive). Scale all to the same parameter count and training FLOPs. Then apply identical instruction-tuning (same vision tasks, same mixing ratio, same RGB encodings) and compare performance across the paper's evaluation benchmarks. If the generative model substantially outperforms the discriminative and reconstructive models, the causal claim is supported. If all three perform similarly, the understanding capability is a function of scale and data diversity, not generation per se. If the discriminative model outperforms, the paper's thesis is contradicted. This experiment is expensive but scientifically decisive.

Systematic ablation of the instruction-tuning mixing ratio. The paper claims that mixing vision data "at a very low ratio" preserves generation and unlocks understanding, but no specific ratio is given and no alternative ratios are tested (Section 2). An ablation sweeping the ratio of vision-to-generation data from 0% (untuned baseline) to 100% (vision-only fine-tuning) in log-spaced increments (e.g., 0.1%, 0.5%, 1%, 5%, 10%, 50%, 100%) would reveal the tradeoff curve. The key measurements: (a) at what minimum ratio does SOTA understanding performance emerge? (b) at what ratio does generation quality begin to degrade measurably? (c) is there a sweet spot where understanding is near-maximal and generation is near-baseline? The answers would validate or constrain the "latent capability" interpretation: if understanding performance saturates at very low ratios (e.g., 0.5%), the capability is likely pre-existing and merely requires format alignment. If performance continues to improve linearly with more vision data, the model may be learning understanding during instruction-tuning rather than unlocking it. The existing evidence — a single unspecified "low ratio" — is insufficient to distinguish these interpretations.

Cross-model reproducibility: instruction-tune alternative image generators. To determine whether the paper's findings generalize beyond Nano Banana Pro, replicate the instruction-tuning protocol on alternative generative models with different architectures, training objectives, and scales. Candidates: a latent diffusion model (e.g., Stable Diffusion 3 or SDXL), an autoregressive visual token model, and a smaller-scale diffusion model to test whether understanding scales with model size or is a threshold phenomenon. For each, measure (a) the understanding performance after instruction-tuning on the same vision tasks and (b) the generation preservation (win rate vs. the untuned base on GenAI-Bench). A positive result across architectures would strongly support the paper's claim that generation, not a specific implementation, drives understanding. A negative result on some architectures would reveal necessary conditions: perhaps only certain types of generative training (e.g., text-conditional diffusion at a minimum scale) produce extractable understanding. This experiment is particularly important for the open-source community, which cannot access Nano Banana Pro.

Can test-time compute improve vision understanding in generative models? The paper uses a single forward pass (one generation) per vision task inference. But generative models have a natural test-time compute knob: the number of denoising steps, or the number of samples drawn and aggregated. For depth estimation, one could sample N depth visualizations from Vision Banana (with different random seeds), decode each to metric depth, and aggregate (e.g., pixel-wise median) to potentially reduce variance and improve accuracy. The paper already notes that generative models "inherently learn the full data distribution, gracefully managing ambiguity by design" (Section 4) — this suggests that multiple samples capture the posterior over valid outputs, and aggregating them could improve metrics like AbsRel (which penalize outliers) even if the mean prediction is similar. An experiment measuring how δ1 and AbsRel improve as a function of N samples (1, 4, 16, 64, 256) on depth estimation benchmarks would quantify whether test-time compute is a viable accuracy-efficiency tradeoff. This connects naturally to the broader literature on inference-time scaling laws for language models.

Design and evaluate encodings for instance segmentation that don't require per-class inference. The paper's weakest result is instance segmentation (0.540 pmF1 vs. 0.552 for DINO-X under zero-shot), constrained by the per-class inference strategy. Research into alternative RGB encoding schemes that can represent an unknown number of instances across multiple classes in a single forward pass would directly address this limitation. Candidates: (a) two-stage generation where the model first outputs a texture map assigning a unique random color to each instance, then a legend image mapping colors to class names; (b) encoding instance IDs as RGB values where the color uniquely identifies the instance, with a separate prompt-to-color registry; (c) leveraging the model's text generation capability (if available) to output instance counts and color assignments before generating the mask. A successful scheme would be measured by pmF1 on SA-Co/Gold and the ability to handle variable instance counts across arbitrary classes in one inference pass. This is a hard problem — the RGB output space fundamentally struggles with unknown-quantity structured outputs — but solving it would remove the last clear domain where specialist architectures outperform the generative approach.

Practical Applications and Downstream Use Cases

Unified perception-generation stack for robotic manipulation. A robot operating in unstructured environments needs both visual understanding (segment the object to grasp, estimate its 3D pose and distance) and, increasingly, the ability to simulate or visualize possible outcomes (what will the scene look like after I move this object?). Vision Banana's single-model architecture means a robotic system could use the same neural network for depth estimation, instance segmentation, and surface normal prediction and for generating goal-state visualizations for planning and human communication — all without loading separate specialist models onto the embedded hardware. The paper's metric depth results are directly relevant: achieving 0.948 δ1 on NYUv2 (indoor environments typical for manipulation) and demonstrated zero-shot transfer to consumer smartphone photos (Figure 7, AbsRel ≈ 0.065 on an in-the-wild measurement) suggests practical reliability for grasp planning where depth accuracy within ~10% at typical manipulation distances (0.3–2 meters) is sufficient. The 3D reconstruction capability demonstrated in Figure 6 shows that the depth maps produce geometrically consistent point clouds when unprojected, meeting the input requirements of standard 3D scene understanding pipelines. The primary deployment hurdle is inference latency: the paper acknowledges that generative models incur "significantly higher computational overhead" than lightweight specialists (Section 4), and a robot requiring real-time perception (30 Hz) may need distillation or acceleration before Vision Banana is deployable.

Single-model pipeline for e-commerce product imagery. E-commerce platforms routinely need to (a) segment products from backgrounds (for white-background product photos, a semantic and instance segmentation task), (b) estimate product depth for 3D visualization features, and (c) generate lifestyle images of products in context (e.g., "show this sofa in a modern living room," a text-to-image editing task). Today, this pipeline uses three or more separate models. Vision Banana offers the prospect of a single model handling all three tasks: a product photo is uploaded, the model segments the product from its background (using a prompt like "Generate a segmentation mask for the sofa, rendered in solid yellow, background black"), estimates depth for 3D rendering (prompt: "Generate a depth map using the standard colormap"), and edits the product into a lifestyle scene (prompt: "Place this sofa in a sunlit living room with hardwood floors"). The paper demonstrates all three capabilities at quality levels competitive with specialists: 0.699 mIoU on Cityscapes (complex urban scenes, harder than typical product images with clean backgrounds), 0.929 average δ1 on depth vs. Depth Anything V3, and 47.8% win rate on image editing vs. the base model. For e-commerce where visual quality directly impacts revenue, the editing win rate near 50% against a high-quality generator is practically significant — it means automated lifestyle imagery is achievable without a separate editing pipeline. The operational benefit is reduced model maintenance (one model version to track instead of three) and simplified infrastructure (one inference server, one set of weights to deploy).

Medical image analysis with verifiable measurement through invertible encodings. Medical imaging workflows often require multiple dense prediction tasks from the same scan: organ segmentation (which regions are liver, kidney, tumor?), depth/disparity estimation (for surgical planning or 3D reconstruction from endoscopy), and surface-based measurements (curvature, normal orientation for implant fitting). The paper's invertible RGB encoding approach has a unique advantage in regulated medical settings: because the model's output is a visualization image that directly maps to physical measurements (depth values, segmentation classes), a clinician can visually inspect the model's output before trusting the decoded measurement. This is an auditability feature that black-box specialist models lack — if a depth regressor outputs "3.2 cm" for a tumor distance, the clinician cannot verify the reasoning; if Vision Banana outputs a depth colormap where the tumor region maps to a specific hue that decodes to ~3.2 meters, the clinician can visually confirm that the colormap looks plausible and that the depth ordering is consistent with anatomical expectations. The paper's depth encoding is explicitly a bijection between metric depth and RGB (Equation 1, with power transform parameters λ = −3, c = 10/3), meaning the mapping is lossless and auditable. The surface normal encoding maps 3D orientation directly to the standard normal-map color convention used in medical visualization tools. A practical deployment would instruction-tune Vision Banana (or an equivalent medical-domain generator) on annotated medical imaging data with the same invertible encodings, then integrate the decoded measurements into existing clinical measurement workflows alongside the generated visualizations for human verification. The regulatory pathway (FDA/CE marking) would need to validate both the model's accuracy on in-domain benchmarks and the invertibility of the encoding — a tractable validation problem given the deterministic mathematical mapping.

Interactive content creation tools for non-experts. Current image editing tools require users to manually select objects (lasso, magic wand) before applying edits, a bottleneck that limits adoption by non-expert users. Vision Banana can accept natural language referring expressions to specify edit targets, demonstrated by the segmentation prompts in Figure 4 ("the man in pink t shirt," "the stretching cat") and the image editing prompts in Figure 10 ("Change the vehicle's color to red," "Change the background of the suit from a blank wall to a luxurious office setting"). A creative tool built on Vision Banana could accept a single prompt like "make the sky more dramatic and remove the person on the left" — the model internally performs referring expression segmentation to identify the person and sky, then edits the identified regions, all within a single generative process. The paper's results on referring expression segmentation (0.738 cIoU on RefCOCOg, 0.793 gIoU on ReasonSeg) are state-of-the-art, suggesting the model can accurately identify user-specified regions from casual language descriptions. The image editing preservation at 47.8% win rate against the base model indicates that the editing quality is not degraded by the instruction-tuning. For a creative tool, this means users can specify edits in natural language without learning selection tools, with the model handling both the "what to edit" and "how to edit it" in a unified pipeline. The primary deployment consideration is user tolerance for latency — a single generation pass of a large diffusion model takes seconds rather than the milliseconds of traditional selection tools.

When to Prefer This Method

The paper positions Vision Banana explicitly against both specialist discriminative models (e.g., SAM 3, Depth Anything V3) and prior generative-to-understanding adaptation methods that use full fine-tuning or task-specific heads (e.g., Marigold, Diception, InstructCV). The tradeoffs are articulated clearly enough to extract a decision framework:

Prefer the Vision Banana approach (instruction-tune a generative model with low-ratio vision data, keeping RGB output) when: (1) you already have access to a high-quality generative base model whose training cost is sunk; (2) you need a single model to handle multiple diverse vision tasks plus generation, eliminating the need for separate specialist deployments; (3) the tasks of interest map naturally to dense per-pixel outputs with known output structure (semantic segmentation, depth, normals, referring expression segmentation); (4) you value auditability — the generated RGB visualization can be inspected by humans before the decoded measurement is used; (5) inference latency is not the primary constraint (the paper acknowledges the computational overhead in Section 4). The paper's evidence supports this preference: a single model achieved SOTA on 6 of 7 evaluated tasks while retaining generation quality.
Prefer specialist discriminative models when: (1) you are deploying in a compute- or latency-constrained environment where the inference overhead of a generative model is prohibitive — lightweight specialists like Depth Anything V3 or efficient SAM variants are orders of magnitude faster; (2) the task requires output formats that are inherently awkward as RGB images, most notably instance segmentation with unknown instance counts across many classes (Vision Banana's 0.540 pmF1 trails DINO-X at 0.552, and the per-class inference strategy scales poorly); (3) you need to train a model from scratch on a limited budget — pretraining a large generative model is enormously expensive, and if you don't already have the base model, the total cost (pretraining + instruction-tuning data curation + instruction-tuning) may exceed specialist training; (4) you need guaranteed SOTA on a single task and can afford to build a dedicated pipeline — Vision Banana is competitive but not universally dominant (it trails DINO-X on instance segmentation and Lotus-2 on outdoor surface normals when the specialist has in-domain training).
Prefer full fine-tuning of generative models (e.g., Marigold, Diception) when: (1) you need maximal performance on a single understanding task and are willing to sacrifice generation capability and cross-task generality; (2) you can modify the architecture with task-specific heads. The paper's ablation on this is implicit — it cites prior full-fine-tuning work as losing generation but does not provide its own direct comparison. The tradeoff is qualitatively clear: full fine-tuning may squeeze out extra task-specific performance at the cost of the universal interface, while low-ratio instruction-tuning under the Vision Banana approach preserves generality. A practitioner should measure this tradeoff on their specific task and base model, since the paper does not provide a controlled comparison for Nano Banana Pro specifically.