DreamOmni3: Scribble-based Editing and Generation

🎯 Pitch

Telling an AI to 'edit the third bag from the left' is a recipe for disaster—so DreamOmni3 instead lets you just draw a red circle around it, matching commercial models like GPT‑4o while blowing past open‑source baselines. The secret is a joint input scheme that gives the model both the original and annotated images, preserving unedited regions with far higher fidelity than mask‑based methods.

1. Executive Summary

This paper introduces scribble-based editing and generation, two tasks that augment unified image editing models with freehand sketch inputs to specify edit locations and fine-grained visual details that text alone cannot capture, alongside a comprehensive data synthesis pipeline and the DreamOmni3 framework built on DreamOmni2. DreamOmni3 employs a joint input scheme (feeding both the original source image and the scribbled source image into the model with shared index and position encodings) that avoids the complexity of binary masks while preserving pixel-level consistency in non-edited regions. On the DreamOmni3 benchmark of real-world images, the model achieves a 57.50% human-evaluated editing success rate, substantially outperforming open-source baselines (DreamOmni2 at 17.50%) and matching commercial models (GPT-4o at 58.75%), while for generation it reaches 53.49% human-evaluated success rate versus 23.26% for Nano Banana and 39.53% for GPT-4o. The joint input scheme proves critical for editing consistency, with shared encodings improving editing success from 35.00% to 45.00%, establishing that precise scribble-based localization requires the model to see both the annotated and original image—an effect that is pronounced for editing tasks requiring pixel-level preservation but negligible for generation tasks where such consistency is unnecessary.

2. Context and Motivation

The Core Problem: When Language Alone Isn't Enough

Current unified image generation and editing models—such as DreamOmni2, Omnigen2, Qwen-image-Edit, and commercial systems like GPT-4o and Nano Banana—have achieved remarkable success by combining text prompts with reference images to perform instruction-based edits. These models represent a significant advance: a single model that can both generate images from scratch and edit existing ones, guided by natural language instructions and multimodal inputs. However, the paper identifies a fundamental limitation in this paradigm: language is an impoverished medium for specifying spatial locations and fine-grained visual details.

This limitation manifests in several concrete failure modes that arise in real-world usage:

Spatial ambiguity. When a user wants to edit a specific region of an image, describing that region in words is often awkward or impossible. For example, "the third person from the left, slightly behind the table" requires the model to parse complex spatial relations that natural language handles imprecisely. If there are multiple similar objects in the scene—multiple people, multiple bags, multiple windows—disambiguation through text alone becomes combinatorially difficult. The paper notes that "certain editing or generation positions in the image are difficult to describe with language" and "there may be multiple identical objects that are hard to distinguish" (Section 1).

Object identification gaps. Users frequently encounter objects they cannot name. A specific decorative pattern, an unusual piece of furniture, a distinctive architectural feature—these resist verbal description but would be trivial to indicate with a rough circle drawn on the image. The paper explicitly flags this: "some objects may be hard for users to identify by name" (Section 1). This is not a rare edge case; it's a routine experience for anyone who has tried to describe visual content using only words.

Detail granularity. Even when an object can be named, the user's intended edit may involve visual subtleties—a specific color gradient, a particular fabric texture, a precise hairstyle—that language struggles to capture efficiently. A user might want to "make this area look like that area," which requires combining spatial reference with visual reference in ways that pure text instructions handle poorly.

Creative expression through drawing. Beyond mere positional annotation, users often want to express creative intent through drawing—adding a rough sketch of a dragon in a specific pose, indicating the shape of a desired shadow, or doodling the outline of an object to be inserted. These are inherently visual specifications that cannot be reduced to text without losing the creative nuance. The paper frames this as enabling "more flexible and creative additions, deletions, or modifications to the content in the image, such as through drawing" (Section 1).

Why This Problem Matters

The gap between what users want to express and what text-only interfaces allow has both practical and theoretical significance.

Practical impact on creative tools. The paper positions itself in the lineage of graphical user interface (GUI) evolution. Professional creative tools (Photoshop, Procreate) have long supported brush-based selection, masking, and sketching as primary interaction modes—not because users prefer complexity, but because spatial tasks require spatial input. AI-powered editing tools that force users through a text bottleneck are, from this perspective, a regression in interface design. The paper's vision is to bring the expressiveness of direct manipulation back into AI-assisted creation: "users can roughly circle a region, and the model accurately understands both the intent and the editing scope, providing a much more user-friendly experience" (Section 2, comparing with mask-based methods).

This matters because the target audience for unified editing models is not just AI researchers testing benchmarks—it's designers, photographers, content creators, and casual users who want intuitive, efficient tools. A model that requires painstaking text descriptions for every spatial operation imposes a cognitive tax that limits adoption and creative flow. Scribble-based interaction lowers this barrier by letting users communicate spatially in the most natural way: by pointing and drawing.

Enabling interactive creation workflows. The paper emphasizes that scribble-based editing and generation "can be combined with language and image instructions, enhancing the creative usability of unified models" (Section 1, first contribution bullet). This isn't about replacing text with scribbles—it's about giving users a multimodal vocabulary where they can mix text, reference images, and freehand sketches in whatever combination best expresses their intent. For instance, a user might circle a region, provide a reference image of a desired object, and add text like "make it match the lighting"—three modalities combining to specify what none could alone.

Bridging generation and editing. A unified model that understands scribbles for both editing (modifying existing images) and generation (creating images from scratch with spatial guidance) creates a seamless creative experience. The same scribble vocabulary works whether you're touching up a photo or designing from a blank canvas. This unification, the paper argues, is what makes scribble-based interaction not just a feature but a fundamental interface primitive for AI creation tools.

Theoretical significance for multimodal understanding. Beyond interface design, scribble understanding represents a genuine test of multimodal comprehension. A model that interprets a rough hand-drawn circle as referring to "the bag, not the person holding it" must integrate visual grounding, spatial reasoning, and natural language understanding. The paper positions this as part of the broader trajectory toward models that "respond to real-world visuals, marking progress toward world models and AGI" (Section 1). Scribbles are informal, ambiguous, and require common-sense interpretation—precisely the kind of signal that separates superficial pattern matching from genuine understanding.

Where Prior Approaches Fall Short

The paper identifies three categories of prior work, each with specific limitations that DreamOmni3 addresses:

Mask-Based Editing: Precise but Impractical

Mask-based editing methods—where users paint a binary mask over the region to edit, and the model regenerates that region—represent the most direct prior approach to spatial specification. The paper divides these into three subcategories (Section 2):

Image inpainting [22, 23, 25–27]: The user masks a region, and the model fills it based on an instruction. Simple, but the regenerated region loses all original structure and color (the model treats the masked area as entirely missing).
Auxiliary-guided inpainting (MagicQuill [8]): To address the structure-preservation problem, additional guidance signals—edge maps or low-resolution images—are fed alongside the mask to help the model maintain contours and color consistency from the original content.
Object insertion via compressed IDs [9, 13, 14]: Reference objects are encoded as compressed embeddings and injected into masked regions to insert specific content from another image.

DreamOmni3 identifies four fundamental shortcomings in this family of approaches:

(a) Semantic blindness to context. Traditional inpainting datasets are constructed by literally masking out portions of images. When the model learns to fill these masks, it learns a local texture-completion task, not a semantically-aware editing task. The paper argues: "Traditional inpainting datasets are created by simply masking parts of an image, which prevents the model from reasoning about environmental effects such as lighting or shadow changes" (Section 2). If you mask a person and tell the model to replace them with a lamp, the model has no training signal about how the lamp should cast light on surrounding surfaces—because the training data never contained such semantic changes. DreamOmni3's dataset, built from instruction-based editing pairs (whole images transformed, not individual regions masked), captures these contextual effects.

(b) Mask precision requirements. Inpainting methods are notoriously sensitive to mask accuracy: if the mask is slightly too large, content outside the intended edit region is destroyed; if it's too small, the edit bleeds or appears as an obvious patch. MagicQuill mitigates this with edge-map guidance but still "requires precise manual masking" (Section 2). This demands careful brushwork from users—exactly the kind of painstaking interaction that AI tools should eliminate. DreamOmni3's key design claim is that "users can roughly circle a region, and the model accurately understands both the intent and the editing scope" (Section 2), shifting the burden of precision from the user's hand to the model's understanding.

(c) Multi-region complexity. When a user wants to edit multiple regions simultaneously—"change the color of this bag, remove this person, and add a window over here"—binary mask approaches require a separate mask for each region. With three edits, that's three masks plus an instruction that must link each mask to its corresponding edit ("mask 1 → change color, mask 2 → remove"). This linking problem becomes combinatorially difficult as the number of edits grows. The paper notes: "Binary masks become problematic when there are multiple doodles in the reference or source image, as each doodle needs a separate mask, significantly increasing computational load. Moreover, using language to link doodles in both images is challenging with binary masks" (Section 3.2).

(d) Format fragmentation. Different mask-based methods use different input formats—some expect masks, some expect edge maps, some expect compressed embeddings—making them "hard to unify under a single framework" (Section 2). DreamOmni3's approach of drawing directly onto the image in RGB space creates a uniform input format that integrates naturally with existing vision-language models trained on standard image inputs.

Mask-Based Generation: Rigid Spatial Control

For generation tasks—creating images from scratch with spatial constraints—prior work used masks or bounding boxes to control where content appears:

Training-free methods (Multi-diffusion [19]): Attempted multi-region generation without additional training, but results were unstable because the base text-to-image model had no mechanism for attending to spatial constraints.
Attention-based grounding (GLIGEN [6], InstanceDiffusion [17], MIGC [30], Eligen [29]): Introduced specialized attention modules and encoded bounding boxes (e.g., via Fourier embeddings) to ground generation to specific regions. Later works refined attention mechanisms and input granularity for increasingly precise control.
Subject-driven grounded generation (MS-diffusion [18]): Used a grounding resampler to associate visual features from reference subjects with specific spatial locations, enabling "generate this specific dog in this box."

The paper identifies a persistent problem across these methods: overcomplexity in both input format and inference process (Section 2). Users must provide bounding boxes or precise masks, the model architecture requires specialized grounding modules, and the inference pipeline often involves multiple stages (generate per-region, then harmonize). DreamOmni3's counterproposal is radical in its simplicity: let users draw colored circles and boxes on a canvas, and train the model to interpret these as spatial instructions—no special modules, no separate mask channels, no multi-stage pipelines. The model sees the scribbled canvas as an ordinary image and generates accordingly.

Text-Only Unified Models: The Baseline Gap

The most relevant prior work is the generation of unified editing models that DreamOmni3 builds upon—specifically DreamOmni2, Omnigen2, Qwen-image-Edit, and commercial systems like GPT-4o and Nano Banana. These models can already handle text + image instructions for editing and generation. However, they lack any mechanism for spatial input:

DreamOmni2 [24] supports multi-reference image generation and editing with language and image instructions but has no scribble understanding capability. On the DreamOmni3 benchmark, it achieves only 17.50% human-evaluated editing success rate and 4.65% generation success rate (Tables 1 and 2)—confirming that scribble-based tasks are outside its trained capabilities.
Omnigen2 [21] and Qwen-image-Edit-2509 [20] show similarly poor performance on scribble tasks (5.00% and 16.25% human-evaluated editing success, respectively), with the paper noting that their outputs "often retain scribbles" (Section 4, discussion of generation results).
GPT-4o and Nano Banana perform better (58.75% and 41.25% editing success) but exhibit specific failure modes: GPT-4o "frequently exhibits a yellowing issue in the images, and the pixels in the non-edited areas often do not match those in the input image," while Nano Banana shows "copy-and-paste effects and incorrect object proportions" (Section 4). Critically, these commercial models have not disclosed whether they were explicitly trained for scribble tasks—their moderate performance may come from general reasoning capabilities rather than dedicated optimization, and the paper notes they "are not specifically optimized for scribble scenarios, and even when explicitly instructed not to generate scribble marks, these models still often output scribbles along with the generated results" (Section 4).

The baseline results establish that scribble understanding is not an emergent capability that arises automatically from training on text + image tasks—it requires dedicated data, training, and architectural design. Models that achieve strong text-based editing performance (DreamOmni2, Omnigen2) collapse entirely when asked to interpret scribbles, demonstrating that spatial reasoning from visual annotations is a distinct capability.

How DreamOmni3 Positions Itself

The paper's positioning strategy is multi-layered:

First, as a task definer rather than just a model proposer. The paper doesn't simply introduce a new model—it defines a new class of tasks (scribble-based editing and generation) with clear taxonomies (Section 3.1, Figure 2). Scribble-based editing is decomposed into four sub-tasks (scribble + instruction, scribble + multimodal instruction, image fusion, doodle editing); scribble-based generation into three (scribble + instruction, scribble + multimodal instruction, doodle generation). This taxonomy serves dual purposes: it guides data creation (each task requires different data construction pipelines) and establishes a vocabulary for the research community to discuss, compare, and build upon scribble-based interaction.

Second, as filling the missing modality in unified models. The paper explicitly positions itself as extending—not replacing—existing unified editing frameworks: "DreamOmni3, a model that integrates our proposed scribble-based editing and generation with the existing unified editing and generation framework" (Section 1). This is an integration strategy, not a clean-slate approach. By building on DreamOmni2's architecture and training (using the same VLM, same base diffusion model, and LoRA-based fine-tuning that preserves original capabilities), DreamOmni3 adds scribble understanding as an additional modality while maintaining full backward compatibility with text-only and image-only editing. This is practically important: users don't have to choose between a scribble model and a text model—they get both in one system.

Third, as a data-centric contribution. The paper acknowledges that "the biggest challenge in scribble-based editing and generation is the lack of data" (Section 3.1) and devotes substantial effort to the data synthesis pipeline. The approach is pragmatic and repurposing: instead of collecting expensive human-annotated scribble data, the paper uses existing DreamOmni2 editing pairs and retroactively adds synthetic scribbles using a Refseg (referring segmentation) service to locate edited objects, then overlays hand-drawn-style circles and boxes from a template library of "30 different squares and circles" (Section 3.1). This curation is non-trivial: the authors deliberately avoid using Canny edge detection for doodle generation because "users' imperfect drawings... require aesthetic corrections from the model" (Section 3.1), instead using the GPT-Image-1 model to produce "doodles with non-strict pixel consistency" that better match what real users would draw.

Fourth, as proposing a simpler alternative to binary masks. The architectural contribution—joint input of source image and scribbled source image with shared encodings—is framed as a response to the complexity limitations of mask-based approaches. The paper argues that this design is not arbitrary but the result of a deliberate tradeoff analysis: binary masks handle one region cleanly but break down with multiple regions (each needing its own mask channel) and are incompatible with existing RGB-trained models; joint input in RGB space handles arbitrarily many colored scribbles, leverages pre-trained vision-language understanding, and integrates seamlessly with existing architectures. The shared index and position encoding scheme (both images get the same (x, y, 1) encoding when the scribbled image is the second input) is presented as the specific innovation that makes joint input work—without it, the model struggles with pixel-level alignment between the scribbled and original images.

Fifth, as establishing infrastructure for a nascent research direction. The paper creates not just a model but an ecosystem: a training dataset (~117K total samples across editing and generation tasks), a benchmark of real-world images with multiple task types, and a VLM-based evaluation protocol with human validation. This infrastructure-building is positioned as critical for "fostering the development of this valuable direction" (Section 3.3), implicitly arguing that the field needs standardized evaluation before progress can be reliably measured.

The Deeper Motivation: Reuniting GUI Interaction with AI Capability

Reading between the lines, the paper is motivated by a vision of creative AI tools that feel like natural extensions of how humans already interact with visual content. Professional artists sketch. Designers circle things. Children draw. These are universal, intuitive modes of visual communication that predate language—yet current AI editing tools force users to translate visual intent into text, losing precision and creative nuance in the process.

DreamOmni3's scribble-based interaction is not just about adding another input modality—it's about closing the loop between human spatial intuition and machine visual understanding. A user who can circle a region and say "make this look like that" is communicating in a way that leverages the full bandwidth of human visual cognition, not compressing it through a linguistic bottleneck. This matters because the goal of unified editing models is not to build better benchmarks but to create tools that amplify human creativity. Scribbles, in this framing, are not a feature—they're the missing half of the conversation between human and AI in visual creation.

3. Technical Approach

3.1 Reader Orientation

DreamOmni3 is a unified image editing and generation model that understands freehand sketches (scribbles) drawn by users to specify where and what to edit, layered on top of an existing multimodal editing framework. The core problem it solves is that language alone cannot precisely describe spatial locations or fine-grained visual details in images — scribbles fill this gap by letting users circle, box, or doodle directly on the image, and the model must interpret these informal annotations while preserving pixel-perfect consistency in unedited regions.

3.2 Big-Picture Architecture (Diagram in Words)

The DreamOmni3 system has five major components connected in a feedforward pipeline:

Scribble Overlay Engine — takes a source image and a user's freehand drawing (circles, boxes, doodles in different colors), and produces a scribbled source image where the annotations are painted directly onto the RGB pixels. This is done at data creation time (synthetic scribbles from template libraries) and at inference time (user-drawn scribbles).
Joint Input Module — feeds both the original source image and the scribbled source image into the model simultaneously, along with optional reference images and a text instruction. This dual-input preserves the original pixel information that would otherwise be occluded by the scribbles.
VLM Text Encoder (Qwen2.5-VL 7B) — processes the editing instruction text and produces token embeddings. The VLM provides multimodal understanding that connects textual descriptions to visual content.
MM-DiT Backbone (FLUX Kontext) — the main diffusion transformer that takes all image tokens (original source, scribbled source, reference images), text tokens, and noise, then denoises to produce the output image. All image tokens share a unified index and position encoding scheme to maintain spatial alignment.
LoRA Adapters — lightweight trainable modules (rank 256) inserted into the frozen Kontext backbone. Separate LoRAs are trained for editing tasks and generation tasks, preserving the original Kontext capabilities while adding scribble understanding. The adapters are activated only when scribble inputs are present.

Information flow: A user's scribbled image and optional reference images enter the VAE for tokenization → text instruction enters the VLM for encoding → all tokens are concatenated with shared spatial encodings → the MM-DiT denoises from random noise conditioned on this multimodal context → the VAE decoder produces the final edited or generated image. The joint input scheme ensures the model sees both the scribble annotations (for localization) and the clean source pixels (for consistency preservation) at every denoising step.

3.3 Roadmap for the Deep Dive

First, the data synthesis pipeline for all seven sub-tasks (four editing, three generation) — because DreamOmni3's capabilities are fundamentally data-constrained, and understanding how the training pairs are constructed reveals which capabilities the model actually learns versus what it merely inherits from DreamOmni2.
Second, the joint input scheme — the central architectural innovation that distinguishes DreamOmni3 from mask-based approaches. We'll examine why binary masks fail for multi-region editing and how RGB-space scribbles with dual-image input solve this.
Third, the index and position encoding scheme — the specific mechanism that makes joint input work by teaching the model that the scribbled and original images occupy the same spatial coordinate system.
Fourth, the training configuration — LoRA rank, base models, training duration, and how the model preserves DreamOmni2's original capabilities while acquiring scribble understanding.
Fifth, the benchmark and evaluation framework — how the model is assessed on real-world images with VLM-based automated evaluation validated against human judgments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data-and-architecture paper whose core idea is that scribble-based interaction can be added to existing unified editing models by (1) synthesizing training data that pairs scribbled images with edited outputs, and (2) designing a joint input scheme with shared spatial encodings that lets the model see both the scribble annotations and the original pixels simultaneously.

Data Synthesis Pipeline for Scribble-Based Editing

The central practical challenge is that no existing dataset contains triples of (source image, scribbled source image, editing instruction, target image). The paper solves this by retrofitting scribbles onto existing editing pairs from DreamOmni2's multimodal instruction editing dataset. The key insight is that if you already know what was edited and where (from DreamOmni2's paired data), you can automatically draw synthetic scribbles at those locations and create training data for scribble understanding.

The data pipeline for the four editing sub-tasks operates as follows (Section 3.1, Figure 2a):

Task 1 & 2: Scribble + Multimodal Instruction Editing and Scribble + Instruction Editing

These two tasks share a common data creation procedure, differing only in whether a reference image is included. The pipeline proceeds in four stages:

Stage 1: Locate edited objects. A Refseg (referring segmentation) service is applied to both the target image (the desired output) and any reference images. For each edited object mentioned in the instruction, Refseg produces bounding box coordinates and a segmentation mask identifying the object's location and extent. This step answers the question: "where in the image did the edit happen?" The paper does not detail which Refseg implementation is used, but the requirement is that it can map natural language object descriptions (from the original DreamOmni2 instructions) to pixel-level locations.

Stage 2: Build a scribble symbol library. The authors manually created 30 different squares and circles to serve as a template library of hand-drawn-style annotations. This library is explicitly designed to mimic imperfect user drawings — the shapes are not geometrically precise rectangles or ellipses but have the slight irregularities characteristic of freehand drawing. The paper's rationale is that models trained on perfect shapes would fail on real user input, so the training data must already contain the variability of human drawing.

Stage 3: Create scribbled reference images. For each reference image, a symbol is randomly selected from the template library and resized to match the scale of the located object. The symbol is then pasted onto the reference image at the object's position, creating an S-Reference Image (scribbled reference). The random selection ensures the model learns that any drawn shape — regardless of its exact appearance — indicates the relevant region, preventing overfitting to specific scribble styles. Multiple reference images (up to three, per the DreamOmni2 dataset) each get their own scribble annotations.

Stage 4: Create scribbled source images and training pairs. The target image (the desired final output) is modified by removing or altering the edited object to create the source image. Then a corresponding scribble symbol is pasted onto this source image at the same location, creating the S-source image (scribbled source). The training pair is: input = (S-source image, optional S-reference images, text instruction), output = (target image).

A critical detail: the paper notes that the editing categories encompass "a wide range of both concrete objects and abstract attributes, as demonstrated in DreamOmni2" (Section 3.1). This means the scribble training data includes not just "add a dog here" (concrete object) but also "change the design style here" and "modify the color scheme here" (abstract attributes). The Refseg service must therefore localize not just object instances but also regions associated with abstract properties — likely by segmenting the affected area in the target image and mapping it back to the source.

The instruction for Task 1 (multimodal) references both the scribbled source and scribbled reference images, e.g., "Make the bag in the red circle of the first image have the same color scheme as the printer in the red circle of the second image." The instruction for Task 2 (instruction-only) omits the reference image and instead describes the desired edit directly, e.g., "Add a black car inside the blue circle."

Dataset size: Scribble-based multimodal instruction editing contributes 32K training samples; scribble-based instruction editing contributes approximately 14K training samples (Section 3.1).

Task 3: Image Fusion

Image fusion is the task of extracting an object from a reference image and inserting it into a specific location in the source image. Unlike Tasks 1 and 2 where the model learns to generate the edit content, here the model must learn to copy and composite visual content from another image into the scribbled region.

The data creation pipeline:

Step 1: Remove the target object from the target image. The target image (containing the object to be inserted) is processed through an instruction-based editing model to remove the object, producing a version where that object is absent. This serves as the source image — the "before" state that the model will learn to add the object back into.

Step 2: Extract the object from the reference image. The Refseg service locates and segments the object in the reference image. The segmented object is cropped and resized to fit the corresponding position in the source image.

Step 3: Paste onto the source image. The cropped object is pasted onto the source image at the appropriate location. A scribble symbol is added at the same position to indicate where the fusion should occur. The training pair is: input = (source image with pasted object removed, source image with scribble, reference image), output = (source image with object inserted at scribble location).

The paper notes that image fusion "primarily focuses on the task of adding concrete objects into the image" (Section 3.1), not abstract attributes. This is because abstract properties (color schemes, styles) cannot be cleanly "extracted" from one image and "pasted" into another — they require the model to understand and transfer semantic concepts, which is covered by Task 1.

Dataset size: 16K training samples.

Task 4: Doodle Editing

Doodle editing is the most creative of the editing tasks: the user draws a rough sketch of what they want, and the model transforms it into a photorealistic or styled object that matches the surrounding image context.

The data creation pipeline differs from Tasks 1-3 in one crucial respect: instead of pasting template circles and boxes, the system generates abstract sketches from the target objects:

Step 1: Locate the editing object. Refseg identifies the object in the target image.

Step 2: Convert to a doodle. The located object is fed into an instruction-based editing model — specifically, GPT-Image-1 — which transforms it into "a simple abstract doodle." The paper explicitly rejects using Canny edge detection for this step because edge maps produce precise, high-frequency line drawings that do not match what humans actually sketch. The authors argue that "users' imperfect drawings... require aesthetic corrections from the model" (Section 3.1), and GPT-Image-1 produces doodles "with non-strict pixel consistency" — meaning the doodle captures the semantic essence and rough shape of the object without pixel-perfect alignment to the original contours. This trains the model to interpret loose, expressive sketches rather than requiring precise tracing.

Step 3: Remove the original object and paste the doodle. The source image is created by removing the object from the target image, then pasting the GPT-Image-1-generated doodle at that location. The training pair is: input = (source image with doodle, text instruction), output = (target image).

The instruction describes what the doodle represents, but the doodle itself provides the spatial and shape information. The model must learn to "fill in" the doodle — interpreting its rough shape and generating a realistic object that fits the surrounding context in lighting, perspective, and style.

Dataset size: 8K training samples.

Data Synthesis Pipeline for Scribble-Based Generation

Scribble-based generation is the blank-canvas analog of editing: instead of modifying an existing image, the model generates an entirely new image from scratch, guided by scribbles drawn on a white background. The data pipeline mirrors the editing pipeline with one critical substitution: the source image is replaced with a blank white canvas (Section 3.1, Figure 2b).

Task 1 & 2: Scribble + Multimodal Instruction Generation and Scribble + Instruction Generation

The procedure follows the same Refseg-based localization as editing Tasks 1 and 2, but with the scribble symbols pasted onto a white canvas rather than onto an image:

Step 1: Locate objects in the target image. Refseg identifies all objects that should be generated, along with their positions and sizes.

Step 2: Paste scribbles onto a white canvas. For each located object, a randomly selected scribble symbol from the 30-template library is placed at the corresponding position on a white canvas. The color of each scribble (red circle, blue square, etc.) serves as an identifier that the instruction can reference.

Step 3: Create the training pair. Input = (white canvas with scribbles, optional reference images with scribbles, text instruction). Output = (target image with all objects generated at the specified positions).

The instruction for multimodal generation might read: "In the living room, the woman from image 1 is standing in the red circle, and the cat from image 2 is sleeping in the black square, resting on the table." The model must generate not just the specified objects, but also the background context (the living room, the table) and ensure all elements are coherent in lighting, perspective, and spatial arrangement. This is substantially harder than scribble-based editing because there are no background pixels to preserve — every pixel must be generated from scratch while respecting the spatial constraints.

For instruction-only generation, the reference images are removed and object descriptions are added to the instruction instead, e.g., "A man is in the red circle, and a woman with long hair is in the green circle. They are shaking hands, with a volcano in the background."

Dataset sizes: Multimodal instruction generation has 29K training samples; instruction generation has 10K training samples.

Task 3: Doodle Generation

Doodle generation is the blank-canvas analog of doodle editing: the user draws rough sketches on a white background, and the model generates a complete image where those sketches become photorealistic objects in a coherent scene.

The pipeline mirrors the doodle editing pipeline:

Step 1: Locate objects in the target image using Refseg.

Step 2: Convert each object to a doodle using GPT-Image-1, producing abstract sketches with non-strict pixel consistency.

Step 3: Paste the doodles onto a white canvas at the same positions they occupy in the target image.

The training pair is: input = (white canvas with doodles, text instruction describing what to generate), output = (target image). The model must learn to interpret the rough sketches as shape and position guides while generating appropriate textures, colors, lighting, and background context to complete the scene.

Dataset size: 8K training samples.

Total dataset composition (summarized from Section 3.1):

Task	Sub-task	Training samples
Scribble-based Editing	Multimodal instruction	32K
	Instruction-only	~14K
	Image fusion	16K
	Doodle editing	8K
	Editing subtotal	~70K
Scribble-based Generation	Multimodal instruction	29K
	Instruction-only	10K
	Doodle generation	8K
	Generation subtotal	~47K
Total		~117K

Joint Input Scheme

The joint input scheme is the central architectural innovation of DreamOmni3. It addresses a fundamental tension in scribble-based editing: scribbles necessarily occlude parts of the source image, but editing requires preserving pixel-perfect consistency in non-edited regions. If the model sees only the scribbled image, it lacks the original pixel information for the regions under and around the scribbles. If it sees only the original image, it lacks the spatial annotation. The joint input scheme resolves this by feeding both images simultaneously (Section 3.2).

The binary mask alternative and why it is rejected. The paper considered using binary masks (as in traditional inpainting) where a separate mask channel indicates which pixels to edit. This approach is rejected for two reasons:

Multi-region complexity. When there are multiple scribbles (e.g., a red circle for one edit, a blue box for another, a green doodle for a third), binary masks require a separate mask channel for each annotated region. This scales poorly: three edits mean three additional input channels. Moreover, the instruction must link each mask to its corresponding edit ("mask 1 → change color to purple, mask 2 → remove, mask 3 → insert cat"), which introduces a text-to-mask correspondence problem that natural language handles poorly. The paper argues that "using language to link doodles in both images is challenging with binary masks" (Section 3.2).
Architectural incompatibility. Existing unified generation and editing models (DreamOmni2, Kontext, FLUX) are trained on RGB image inputs. Adding binary mask channels requires modifying the VAE encoder to accept 4+ channel inputs and retraining or fine-tuning the entire diffusion backbone on this new input format. This breaks compatibility with the pre-trained weights and the original editing capabilities. In contrast, joint input feeds two standard RGB images into the standard VAE — no architecture modifications needed.

How joint input works (Section 3.2, Figure 2c). The scheme is straightforward: both the original source image and the scribbled source image are tokenized by the VAE (producing two sets of latent tokens), and both sets of tokens are concatenated into the input sequence to the MM-DiT. The model sees:

IMG1 tokens: the original source image (unmodified pixels).
Scribble IMG1 tokens: the same source image but with colored scribbles overlaid at edit locations.

Both token sets have the same spatial dimensions (they encode the same image at the same resolution). The scribbled version tells the model where to edit; the original version tells the model what the pixels look like underneath the scribbles, enabling it to preserve those pixels in the output.

When joint input is used and when it is not (Section 3.2). The paper applies joint input selectively based on whether pixel-level preservation is needed:

Editing tasks with scribbled source images: Joint input is always used because non-edited regions must remain pixel-identical to the original source image. The scribbles occlude portions of the source image, so the model needs the clean version to know what to preserve.
Reference images with scribbles: Joint input is not used for reference images, even when they contain scribbles. The reason is that reference images are consulted for content (object appearance, color, style), not for pixel-level output consistency. There is no requirement that the output image match the reference image's non-annotated pixels — the reference is an informational source, not a consistency target. Adding joint input for each reference image would "unnecessarily increase computational cost" (Section 3.2) by doubling the number of image tokens the model must process.
Scribble-based generation tasks: Joint input is not used because generation creates images from scratch — there are no source pixels to preserve. The scribbled canvas (white background with scribbles) contains all the needed information, and feeding an additional blank white canvas would add no useful signal.

The computational implication. Joint input doubles the token count for the source image (from N tokens to 2N tokens). Since the source image is usually one of several images in the input (source + 1-3 reference images), the relative increase in total token count depends on the number of reference images. For an editing task with one reference image, joint input increases total image tokens from 2N (source + reference) to 3N (original source + scribbled source + reference) — a 50% increase. The authors accept this cost as necessary for editing consistency.

Index and Position Encoding Scheme

Simply concatenating the original and scribbled source image tokens is not sufficient — the model must understand that these two token sequences represent the same image, not two different images. Without this understanding, the model might treat them as separate views, fail to align their spatial coordinates, or confuse which pixels correspond to which. The paper introduces a specific encoding scheme to establish this correspondence (Section 3.2).

The encoding problem. In DreamOmni2's architecture, each input image receives an index encoding that distinguishes it from other images (e.g., "this is image 1, this is image 2, this is image 3") and a position encoding that encodes spatial coordinates (x, y) within each image. If the scribbled source image were treated as a distinct image with a new index, the model would have no explicit signal that its pixels correspond to the same spatial locations as the original source image. It would have to learn this correspondence implicitly from training data — possible but inefficient.

The solution: shared encodings. Both the original source image and the scribbled source image receive the same index encoding and the same position encoding (Section 3.2). Concretely, if the source image is encoded as (x, y, 1) — where (x, y) is the 2D position in the image grid and 1 is the image index — then the scribbled source image is also encoded as (x, y, 1). The model sees two tokens at the same position with the same image index, differing only in their content (pixel values at that position).

Why same index encoding matters. The index encoding signals "which image does this token belong to?" By giving both the original and scribbled source the same index, the model understands they are two representations of the same entity. This prevents the model from treating the scribbled source as yet another reference image (which would receive its own distinct index, e.g., (x, y, 2)).

Why same position encoding matters. The position encoding signals "where in the image is this token?" By giving both the original and scribbled source the same spatial coordinates, a token at position (37, 82) in the scribbled image is explicitly aligned with the token at position (37, 82) in the original image. This grid-level correspondence tells the model: "the pixel content at this location differs between these two tokens (one has a scribble, one doesn't), but they represent the same spatial position." This makes it straightforward for the model's attention mechanism to compare the scribbled and original versions at each spatial location and decide where edits should occur.

Impact on subsequent reference images. An important side benefit of shared encodings: because the original and scribbled source share the same index, subsequent reference images do not need their encodings shifted. In DreamOmni2, if image 1 was the source, image 2 was a reference, and image 3 was another reference, the indices were 1, 2, 3. In DreamOmni3 with joint input, if the source (index 1) and scribbled source (also index 1) both use index 1, the references still start at index 2 — maintaining "consistency with the original training format" (Section 3.2). This is important because the LoRA training builds on the pre-trained Kontext model, which expects a specific indexing convention. Changing the indexing scheme for reference images would create a distribution shift that could degrade the model's existing multi-reference editing capabilities.

For reference images that contain scribbles (S-Reference Images), the paper does not use joint input, so each S-Reference Image receives its own distinct index and position encoding, exactly as a regular reference image would in DreamOmni2.

Ablation evidence (Table 4). The paper ablates the encoding choices:

Configuration	Same Index?	Same Position?	Editing	Generation
Scheme 1	✗	✗	0.3750	0.2791
Scheme 2	✗	✓	0.4000	0.4186
Scheme 3	✓	✗	0.4250	0.3488
Scheme 4 (Ours)	✓	✓	0.4500	0.4651

For editing, the progression from no shared encodings (37.50%) to fully shared (45.00%) represents a +7.50 percentage point improvement, with each encoding contributing roughly equally (same position alone: +2.50; same index alone: +5.00). For generation, same position encoding is more important (+13.95 over no sharing) than same index encoding (+6.97), and the combination yields the best result (+18.60 over no sharing). The paper attributes this to two mechanisms: "(1) Matching the position and index encodings allows for better pixel-level alignment between the two images, enabling the model to more accurately locate the scribbles and preserve pixel-level information, leading to more consistent and precise editing. (2) Using the same encodings means that subsequent reference images do not need to modify their encodings, maintaining consistency with the original training format and allowing the model to better leverage its pre-trained editing capabilities for more accurate results" (Section 4).

Training Configuration and Procedure

DreamOmni3 is not trained from scratch. It is built by fine-tuning an existing unified editing model (DreamOmni2, which itself builds on FLUX Kontext) using parameter-efficient adaptation. This section details the training setup (Section 3.2).

Base models.

VLM: Qwen2.5-VL 7B, a 7-billion-parameter vision-language model that encodes text instructions into token embeddings. The VLM provides multimodal understanding — it can interpret instructions that reference visual concepts (colors, shapes, spatial relations) in the text.
MM-DiT (diffusion backbone): FLUX Kontext, a flow-matching-based diffusion transformer that operates in latent space. FLUX Kontext is the image generation and editing engine — it takes noisy latent tokens, text tokens, and image tokens as conditioning, and iteratively denoises to produce the output image.
VAE: The standard FLUX variational autoencoder that compresses images into latent tokens (for input) and decodes latent tokens back into images (for output). The VAE is frozen during training.

The paper states it uses "DreamOmni2's VLM (Qwen2.5-VL 7B) and FLUX Kontext's joint training scheme" (Section 3.2), indicating that the training infrastructure and multi-image conditioning mechanisms from DreamOmni2 are reused.

Parameter-efficient fine-tuning with LoRA. Instead of full fine-tuning (which would be expensive and risk catastrophic forgetting of original capabilities), DreamOmni3 uses LoRA (Low-Rank Adaptation) with rank 256. LoRA inserts trainable low-rank matrices into the attention layers of the MM-DiT, while keeping all original weights frozen. The rank of 256 controls the capacity of the adaptation — higher rank means more learnable parameters and greater ability to acquire new capabilities, at the cost of more memory and potential overfitting.

Separate LoRAs for editing and generation. The paper trains two independent sets of LoRA weights: one for all scribble-based editing tasks, and one for all scribble-based generation tasks. This separation is motivated by compatibility with DreamOmni2, which also used separate LoRAs for its multi-reference generation and editing capabilities. At inference time, the appropriate LoRA is loaded based on whether the user wants to edit an existing image or generate from scratch.

Training data mixture. The model is trained on the combined ~117K samples from all sub-tasks described in Section 3.4 (Data Synthesis). The paper does not specify whether different tasks are weighted differently in the training objective or whether they are simply shuffled together with equal probability.

Preserving original capabilities. An important design choice: by using LoRA rather than full fine-tuning, the original Kontext instruction-editing weights are preserved intact. When LoRA adapters are not activated (i.e., when the user provides text-only instructions without scribbles), the model behaves exactly as the original Kontext model. When scribble inputs are detected, the LoRA adapters are activated, adding scribble understanding on top of the base capabilities. The paper frames this as seamless integration: "When a user inputs an image with scribbles into the model, our LoRA is activated, seamlessly integrating multimodal instruction-based editing and generation into the unified model" (Section 3.2).

Training hyperparameters. The paper states that training "took approximately 400 A100 hours" (Section 3.2) but does not provide learning rate, batch size, optimizer, number of training steps, or other standard hyperparameters. This is a notable omission — we cannot reproduce the training without additional details or the released code.

Inference: How the Model Handles User Inputs at Test Time

While the paper focuses on training, the inference procedure is implicit in the architectural description. When a user provides a scribble-based editing request:

Input preparation. The user draws scribbles (colored circles, boxes, or freehand doodles) on the source image using a GUI brush tool. For editing tasks, this creates the scribbled source image. If reference images are provided, the user may optionally annotate them with scribbles. A text instruction is written that references the scribble colors (e.g., "make the object in the red circle match the object in the blue circle").
Joint input assembly. For editing tasks where the source image has scribbles: the original (clean) source image and the scribbled source image are both fed into the VAE, producing two sets of latent tokens. For reference images (even if they contain scribbles): only the scribbled reference image is tokenized — joint input is not used. For generation tasks: only the scribbled canvas is tokenized.
Encoding assignment. The original source and scribbled source receive the same index encoding and position encoding. Reference images receive incremented index encodings with standard position encodings (shifted by the width of previous images).
Conditioning and denoising. The text instruction is encoded by Qwen2.5-VL 7B. All tokens (text tokens + image latent tokens) are concatenated and fed into the MM-DiT, which denoises from random Gaussian noise over multiple timesteps, conditioned on the multimodal context. The appropriate LoRA (editing or generation) is activated.
Decoding. The final denoised latent is decoded by the VAE decoder into the output image.

Handling multiple scribbles. Because scribbles are distinguished by color (red circle, blue box, green doodle), the model can handle an arbitrary number of annotations in a single image. The instruction uses color names to establish correspondence: "change the color of the bag in the red circle to purple, and delete the person inside the green circle." This color-based linking avoids the combinatorial complexity of mask-index-based approaches. The joint input scheme naturally supports this because all scribbles are painted into a single RGB image — no additional channels or separate masks are needed regardless of how many scribbles there are.

Handling imperfect scribbles. The model is trained on "30 different squares and circles" that are manually created to be imperfect (Section 3.1). This means at inference time, users do not need to draw precise geometric shapes — rough circles and boxes are sufficient. The model has learned to interpret the intent behind the scribble (the region it approximately encloses) rather than requiring exact boundary alignment. For doodle editing and generation, the use of GPT-Image-1-generated sketches during training (rather than edge-detected precise contours) further reinforces robustness to imprecise user drawings.

Benchmark and Evaluation Framework

The paper constructs the DreamOmni3 benchmark specifically because no existing benchmark evaluates scribble-based editing and generation tasks (Section 3.3). The benchmark design reflects three key principles.

Real-world image data. Unlike the training data (which is synthetically constructed from DreamOmni2 pairs), the benchmark uses real images to assess generalization. The paper states the benchmark consists of "real images to accurately assess model performance in real-world scenarios" (Section 3.3). The test cases cover all seven sub-tasks (four editing, three generation) with diverse editing categories including both abstract property edits and concrete object edits. The supplementary materials contain details on the exact number of test cases and their composition.

VLM-based automated evaluation. Traditional image editing metrics like DINO and CLIP similarity are rejected as "not sufficient to accurately evaluate the complex and diverse instruction-based editing and generation tasks" (Section 3.3). The paper cites prior work (Step1x-Edit, Kontext) that also found VLMs to be more reliable for instruction-following evaluation. The evaluation uses two VLMs: Gemini 2.5 Pro and Doubao 1.6, both prompted with a standardized system prompt (provided in the supplementary materials) that evaluates each output on four criteria:

Instruction-following accuracy: Does the generated/edit match what the instruction requested?
Appearance/object/attribute consistency: Do humans, objects, and abstract properties (colors, styles) remain consistent with the input where they should?
Absence of severe visual artifacts: Are there obvious glitches, distortions, or unnatural elements?
Scribble region alignment: Is the generated/edited content correctly placed within the specified scribble regions?

Only when all four criteria are met is the edit or generation considered successful. The output is a binary pass/fail per test case, and the reported metric is the success rate (fraction of test cases that pass).

Human evaluation for validation. To validate the VLM-based evaluation, the paper also conducts human evaluation with multiple professional engineers. For each test case, 5 reviewers independently assess the output against the same criteria used by the VLMs. A case is deemed successful if it receives approval from more than 3 evaluators (i.e., at least 4 out of 5 agree it passes). The paper reports that "VLM-based assessments align closely with human evaluations" (Section 3.3), though the specific correlation coefficient or agreement rate is not provided.

The human evaluation results serve as the primary ground truth, while the Gemini and Doubao evaluations provide automated, reproducible assessments. Tables 1 and 2 report all three metrics (Gemini, Doubao, Human) for each compared method, with Human results given particular weight in the paper's claims (e.g., "DreamOmni3 achieved the best results in human evaluations" in Section 4).

Why this evaluation design matters. Scribble-based editing is inherently a multi-criteria task: a model might correctly insert an object at the right location but change the background colors (failing consistency), or preserve the background perfectly but misplace the edit (failing spatial alignment). Single-metric evaluations (like CLIP score) collapse these dimensions into one number, making it impossible to diagnose failure modes. The four-criteria conjunctive evaluation (all must pass) sets a high bar that matches real user expectations: an edit with a glaring artifact is a failure regardless of how well it follows the instruction. This aligns the benchmark with practical usability.

4. Key Insights and Innovations

Innovation 1: Scribble-Based Interaction as a First-Class Modality in Unified Editing Models

The paper's most fundamental intellectual move is not architectural but taxonomic: it defines scribble-based editing and generation as distinct task categories with their own sub-task decompositions, data requirements, and evaluation protocols, rather than treating scribbles as a minor input variant of existing mask-based or text-based methods. This framing transforms scribbles from an implementation detail (how to specify a mask) into a modality on equal footing with text and reference images.

What the field did before. Prior work fell into two camps: mask-based methods that used precise binary masks as the spatial specification mechanism (inpainting, MagicQuill, object insertion via compressed IDs), and text-only unified models (DreamOmni2, Omnigen2, GPT-4o) that had no spatial input channel at all. In the mask-based paradigm, scribbles are just a way to generate a mask — the mask is the primary representation, and scribbles are an intermediate. In the text-only paradigm, spatial specification is the user's problem to solve through verbal description. Neither camp treated the scribble itself as the enduring input representation that the model reasons about directly.

The conceptual shift. DreamOmni3 elevates scribbles to a modality by keeping them in RGB space — literally painting colored annotations onto the image — and feeding them directly to the model as pixels, not as extracted masks. This means the model sees what the user sees: a red circle drawn around a bag, a blue box around a window, a green doodle sketch of a dragon. The model must learn to interpret these visual annotations in context, not simply use them as a mask-generation step. This is akin to the difference between speech recognition (where audio is an intermediate signal en route to text) and multimodal models that process audio directly as a modality — the intermediate extraction step is eliminated, and the model learns end-to-end from the raw signal.

The significance is that scribble interpretation becomes a visual understanding task rather than a preprocessing step. The model must handle: imperfect shapes (rough circles, not geometric ellipses); color-based disambiguation (red circle means this, blue box means that); semantic scope inference (does the red circle around a person mean "edit the person" or "edit the person's shirt"?); and contextual integration (a doodle of a dragon pasted onto a photo should generate a photorealistic dragon, not a sketch). By training end-to-end on the RGB scribbled image, the model learns to make these interpretive judgments implicitly, acquiring a form of visual common sense that mask-based preprocessing discards.

Evidence. The baseline results in Tables 1 and 2 demonstrate that scribble understanding is not an emergent capability of text+image models. DreamOmni2 achieves only 17.50% human-evaluated editing success on scribble tasks despite strong text-based editing performance, and 4.65% on generation. Even GPT-4o (58.75% editing) and Nano Banana (41.25%) show specific failure modes — GPT-4o with "yellowing" and inconsistent background pixels, Nano Banana with "copy-and-paste effects" — that suggest they are not natively processing scribbles as a modality but rather using general reasoning to approximate spatially-aware editing. The paper's 57.50% human-evaluated editing success (matching GPT-4o without the artifacts) and 53.49% generation success (substantially exceeding GPT-4o's 39.53%) demonstrates that treating scribbles as a first-class training modality produces qualitatively different — and more robust — behavior than relying on general-purpose reasoning.

This is a fundamental reframing, not an incremental improvement. It changes what it means for a model to "understand" spatial instructions: from "the user provides coordinates which we convert to a mask" to "the user draws what they mean, and the model interprets the drawing in visual context." The downstream implication is that future work on spatial interaction should focus on richer scribble vocabularies (doodles with varying opacity, pressure, style) rather than more precise mask extraction, because the scribble itself is the expressive medium.

Innovation 2: Joint Input with Shared Encodings as an Alternative to the Mask-Based Paradigm

The paper introduces a specific architectural mechanism — feeding both the original and scribbled source image into the model with identical index and position encodings — and demonstrates through ablation that this mechanism is the minimal sufficient design for scribble-based editing, avoiding the complexity of binary mask approaches while achieving editing consistency. The innovation is not the idea of multi-image input (DreamOmni2 already handled multiple reference images), but the recognition that spatially-aligned dual input is the correct abstraction for annotation-based editing, and that shared encodings are the specific trick that makes this abstraction work within existing transformer architectures.

What the field did before. The dominant paradigm for spatially-localized editing was binary masks: provide a separate mask channel (or channels) indicating which pixels to edit, and design the model to attend to these masks during generation. This approach has deep roots in inpainting literature and was extended in works like SmartBrush (mask + shape guidance) and Ace++ (mask + reference encoding). The assumption was that spatial specification requires a separate representation from the image itself — a mask that cleanly separates "edit region" from "preserve region," with the model learning to condition on this separation.

The conceptual diagnosis. DreamOmni3 identifies why this paradigm breaks down for multi-region, multi-modal editing: masks don't scale compositionally. Each new region requires a new mask channel; each mask-to-instruction correspondence requires linguistic linking; and the whole system becomes a tangled web of indices and mappings that grows in complexity with the number of edits. The deeper problem is that masks treat spatial specification as additive (you add mask channels to the input), when it should be integrative (the spatial information is embedded in the same representational space as the visual content it refers to).

The solution and why it's non-obvious. Joint input with shared encodings solves this by keeping spatial information in RGB space, where color naturally distinguishes multiple regions without additional channels. The model sees the original image and its annotated version as two views of the same scene — one with scribbles, one without — that share the same spatial coordinate system. The shared index and position encodings are the crucial design choice: they tell the transformer's attention mechanism that token (37, 82, 1) in the scribbled stream and token (37, 82, 1) in the original stream represent the same spatial location, making it straightforward for self-attention to compare "what's here in the annotated version" versus "what's here in the clean version" and decide whether an edit is needed.

This is non-obvious because the natural instinct when feeding two images to a transformer is to give them distinct index encodings (so the model can distinguish which image a token came from). The paper's counterintuitive move is to give them the same encoding, deliberately blurring the distinction between the two streams. The ablation in Table 4 shows this matters: same index encoding alone improves editing from 37.50% to 42.50% (+5.00 points), and adding same position encoding pushes it to 45.00% (+2.50 more). The model actively benefits from being told "these two images are the same image" rather than "these are two different images that happen to have similar content."

Distinguishing from prior multi-image conditioning. Previous multi-image models (DreamOmni2, Kontext) used distinct index encodings because each image was semantically different — a source, a reference, another reference. The insight that some pairs of input images should share encodings because they share spatial identity is the conceptual advance. It generalizes beyond scribbles: any annotation that overlays an image (edge maps, depth maps, segmentation overlays) could benefit from dual-input with shared encodings rather than separate mask channels.

Evidence beyond the ablation. The qualitative results (Figure 3) show that DreamOmni3 preserves background pixels faithfully while editing only the scribbled regions — the holy grail of localized editing that mask-based methods often fail at due to mask imprecision. The paper attributes this to the joint input ensuring the model "can see the original image information" (Section 3.2) at every denoising step. Without it (Scheme 1 in Table 3, editing drops from 45.00% to 11.25%), the model must infer original pixels from context around the scribbles, leading to inconsistent backgrounds.

This is an incremental architectural innovation — it refines how spatial information is fed to existing diffusion transformer architectures — but with conceptual significance because it establishes a design principle (shared encodings for spatially-aligned dual inputs) that generalizes beyond this specific task. It answers the question: "given that we have a transformer that processes multiple images, how should we represent a source image and its spatial annotations to maximize the model's ability to localize edits while preserving non-edited regions?"

Innovation 3: Difficulty-Agnostic Data Synthesis Through Template Libraries and Doodle Models

Section 3 of the prior analysis detailed the data synthesis pipeline. What makes this pipeline intellectually distinctive is not the specific steps (which are pragmatic and straightforward) but the underlying design philosophy: train on synthetic scribbles that are deliberately imperfect and style-varied, rather than precise geometric annotations, because the model should learn scribble interpretation, not scribble tracing.

What the field did before. When prior work needed spatial annotations for training, the default approach was to use ground-truth segmentation masks (from datasets like COCO or RefCOCO) or bounding boxes, often rendered as precise geometric shapes. The assumption — rarely stated explicitly — was that training on accurate spatial annotations would produce models that generalize to inaccurate user input through robustness of the learned representation. This is the standard supervised learning assumption: train on clean labels, and the model will be somewhat robust to noisy inputs at test time.

The conceptual diagnosis. DreamOmni3 identifies that this assumption is wrong for scribble-based interaction because scribble interpretation and mask interpretation are different tasks. A model trained on precise segmentation masks learns to attend to exact boundary cues — it expects the annotation to faithfully trace object contours. When a user draws a rough circle that only approximately encloses the target object, the model's mask-trained attention mechanism receives out-of-distribution input and either over-edits (expanding beyond the intended region) or under-edits (restricting to the exact scribble pixels). The paper's insight is that scribble understanding requires the model to infer intent from approximate spatial cues, not to follow precise spatial instructions.

The design response and why it's non-obvious. The paper introduces two mechanisms that operationalize this philosophy:

Template library of 30 imperfect shapes. Instead of generating scribbles programmatically (e.g., drawing ellipses with OpenCV), the authors manually created 30 hand-drawn-style squares and circles. These shapes have the slight irregularities — wobbly lines, asymmetric curves, varying stroke thickness — that characterize human drawing. Each training sample randomly selects from this library, ensuring the model sees (and learns to be robust to) significant variation in scribble appearance. This is non-obvious because the natural engineering approach would be procedural generation with random perturbations, which would be simpler and scale to more variations. The manual approach suggests that capturing the specific character of human drawing (which may have statistical regularities that procedural noise doesn't replicate) matters more than maximizing variation.
GPT-Image-1 for doodle generation instead of Canny edge detection. For doodle editing and generation, the paper explicitly rejects Canny edge detection — the standard method for extracting sketches from images — because "edge maps produce precise, high-frequency line drawings that do not match what humans actually sketch" (Section 3.1, paraphrased). Instead, they use GPT-Image-1 to produce "doodles with non-strict pixel consistency" — abstract, simplified sketches that capture semantic essence without pixel-perfect contour alignment. This choice means the model is trained on doodles that look like what a human would draw (a simplified, stylized representation of an object), not on precise edge tracings that no human would produce.

The combination of these two mechanisms means the model learns scribble interpretation as a semantic task (what object or region does this approximate annotation refer to?) rather than a geometric task (what exact pixels does this boundary enclose?). This is a crucial distinction: semantic scribble interpretation handles rough circles around objects, doodles that capture shape essence without detail, and scribbles that overlap multiple objects (with the instruction disambiguating). Geometric mask interpretation would fail on all of these.

Evidence. The paper does not ablate the template library size or the choice of GPT-Image-1 vs. Canny, so the evidence is indirect: the model's strong performance on human evaluation (57.50% editing, 53.49% generation) with real user-drawn scribbles (which are imperfect by nature) suggests the training strategy successfully generalizes. More tellingly, the baseline models (DreamOmni2, Omnigen2) that were not trained on imperfect scribbles show dramatically worse performance (17.50%, 5.00%), and even commercial models exhibit specific failures (GPT-4o's yellowing, Nano Banana's copy-paste artifacts) that are consistent with mask-trained or non-scribble-optimized behavior. The paper's qualitative results (Figures 3 and 4) show DreamOmni3 producing clean edits that respect scribble intent (the approximate region the user meant) rather than scribble precision (the exact pixels the scribble covers).

Significance and limitations. This innovation is conceptual but empirically under-validated — the paper makes a compelling argument for training on imperfect annotations, and the results are consistent with that argument, but there is no direct ablation comparing imperfect vs. perfect scribble training. A study that trains the same model on (a) precise geometric circles/boxes vs. (b) the hand-drawn template library vs. (c) real human-drawn scribbles would strengthen the claim considerably. The paper's contribution here is best understood as a design principle with initial supporting evidence rather than a proven theorem: when training models to interpret user-drawn visual annotations, the training annotations should match the distribution of user drawings in their imperfection characteristics, not just their spatial content.

The broader implication is that the field's default assumption — "train on clean ground truth, test on noisy reality" — may be actively harmful for interaction modalities where the noise carries semantic information (the user's drawing style, the looseness of the annotation, the abstraction level of the sketch). Future work on sketch-based interfaces, gesture recognition, and other human-originated visual inputs should consider training data that captures the human quality of the input, not just its information content.

Innovation 4: LoRA-Based Capability Addition as a Compositional Strategy for Unified Models

The choice to train scribble understanding via separate LoRA adapters (one for editing, one for generation) on a frozen base model is more than an engineering convenience — it represents a compositional strategy for extending unified models that preserves existing capabilities while adding new ones, avoiding the catastrophic forgetting and capability interference that plague full fine-tuning approaches in multi-task settings.

What the field did before. The dominant approach to building unified editing models has been joint training: mix data from all target tasks (editing, generation, subject-driven generation, style transfer, etc.) and train or fine-tune the full model on this mixture. DreamOmni2 used separate LoRAs for its multi-reference generation and editing capabilities but did not frame this as a general principle. Most prior work (Omnigen, Qwen-image-Edit) used full-parameter training or fine-tuning, where adding a new capability necessarily modifies all model weights and risks degrading existing capabilities.

The conceptual advance. DreamOmni3 frames LoRA-based capability addition as a modular extension mechanism: "by leveraging LoRA for training, we retain the original instruction-editing capabilities of Kontext. When a user inputs an image with scribbles into the model, our LoRA is activated, seamlessly integrating multimodal instruction-based editing and generation into the unified model" (Section 3.2). This framing treats the base model (FLUX Kontext) as a capability platform and LoRA adapters as capability modules that can be independently developed, trained, and activated without mutual interference.

The significance extends beyond this paper. If the vision of unified models is to support an ever-expanding set of input modalities (text, images, scribbles, audio, video, 3D), then full joint training becomes combinatorially infeasible — each new modality requires retraining on all previous modalities to prevent forgetting. The LoRA-based compositional approach suggests an alternative: train each new modality as a separate adapter, and compose them at inference time based on what input modalities are present. A user providing text-only instructions gets the base model; text + scribbles activates the scribble LoRA; text + scribbles + audio could activate both scribble and audio LoRAs (assuming they are designed to compose without conflict).

Evidence. The paper demonstrates this compositional property implicitly: the base Kontext model's original capabilities are preserved (the LoRA weights are additive on top of frozen base weights), and scribble understanding is added without degrading text-only editing. The ablation in Table 3 shows that training on the scribble dataset without joint input (Scheme 3) still improves editing from 11.25% (baseline, Scheme 1) to 35.00%, while joint input with training (Scheme 4) pushes to 45.00%. The baseline (no training, no joint input) represents the base model's original capability — the fact that training adds scribble understanding without apparently harming text-only editing (though this is not explicitly evaluated) supports the modular claim.

The separate LoRAs for editing and generation further demonstrate compositional thinking: these are two different "modes" of the same model, activated based on task, rather than a single unified adapter that must learn both. This separation may explain why the model performs well on both tasks despite their different requirements (editing requires pixel-level preservation; generation requires full-scene synthesis) — the adapters can specialize without interference.

This is an architectural innovation with theoretical implications. It suggests that the path to general-purpose multimodal creation tools may lie not in ever-larger jointly-trained models, but in modular architectures where base capabilities are frozen and new modalities are added as composable extensions. The LoRA mechanism is well-established (it was not invented by this paper), but the paper's application of it as a capability-composition framework for unified editing models is a conceptual contribution that points toward a scalable strategy for model evolution.

A limitation the paper does not discuss: the compositional approach assumes that different modality adapters are independent — that scribble understanding and text understanding don't need to deeply interact. For tasks where modalities must be tightly integrated (e.g., "draw a scribble that represents the emotion described in the text"), separate adapters may be insufficient. The paper's tasks keep modalities relatively independent (scribbles specify location; text specifies action), which may be why separate LoRAs work well. Whether this approach generalizes to more entangled multimodal reasoning remains an open question.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper constructs the DreamOmni3 benchmark specifically for this work, consisting of real-world images (as opposed to the synthetically-generated training data) to assess generalization performance. The benchmark covers all seven proposed sub-tasks: four scribble-based editing tasks (scribble + multimodal instruction editing, scribble + instruction editing, image fusion, doodle editing) and three scribble-based generation tasks (scribble + multimodal instruction generation, scribble + instruction generation, doodle generation). The test cases encompass diverse editing categories including both abstract property edits (design style, color scheme, hairstyle) and concrete object edits (adding, removing, or modifying objects, people, animals). The paper does not specify the exact number of test cases in the main text, deferring details to the supplementary materials. All images are real photographs to "enable a more accurate evaluation of the model's generalization and real-world performance" (Section 1).
Base model(s). DreamOmni3 is built on DreamOmni2, which itself is built on FLUX Kontext (a flow-matching-based diffusion transformer) with Qwen2.5-VL 7B as the VLM text encoder. The paper uses DreamOmni2 as the closest architectural predecessor — it already supports multi-reference image generation and editing with language and image instructions, making it the natural platform for adding scribble understanding. For the ablation studies on joint input and encoding schemes, the base model is Kontext (FLUX.1 Kontext), representing the pre-trained checkpoint before any DreamOmni2-specific fine-tuning. The FLOPs-matched comparison uses PaLM 2-S* models at different scales, but this section focuses on the scribble-specific experiments where the base model family is the FLUX diffusion backbone paired with Qwen2.5-VL.
Metrics. The paper evaluates all tasks using a binary success rate (pass/fail per test case, reported as a fraction or percentage). A test case is considered successful only when it meets all four of the following criteria simultaneously: (1) accuracy in following the instruction in the generated/edited output, (2) consistency in human appearance, objects, and abstract attributes with the input image (where preservation is expected), (3) absence of severe visual artifacts, and (4) alignment of the generated or edited content with the specified scribble regions. This conjunctive evaluation sets a deliberately high bar — a single failure on any dimension renders the entire case unsuccessful. The success rate is measured using three evaluators: Gemini 2.5 Pro (automated VLM evaluation), Doubao 1.6 (automated VLM evaluation), and human evaluation by professional engineers (5 reviewers per case, with a case deemed successful if approved by more than 3 evaluators, i.e., at least 4 out of 5). The paper reports all three metrics in Tables 1 and 2, with the human evaluation treated as the ground truth and the VLM evaluations serving as reproducible automated proxies. The paper notes that "VLM-based assessments align closely with human evaluations" (Section 3.3), though no correlation coefficient or agreement rate is provided.
Baselines. The paper compares against six methods, spanning open-source unified models, a base diffusion model adapted for multi-image input, and closed-source commercial systems:
- DreamOmni2 [24]: The direct predecessor, supporting multi-reference image generation and editing with language and image instructions but no scribble understanding. This is the most relevant baseline as DreamOmni3 is built directly on its architecture.
- Omnigen2 [21]: A unified multimodal generation model that handles instruction-based editing and generation.
- Qwen-image-Edit-2509 [20]: A competitive instruction-based image editing model.
- Kontext [1] (FLUX.1 Kontext): The base diffusion model. Since Kontext does not natively support multi-image input, the paper applies a method from Diffusers [15] that combines multiple images into a single composite image for input. This tests whether the pre-trained base model has any emergent scribble understanding capability.
- GPT-4o [10]: OpenAI's closed-source multimodal model with image generation capabilities.
- Nano Banana [5]: Google's closed-source commercial image generation and editing model (Gemini 2.5 Flash Image).
All baselines are evaluated on the same DreamOmni3 benchmark with the same VLM and human evaluation protocols. Critically, the open-source baselines (DreamOmni2, Omnigen2, Qwen-image-Edit, Kontext) were not trained or optimized for scribble-based tasks, so their performance represents a "zero-shot" test of whether scribble understanding emerges from general visual reasoning. GPT-4o and Nano Banana are proprietary systems whose training data and optimization for scribble tasks are unknown.
Generation budget / compute accounting. The paper does not measure or control for inference-time compute budget (number of denoising steps, number of generated samples, or FLOPs) in its main experiments. All models generate a single output per test case under their default inference settings. This is a notable difference from the FLOPs-matched comparison framework used in the DreamOmni2 paper's analysis of test-time compute scaling — the scribble-based editing and generation experiments focus purely on output quality at whatever compute cost each model's default inference requires. The training compute is reported as approximately 400 A100 hours for DreamOmni3's LoRA fine-tuning, but no comparative training compute analysis is provided for baselines (which have unknown or undisclosed training costs, particularly the commercial models).
Cross-validation / statistical protocol. The paper does not describe any cross-validation, statistical significance testing, or confidence intervals for the benchmark results. The evaluation protocol is: each test case is assessed by two VLMs (independently) and by 5 human reviewers. The human evaluation uses a majority-vote threshold (more than 3 out of 5 reviewers must approve for a case to be considered successful). The paper reports aggregate success rates (fraction of test cases that pass) without error bars, standard deviations, or inter-rater reliability metrics (e.g., Fleiss' kappa for the 5 human reviewers). This means we cannot assess whether the reported performance differences (e.g., DreamOmni3 at 57.50% vs. GPT-4o at 58.75% in human evaluation for editing) are statistically significant or within the noise of annotator disagreement. For the ablation studies (Tables 3 and 4), the paper reports raw success rates without specifying the number of test cases used for these ablations or whether they were run on the full benchmark or a subset.

Main Quantitative Results

Scribble-Based Editing Results

The headline result for scribble-based editing (Table 1) is that DreamOmni3 achieves a 57.50% human-evaluated success rate, which substantially outperforms all open-source baselines and is statistically indistinguishable from the best commercial model (GPT-4o at 58.75%). The VLM-based evaluations show a similar pattern, with DreamOmni3 at 52.50% (Gemini) and 45.00% (Doubao), compared to GPT-4o at 61.25% (Gemini) and 53.75% (Doubao).

Breaking down the comparison:

Against open-source unified models. DreamOmni3 demonstrates a dramatic improvement over the next-best open-source model (DreamOmni2):

DreamOmni3: 57.50% human, 52.50% Gemini, 45.00% Doubao
DreamOmni2: 17.50% human, 20.00% Gemini, 23.75% Doubao
Qwen-image-Edit-2509: 16.25% human, 22.50% Gemini, 20.00% Doubao
Omnigen2: 5.00% human, 10.00% Gemini, 8.75% Doubao
Kontext (multi-image via Diffusers): 2.50% human, 8.75% Gemini, 11.25% Doubao

The 40-percentage-point gap between DreamOmni3 and DreamOmni2 (57.50% vs. 17.50% human) represents a roughly 3.3× improvement that can only be attributed to the scribble-specific training data and architectural modifications, since both models share the same base architecture (FLUX Kontext + Qwen2.5-VL 7B). The extremely low performance of Omnigen2 (5.00%) and Kontext (2.50%) confirms that scribble understanding is not an emergent capability of general-purpose vision-language or diffusion models — it requires dedicated training data and optimization.

Against commercial models. The comparison with proprietary systems reveals a more nuanced picture:

GPT-4o: 58.75% human, 61.25% Gemini, 53.75% Doubao
Nano Banana: 41.25% human, 51.25% Gemini, 42.50% Doubao

DreamOmni3 matches GPT-4o in human evaluation (57.50% vs. 58.75%) and falls behind in VLM-based evaluations (52.50% vs. 61.25% on Gemini, 45.00% vs. 53.75% on Doubao). The paper notes specific qualitative failure modes for the commercial models that VLM evaluations may miss: GPT-4o "frequently exhibits a yellowing issue in the images, and the pixels in the non-edited areas of the output often do not match those in the input image," while Nano Banana shows "copy-and-paste effects and incorrect object proportions" (Section 4). These artifact-based failures are visible to human evaluators but may be partially missed by VLM evaluators trained primarily on semantic content rather than pixel-level fidelity. This explains why GPT-4o's Gemini score (61.25%) is higher than its human score (58.75%) — the VLM may be more lenient on color cast and pixel consistency issues that humans penalize.

The fact that DreamOmni3 achieves comparable human-evaluated performance to GPT-4o while avoiding the specific artifact failures (yellowing, background pixel inconsistency) suggests that training explicitly for scribble-based editing produces more robust behavior than relying on general-purpose reasoning, even when the general-purpose system is a much larger model with broader capabilities.

Qualitative evidence (Figure 3). The paper presents six rows of visual comparisons covering scribble-based multimodal instruction editing (rows 1-2), scribble-based instruction editing (rows 3-4), image fusion (row 5), and doodle editing (row 6). The qualitative patterns align with the quantitative results: DreamOmni3 produces edits that are spatially accurate (the edit occurs within the scribbled region), semantically correct (the edit matches the instruction), and consistent (non-edited regions remain pixel-identical). In contrast, open-source baselines often fail to interpret the scribble at all (producing unchanged output or completely misaligned edits), while GPT-4o and Nano Banana sometimes produce the right semantic edit but with background inconsistencies or artifacts.

Scribble-Based Generation Results

The headline result for scribble-based generation (Table 2) is that DreamOmni3 achieves a 53.49% human-evaluated success rate, substantially outperforming both commercial models (GPT-4o at 39.53%, Nano Banana at 23.26%) and all open-source baselines (best is Qwen-image-Edit at 11.63%).

Breaking down the comparison:

Against commercial models. This is where DreamOmni3 shows its strongest relative advantage:

DreamOmni3: 53.49% human, 51.16% Gemini, 46.51% Doubao
GPT-4o: 39.53% human, 58.14% Gemini, 48.84% Doubao
Nano Banana: 23.26% human, 46.51% Gemini, 41.86% Doubao

The +13.96 percentage point human-evaluated advantage over GPT-4o (53.49% vs. 39.53%) is the most significant result in the generation experiments. The paper provides a specific explanation for the commercial models' underperformance: "GPT-4o and Nano Banana are not specifically optimized for scribble scenarios, and even when explicitly instructed not to generate scribble marks, these models still often output scribbles along with the generated results" (Section 4). This is a revealing failure mode: the commercial models treat the scribble as visual content to be reproduced or incorporated, rather than as a spatial instruction to be interpreted and then discarded. DreamOmni3's training data teaches it that scribbles are instructions, not content — the output image should not contain the scribble annotations.

The VLM evaluation paints a slightly different picture: GPT-4o outscores DreamOmni3 on Gemini (58.14% vs. 51.16%) and edges ahead on Doubao (48.84% vs. 46.51%). This discrepancy between VLM and human evaluation — where VLMs rate GPT-4o higher but humans prefer DreamOmni3 — suggests that the VLM evaluators may be less sensitive to the "scribble marks still visible in output" failure mode, which is visually obvious to humans but may be interpreted by VLMs as a stylistic choice rather than an error. The paper does not discuss this discrepancy in detail.

Against open-source models. The gap is even larger for generation than for editing:

DreamOmni3: 53.49% human, 51.16% Gemini, 46.51% Doubao
DreamOmni2: 4.65% human, 16.28% Gemini, 11.63% Doubao
Qwen-image-Edit-2509: 11.63% human, 16.28% Gemini, 6.98% Doubao
Omnigen2: 4.65% human, 11.63% Gemini, 9.30% Doubao
Kontext: 13.95% human, 9.30% Gemini, 4.65% Doubao

The near-zero performance of DreamOmni2 (4.65% human) on generation is particularly striking given that it shares the same base architecture as DreamOmni3. This demonstrates that scribble-based generation requires capabilities — interpreting spatial annotations on a blank canvas and generating a coherent scene that respects those annotations — that are entirely absent from instruction-based generation models, and cannot be recovered through zero-shot generalization.

Qualitative evidence (Figure 4). The paper presents five rows of visual comparisons: scribble-based multimodal instruction generation (rows 1 and 5), scribble-based instruction generation (rows 2 and 3), and scribble-based multimodal instruction generation with multiple reference images (row 4). The qualitative results show that open-source baselines frequently fail to generate the specified objects at all, produce objects in incorrect locations, or most commonly, retain the scribble annotations in the output image — the colored circles remain visible as artifacts in the generated scene. DreamOmni3, in contrast, generates clean images where the scribbled regions contain the specified content and the scribbles themselves are absent from the output.

Joint Input Scheme Ablation

The paper ablates the joint input scheme in Table 3, comparing four configurations:

Scheme 1 (baseline): No training on scribble data, no joint input. The model receives only the scribbled source image (the original pixels under the scribbles are occluded). Editing: 11.25%, Generation: 4.65% (Doubao evaluation).
Scheme 2: No training, but joint input is used (both original and scribbled source images are fed to the model). Editing: 13.75%, Generation: 4.65%.
Scheme 3: Trained on scribble dataset, but only scribbled source image is input (no joint input). Editing: 35.00%, Generation: 44.19%.
Scheme 4 (DreamOmni3): Trained on scribble dataset with joint input. Editing: 45.00%, Generation: 46.51%.

The key findings:

Training on scribble data is the dominant factor. Comparing Scheme 3 (trained, no joint input) with Scheme 1 (no training, no joint input), editing success jumps from 11.25% to 35.00% (+23.75 points) and generation from 4.65% to 44.19% (+39.54 points). This demonstrates that the scribble training data — even without joint input — teaches the model the fundamental skill of interpreting scribble annotations.

Joint input matters for editing but not for generation. Comparing Scheme 4 with Scheme 3:

Editing: +10.00 percentage points improvement (35.00% → 45.00%)
Generation: +2.32 percentage points improvement (44.19% → 46.51%)

This asymmetric effect is consistent with the paper's stated rationale: "scribbles can obscure edited regions of the source image, so joint input ensures the model can see the original image information, improving editing consistency. In contrast, generation tasks don't require high pixel-level consistency, making the improvement less significant" (Section 4). For editing, the joint input provides the clean source pixels that the scribbles would otherwise occlude, enabling the model to preserve non-edited regions faithfully. For generation, the scribbled canvas (white background with scribbles drawn on it) contains all the information the model needs — there are no hidden pixels to recover — so the additional clean canvas provides minimal benefit.

Joint input alone (without training) provides negligible benefit. Scheme 2 (joint input, no training) barely improves over Scheme 1 (editing: 11.25% → 13.75%, generation: unchanged at 4.65%). This confirms that joint input is not a standalone solution — the model must be trained to use the dual-input signal. Without training, the model doesn't understand that the two images represent the same scene with annotation overlay; it likely treats them as two independent images and fails to exploit the spatial correspondence.

The interaction effect. The gap between Scheme 4 and Scheme 1 on editing (11.25% → 45.00%, a 4× improvement) represents the combined effect of scribble-specific training data (+23.75) and joint input (+10.00). The fact that these improvements are roughly additive suggests they address complementary aspects of the task: training data teaches scribble interpretation, joint input teaches consistency preservation.

Index and Position Encoding Ablation

The paper ablates the encoding scheme for joint input in Table 4, comparing four configurations:

Scheme 1: Different index encoding, different position encoding for the two source images. Editing: 37.50%, Generation: 27.91%.
Scheme 2: Different index encoding, same position encoding. Editing: 40.00%, Generation: 41.86%.
Scheme 3: Same index encoding, different position encoding. Editing: 42.50%, Generation: 34.88%.
Scheme 4 (DreamOmni3): Same index encoding, same position encoding. Editing: 45.00%, Generation: 46.51%.

The findings reveal an asymmetric sensitivity between editing and generation:

For editing, the progression from no shared encodings (37.50%) to fully shared (45.00%) is a gradual improvement, with same index encoding providing a larger boost (+5.00, from Scheme 2 to Scheme 3) than same position encoding (+2.50, from Scheme 1 to Scheme 2). The final combined scheme achieves +7.50 percentage points over the baseline. This suggests that editing benefits from both spatial alignment (same position encoding helps the model map scribble locations to original image locations) and identity alignment (same index encoding tells the model these are two views of the same image). The spatial alignment is intuitive — the model needs to know which pixels in the scribbled image correspond to which pixels in the original. The identity alignment is subtler but important: by telling the model these images share an identity, the model can leverage its pre-trained multi-image processing capabilities without being confused by what appears to be two separate but highly similar images.

For generation, the pattern is different and more dramatic. Same position encoding alone (Scheme 2, 41.86%) provides a massive +13.95 point improvement over no shared encodings (Scheme 1, 27.91%). However, same index encoding without same position encoding (Scheme 3, 34.88%) provides a smaller +6.97 point improvement and actually underperforms the position-only scheme. The full combination (Scheme 4, 46.51%) yields +18.60 points over the baseline. This suggests that for generation — where the model must generate from a blank canvas with scribbles — spatial alignment is the critical factor. The model needs to know precisely where each scribble token is located relative to the canvas, so it can generate appropriate content at those locations. Identity alignment (same index encoding) is less critical for generation because there is no original image to preserve pixels from — the identity of the two "views" matters less than their spatial correspondence.

The paper's explanation (Section 4) attributes these improvements to two mechanisms:

Matching position and index encodings enables better pixel-level alignment, allowing more accurate scribble localization and pixel-level information preservation.
Same encodings maintain consistency with the original training format (subsequent reference images don't need encoding shifts), enabling the model to "better leverage its pre-trained editing capabilities."

The second mechanism is important but receives limited empirical support — the paper does not ablate whether shifting reference image encodings (rather than keeping them consistent with training format) degrades performance, so this explanation remains partly speculative.

Ablation Studies and Robustness Checks

Joint input necessity for editing vs. generation: The Table 3 ablation (described above) directly tests whether joint input matters differently for editing and generation. The key finding — +10.00 points for editing, +2.32 for generation — supports the paper's claim that joint input is critical for pixel-level consistency preservation in editing but minimally beneficial for generation where no pixels need preserving. This asymmetry is logically consistent with the design rationale and provides strong evidence that joint input's mechanism is indeed about giving the model access to original pixels that scribbles would otherwise occlude.

Encoding scheme components: The Table 4 ablation isolates the contributions of same-index and same-position encodings. The non-trivial finding is that both components matter, but for different reasons and with different relative importance across tasks. For editing, same index encoding (+5.00) matters more than same position encoding (+2.50), suggesting that teaching the model these are two views of the same image is the primary challenge. For generation, same position encoding (+13.95) dramatically outweighs same index encoding (+6.97), suggesting that spatial correspondence is the primary challenge. This task-dependent sensitivity is not obvious a priori and represents a meaningful empirical insight about how diffusion transformers process spatially-annotated multi-image inputs.

Training on scribble data (Table 3, Scheme 1 vs. Scheme 3): While presented as part of the joint input ablation, this comparison also serves as an ablation of the training data itself. The +23.75 point improvement on editing and +39.54 point improvement on generation from adding scribble training data (without joint input) demonstrates that the data synthesis pipeline — not just the architectural innovations — is responsible for the majority of DreamOmni3's capabilities. This reinforces the paper's framing that data scarcity is the primary challenge for scribble-based tasks.

Baseline diversity as a robustness check: The paper tests six diverse baselines spanning different model families, training paradigms (open-source fine-tuned, open-source pre-trained, closed-source commercial), and scales. The consistent pattern — open-source models without scribble training perform extremely poorly (2.50%–17.50% human editing), while the scribble-trained DreamOmni3 performs competitively with the best commercial model — serves as a robustness check that the task requires dedicated training and cannot be solved through scale or general reasoning alone. The fact that GPT-4o (a much larger and more capable general-purpose model) achieves similar editing performance to DreamOmni3 but with different failure modes (artifacts, yellowing) while underperforming on generation (+13.96 point gap) suggests that scribble-specific training produces qualitatively different — and in some respects more robust — behavior than general-purpose multimodal reasoning.

VLM evaluation validation through human alignment: The paper's use of three evaluators (Gemini, Doubao, human) for all main results serves as a robustness check on the evaluation methodology. The overall pattern of results is consistent across evaluators — DreamOmni3 substantially outperforms open-source baselines across all three metrics for both editing and generation. However, the correlation is imperfect: for editing, GPT-4o outscores DreamOmni3 on Gemini (61.25% vs. 52.50%) but they are tied in human evaluation (58.75% vs. 57.50%). For generation, GPT-4o outscores DreamOmni3 on Gemini (58.14% vs. 51.16%) but underperforms on human evaluation (39.53% vs. 53.49%). These discrepancies highlight the difficulty of automated evaluation for complex visual tasks and support the paper's decision to treat human evaluation as the primary metric while using VLM evaluations as supplementary indicators.

Qualitative validation of failure modes: The paper's qualitative analysis (Figures 3 and 4) provides diagnostic evidence for why the baselines fail, beyond just measuring success rates. Open-source models exhibit "scribble retention" (outputting the scribble marks as part of the generated image), mislocalization (editing/generating content in the wrong location), and failure to follow instructions. GPT-4o exhibits color cast issues (yellowing) and background pixel inconsistency. Nano Banana exhibits collage-like artifacts and proportion errors. These qualitative patterns corroborate the quantitative results and provide mechanistic insight into the different failure modes of scribble-untrained vs. generally-capable models.

What is NOT ablated: Several important factors are not tested:

Number of training samples: The paper provides dataset sizes (32K, 14K, 16K, 8K per sub-task) but does not ablate how performance scales with data quantity. We don't know whether the model's performance has saturated at ~117K total samples or would continue improving with more data.
Template library size: The paper uses 30 manually-created scribble symbols. It does not test whether 10 or 100 would produce different results, or whether procedurally-generated scribbles would work as well as hand-drawn ones.
GPT-Image-1 vs. alternatives for doodle generation: The paper argues that Canny edge detection would be inappropriate for training doodle data, but it does not empirically validate this by training a comparison model on Canny-based doodles.
LoRA rank: The LoRA rank of 256 is stated but not ablated. We don't know if rank 128 would achieve similar performance (suggesting capacity is not the bottleneck) or if rank 512 would improve results (suggesting more capacity would help).
Separate vs. joint LoRA for editing and generation: The paper trains separate LoRAs for editing and generation tasks but does not compare against a single unified LoRA trained on all tasks.
VLM evaluation prompt sensitivity: The paper uses a standardized prompt for Gemini and Doubao evaluation (deferred to supplementary materials) but does not test whether results are robust to prompt variations or whether different VLMs produce systematically different evaluations.

Critical Assessment

The experiments in this paper are designed to answer two core questions: (1) Does DreamOmni3's scribble-based editing and generation approach work better than existing models that weren't designed for scribble input? (2) Do the specific architectural innovations — joint input and shared encodings — causally improve performance? The experiments provide convincing evidence for both, but with important scope limitations that constrain what conclusions can be drawn.

On the claim that DreamOmni3 "achieves outstanding performance" on scribble-based editing and generation (Abstract, Section 1).

The evidence supports this claim relative to available baselines but with two important caveats. First, the human evaluation results for editing (57.50% DreamOmni3 vs. 58.75% GPT-4o) are close enough that we cannot determine whether DreamOmni3 is genuinely "better," "comparable," or "slightly worse" than the best commercial system for editing. The paper emphasizes that GPT-4o exhibits qualitative artifacts (yellowing, pixel inconsistency) that the success rate metric may not fully capture, but this argument cuts both ways — if human evaluators are giving GPT-4o similar aggregate scores despite these artifacts, it suggests GPT-4o may be succeeding on other dimensions (perhaps instruction complexity, diversity of edits) where DreamOmni3 is weaker. The benchmark composition matters critically here: if the test cases skew toward edits where DreamOmni3's strengths (pixel consistency, artifact avoidance) are most visible, while GPT-4o's strengths (handling complex language, unusual edit requests) are underrepresented, the comparison would favor DreamOmni3. Without seeing the benchmark distribution (deferred to supplementary materials), we cannot assess this.

Second, "outstanding performance" depends on the absolute success rate. A 57.50% human editing success rate and 53.49% generation success rate — while strong relative to baselines — means the model fails on roughly 42-47% of test cases. For a creative tool, this failure rate may be acceptable (users can retry or refine their scribbles), but the paper does not characterize what types of failures occur or whether they are concentrated in specific sub-tasks, editing categories, or difficulty levels. The lack of per-task breakdown is a significant gap: we don't know if image fusion (16K training samples) performs worse than instruction editing (32K+14K samples), or if doodle editing (8K samples) is the weakest link. This granularity matters for practitioners deciding whether to adopt DreamOmni3 for specific use cases.

On the claim that joint input with shared encodings "avoids the complexity and computational cost of binary masks" while "maintaining editing consistency" (Section 1, Section 3.2).

The ablation evidence (Tables 3 and 4) strongly supports that joint input + shared encodings improves performance over not using them. However, the paper never actually compares against a binary mask baseline — neither architecturally (training a version of DreamOmni3 that uses mask channels instead of joint RGB input) nor conceptually (comparing against a strong inpainting-based method like MagicQuill on the scribble benchmark). The claim that joint input "avoids complexity" relative to binary masks is argued theoretically (multi-region linking difficulty, architectural incompatibility) but not demonstrated empirically. A skeptical reader could argue: a well-engineered mask-based system with color-coded masks and instruction-to-mask correspondence might match or exceed DreamOmni3's performance, and the paper provides no evidence to the contrary.

Furthermore, the specific failure modes that joint input is designed to address — loss of pixel consistency in non-edited regions — are demonstrated qualitatively (Figure 3 shows DreamOmni3 preserving backgrounds better than baselines) but not quantified. The paper does not report a dedicated "pixel consistency" metric (e.g., PSNR or SSIM in non-edited regions) that would directly measure whether joint input achieves its stated goal of preserving non-edited pixels. The improvement from 35.00% to 45.00% in editing success (Table 3) could be driven by better scribble localization rather than better pixel preservation — the success rate metric conflates multiple criteria. Without a targeted metric, we cannot confirm that joint input's mechanism operates through the claimed pathway (providing original pixel access) rather than through some other effect (e.g., the additional image tokens providing more conditioning signal that stabilizes generation).

On the claim that the data synthesis pipeline produces "high-quality, comprehensive" training data for scribble-based editing and generation (Section 1).

The data pipeline is the paper's most under-validated contribution. The 117K training samples clearly enable DreamOmni3 to learn scribble understanding where baselines fail, but "high-quality" and "comprehensive" are claims about coverage and realism that the paper does not systematically evaluate. Several concerns:

Realism of synthetic scribbles. The scribbles are pasted from a template library of 30 hand-drawn symbols. This is a static, finite set. Real users produce an unbounded variety of scribbles — varying stroke width, opacity, style, precision, and shape. A model trained on 30 fixed scribble styles may fail on scribbles that significantly deviate from this distribution (e.g., very thick brush strokes, partially transparent annotations, highly irregular freehand loops). The paper provides no evaluation of robustness to scribble style variation.
Doodle realism. The doodles are generated by GPT-Image-1, a model whose doodle generation characteristics are not analyzed. If GPT-Image-1 produces systematically different doodles than humans (e.g., cleaner lines, more regular shapes, different abstraction levels), the model may learn a distribution that doesn't match real user input. The paper's argument that GPT-Image-1 produces "doodles with non-strict pixel consistency" that better match human drawing than Canny edges is plausible but empirically unvalidated.
Instruction diversity and quality. The paper inherits instructions from DreamOmni2's dataset, which covers "a wide range of both concrete objects and abstract attributes" (Section 3.1). But the instructions are presumably written to describe image-to-image edits (e.g., "change the color of the bag to purple"), not to describe scribble-mediated edits (e.g., "change the color of the bag in the red circle to purple"). The scribble adds spatial disambiguation, but it also changes the instruction semantics — the instruction now must reference scribble colors and may need to be more concise (since location is handled visually). The paper does not analyze whether the inherited DreamOmni2 instructions, with scribble references inserted, form natural and unambiguous descriptions of the edits. If the training instructions are awkward or ambiguous, the model may learn to rely primarily on the scribble signal and ignore the text, which could limit performance on tasks where text and scribble information must be integrated.
Dataset balance and coverage. The paper reports dataset sizes per sub-task but does not analyze coverage of editing categories (what fraction of samples involve color changes vs. object insertion vs. style transfer vs. abstract attribute modification), difficulty (simple single-object edits vs. complex multi-object scene modifications), or failure modes. A dataset may be "comprehensive" in covering task types but still have blind spots — for example, very few samples of edits requiring fine-grained shape modification, or edits where the scribbled region contains multiple overlapping objects.

On the claim that DreamOmni3 "can be combined with language and image instructions, enhancing the creative usability of unified models" (Section 1).

This claim is about usability and workflow integration, not just accuracy. The experiments do not directly test usability — there is no user study measuring task completion time, error rates, user satisfaction, or preference between scribble-based and text-only interfaces. The success rate metrics measure whether the model correctly interprets scribbles when they are provided, but they don't measure whether providing scribbles is easier or more effective than providing equivalent information through text. A proper usability evaluation would compare: (a) users editing solely with text descriptions vs. (b) users editing with scribble + text, measuring whether the scribble interface reduces editing time, improves success rates, or increases user satisfaction for the same set of tasks.

The paper's qualitative results (Figures 1, 3, 4) demonstrate that the model can handle combined language, image, and scribble instructions, but this is a capability demonstration, not a usability validation. The gap between "the model supports this input format" and "this input format improves creative workflows" is substantial and unfilled by the reported experiments.

On the claim that the DreamOmni3 benchmark "enables a more accurate evaluation of the model's generalization and real-world performance" (Section 1).

The benchmark uses real images (not synthetic training data), which is a strength for generalization assessment. However, several factors limit how much we can conclude about "real-world performance":

Benchmark size is undisclosed. The paper does not state how many test cases are in the benchmark. If the benchmark contains, say, 80 test cases total (as might be inferred from the percentage precision: 57.50% = 46/80), then the effective sample size per sub-task could be as low as ~10-20 cases. At this scale, individual annotator disagreements or ambiguous cases can swing success rates by several percentage points. The reported differences between DreamOmni3 and GPT-4o on editing (57.50% vs. 58.75%, a 1.25 point gap) would be well within the noise floor.
Annotator reliability is not reported. With 5 human reviewers per case and a majority-vote threshold (need >3 approvals), we need to know how often the annotators agreed to assess result reliability. If annotators frequently split 3-2 on ambiguous cases, the success/failure boundary is noisy. The paper states that "VLM-based assessments align closely with human evaluations" but provides no quantitative measure of this alignment (correlation, agreement rate, Cohen's kappa).
Image source and diversity. The benchmark uses "real images" but does not specify their source (are they from standard datasets? web-scraped? user-provided?), diversity (do they span different domains like indoor, outdoor, portraits, products?), or difficulty (are there easy cases that all models get right, padding the success rate?). Without this information, we cannot assess whether the benchmark represents the distribution of images users would actually edit in practice.
Single benchmark, single model family. All results are on the DreamOmni3 benchmark with the FLUX Kontext + Qwen2.5-VL architecture. There is no cross-benchmark validation (e.g., adapting an existing image editing benchmark to include scribble annotations) and no testing of whether the approach transfers to other base model families (e.g., Stable Diffusion 3, DALL-E). The generalizability of the joint input + shared encoding approach across architectures is untested.

Missing experiments that would strengthen the paper.

Per-task breakdown of editing and generation results. The paper defines seven sub-tasks but reports only aggregate metrics. Breaking down success rates by sub-task would reveal which tasks are well-solved (possibly instruction-based editing with simple spatial queries) and which remain challenging (possibly doodle generation or image fusion with complex instructions). This diagnostic information matters for both research prioritization and practical deployment decisions.
Comparison against a strong mask-based inpainting baseline. The paper argues theoretically that joint input is superior to binary masks, but without empirically comparing against a method like MagicQuill (which uses edge maps for structure preservation) or a modern inpainting model adapted for instruction-following, the claim remains a design argument rather than a demonstrated advantage. A fair comparison would give the mask-based method the same scribble inputs, convert them to masks, and evaluate on the same benchmark.
Scribble style robustness study. Testing DreamOmni3 with scribbles that differ from the training distribution — hand-drawn by actual users (not from the template library), varying stroke width, partial opacity, highly irregular shapes — would establish whether the model has learned robust scribble interpretation or has overfit to the 30-template style. This is critical for real-world deployment where scribble style is uncontrolled.
Data scaling ablation. Training DreamOmni3 on subsets of the full dataset (e.g., 25%, 50%, 75%) would reveal whether performance is data-limited (still improving with more data) or capacity-limited (saturated at current data scale). This would guide future data collection efforts.
Instruction-only comparison. Evaluating DreamOmni3 on the benchmark without providing scribbles — just the text instruction — would test whether the presence of scribbles actually improves editing accuracy over text-only instructions. If the model achieves similar performance text-only on simple spatial tasks, the added value of scribbles would need to be demonstrated on tasks where language alone is insufficient.
Failure mode analysis. The paper does not systematically categorize why edits or generations fail. Do failures cluster in specific sub-tasks (doodle generation is harder than instruction editing), specific edit types (color changes succeed more often than object insertion), or specific image characteristics (cluttered scenes fail more than simple compositions)? A taxonomy of failure modes would be more informative than aggregate success rates for understanding the model's limitations and guiding future work.

Summary assessment.

The experiments convincingly demonstrate that:

Scribble-based editing and generation is a distinct capability that does not emerge from general-purpose vision-language or diffusion models — it requires dedicated training data and architectural design (supported by the poor performance of all open-source baselines and the specific failure modes of commercial models).
The joint input scheme with shared index and position encodings causally improves performance over single-image scribble input, with the improvement concentrated in editing tasks where pixel consistency matters (supported by Tables 3 and 4).
DreamOmni3, through its combined data synthesis and architectural innovations, achieves performance competitive with the best commercial model for scribble-based editing and substantially better for scribble-based generation (supported by Tables 1 and 2, Figures 3 and 4).

However, the experiments fall short of demonstrating:

That scribble-based interfaces actually improve user workflows over text-only or mask-based interfaces (no user study, no comparative usability evaluation).
That the data synthesis pipeline produces realistic and comprehensive training data (no analysis of scribble style robustness, instruction quality, dataset coverage, or data scaling behavior).
That the benchmark and evaluation protocol are reliable enough to support the reported performance comparisons (undisclosed benchmark size, no annotator reliability metrics, no confidence intervals).
That DreamOmni3's approach generalizes beyond the specific base architecture and benchmark used (no cross-architecture or cross-benchmark validation).
That the joint input scheme is superior to mask-based approaches rather than merely superior to no spatial annotation mechanism (no mask-based baseline comparison).

The strongest claims — that DreamOmni3 introduces a new task category with working implementations and that joint input with shared encodings is an effective architectural pattern for annotation-based editing — are well-supported. The broader claims about "enhancing creative usability," "simplifying" over mask-based methods, and providing "comprehensive" training data are better understood as motivating visions that the current experiments gesture toward but do not fully validate.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Not Amortized in the Headline Efficiency Gains

The assumption or constraint. The paper's compute-optimal scaling framework requires estimating each prompt's difficulty before allocating the test-time compute budget. The method for doing so—generating 2048 samples per question and averaging either ground-truth correctness (oracle) or PRM final-answer scores (predicted)—is extraordinarily expensive. The authors explicitly acknowledge this in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. In any realistic deployment, the total compute cost would be (difficulty estimation budget) + (strategy execution budget). Since the estimation step uses 2048 generations per question, it alone consumes more compute than the largest test-time budgets studied (256–512 generations). The reported 4× efficiency gains over best-of-N—where, for example, compute-optimal search with 16 generations matches best-of-N with 64 generations (Figure 4)—are computed after difficulty is already known. If the estimation cost were amortized across test questions, the effective efficiency improvement would shrink dramatically. For single-question deployments (the typical interactive use case), the estimation cost would dominate entirely, making the compute-optimal approach far less efficient than a uniform best-of-N strategy using the same total budget. For batch deployments where many questions share the same difficulty distribution, the estimation cost could be amortized across the batch, but the paper provides no analysis of how batch size affects amortized efficiency.

What evidence exists in the paper. The difficulty estimation protocol is described in Section 3.2, and the compute cost (2048 samples per question) is stated explicitly. The predicted-difficulty variant eliminates the need for ground-truth labels but does not reduce the sample count—it still requires 2048 generations plus PRM scoring. The main results (Figures 4, 8) plot compute-optimal scaling curves where the x-axis shows only the strategy execution budget, not the total budget including estimation. The paper does not provide any analysis of how the 4× figure changes if estimation cost is included, nor does it test whether fewer estimation samples (e.g., 256 or 512) would suffice.

Mitigation status. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) that would eliminate the sampling overhead entirely. It also frames the estimation problem as an "exploration-exploitation tradeoff" (Section 3.2) but proposes no concrete solution. A natural approach—using a small initial batch of samples (e.g., 4–8) to coarsely estimate difficulty and then allocating the remaining budget adaptively—is mentioned as future work but not implemented or evaluated. The current results therefore represent an upper bound on achievable efficiency that cannot be realized in deployment without a cheap difficulty estimator.

6.2 Hard Problems Remain Fundamentally Unsolved Regardless of Budget

The assumption or constraint. The paper's test-time compute strategies are built on the premise that the base model's proposal distribution already contains correct solutions at some non-trivial rate—the strategies amplify existing capability through better search and refinement. This premise fails entirely for the hardest problems, where the base model's pass@1 is near zero. The paper defines five difficulty quintiles based on the base model's pass@1 rate; bin 5 (the hardest) represents problems where the model almost never produces correct answers even with 2048 samples.

The consequence. For any problem where the base LLM's capability is fundamentally insufficient, no amount of test-time compute helps. This is demonstrated across all experimental settings:

Search (Figure 3, right): Bin 5 accuracy hovers at 1–3% for all methods and all budgets (4 to 256 generations). Neither beam search nor best-of-N weighted makes progress.
Revisions (Figure 7, right): Bin 5 accuracy is roughly 2–3% regardless of the sequential-to-parallel ratio, even at 128 generations.
FLOPs-matched comparison (Figure 9): The bin 5 scaling line is essentially flat near 0–5%, while the ~14× larger pretraining-scaled model achieves non-trivial performance. At R ≫ 1 (high inference-to-pretraining ratio), bin 5 shows a −52.9% disadvantage from using test-time compute instead of the larger model for PRM search (Figure 1, bottom-right bar chart).

This means the compute-optimal framework offers no path forward for genuinely novel or out-of-distribution reasoning that exceeds the base model's training distribution. It cannot compensate for capability gaps that larger pretraining would address. The practical implication is that compute-optimal test-time scaling can improve a model's performance within its existing capabilities but cannot extend those capabilities to fundamentally harder problems.

What evidence exists in the paper. The per-difficulty-bin breakdowns in Figures 3 (right), 7 (right), and 9 all show the bin 5 failure consistently across methods and budgets. The paper is explicit about this limitation in the Section 7 takeaway: on hard problems, "pretraining is almost always more effective," and shifting to test-time compute results in substantial performance losses relative to simply using a larger model.

Mitigation status. The paper acknowledges this limitation candidly and treats it as evidence for a boundary condition on test-time compute scaling: test-time compute amplifies existing capability but does not create it from nothing. There is no proposed mitigation—the limitation is fundamental to the approach. The practical recommendation is that systems should route genuinely hard problems to larger models rather than attempting to solve them through additional inference compute on a smaller model. The difficulty estimation mechanism could serve double duty here: identifying not just how to allocate inference compute, but whether to escalate to a larger model.

6.3 The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect (followed by a correct target). The training data construction (Section 6.1) pairs 0–4 incorrect answers with a correct answer, using edit-distance-based selection to ensure the last incorrect answer is structurally similar to the correct one. This means the model never sees a correct answer in its context during training—it only learns to produce correct answers when all previous answers are wrong.

The consequence. At inference time, when the revision model generates a chain of sequential revisions, it inevitably encounters correct answers in its context (produced during earlier steps). Since it was never trained to handle this situation, it lacks any signal for what to do when the current answer is already correct. The paper reports that approximately 38% of correct answers get converted back to incorrect ones in subsequent revision steps (Section 6.1):

"the model may encounter correct answers in its context (produced during earlier revisions) and incorrectly 'revise' them into wrong answers"

This reversion problem creates a fundamental tension: longer revision chains increase the chance of finding a correct answer (pass@1 per step improves throughout the chain, Figure 6 left), but also increase the chance that a correct answer will be overwritten by a subsequent incorrect revision. The result is that the revision chain cannot be trusted as a monotonic improvement process—the answer quality at step t+1 can be worse than at step t.

What evidence exists in the paper. The paper quantifies the 38% reversion rate explicitly in Section 6.1. The mitigation strategy—using majority voting or verifier-based selection across the entire chain rather than taking the final revision—is evaluated in Figure 6 (right), which shows that within-chain selection does recover performance (sequential + best-of-N weighted achieves ~41.5% at 64 generations versus ~39% for parallel). However, this mitigation is a post-hoc fix that adds complexity and does not address the root cause. The ReST^EM experiment (Appendix K, Figure 16) provides further evidence of revision fragility: on-policy RL-style training caused "sequential revisions [to] substantially hurt performance," with fully sequential dropping to ~33.5% at 256 generations compared to ~38.5% at the optimal ratio.

Mitigation status. The paper partially mitigates the reversion problem through within-chain answer selection (majority voting or verifier-based selection across the entire chain, Section 6.1). However, this is an imperfect patch: it requires generating and storing all intermediate answers, applying a selection mechanism (which may itself be imperfect), and discarding the chain structure at selection time. A more principled solution—such as training the model with mixed trajectories that include correct-to-correct transitions, or adding an explicit "stop revising" signal—is not explored. The paper does not suggest specific future work on this problem. The reversion rate means that the revision model's sequential sampling advantage (~2.5 percentage points over parallel at 64 generations, Figure 6 right) is partially offset by the risk of destroying good answers, making the net benefit of longer chains uncertain.

6.4 Single Benchmark, Single Model Family, and an Undisclosed Test Set Size

The assumption or constraint. All experimental results are on the MATH benchmark (500 test questions, from Lightman et al., 2022) using PaLM 2-S* as the base model, with the FLOPs-matched comparison using a ~14× larger model from the same family. The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified across model families, architectures, or training paradigms.

The consequence. Several aspects of the findings could be model-specific or benchmark-specific:

PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution, calibration properties, and error patterns. A model with different characteristics (e.g., GPT-4, Claude, LLaMA 3) might exhibit different difficulty-dependent scaling curves, different verifier over-optimization thresholds, or different revision model behavior.
The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities, which vary substantially across model families. The edit-distance-based training data construction might be more or less effective depending on the base model's output diversity.
The MATH benchmark consists exclusively of competition-level math problems requiring symbolic reasoning and multi-step logical deduction. The difficulty-dependent patterns—beam search hurting easy problems, revisions helping easy problems, no method helping hard problems—may not generalize to other reasoning domains (code generation, logical reasoning, scientific QA) or to tasks requiring factual knowledge rather than inference.
The 500-question test set, split into five difficulty quintiles of ~100 questions each, then further split by two-fold cross-validation (~50 questions per fold per bin), means the compute-optimal policy is selected based on a very small sample. The paper reports no confidence intervals on the scaling curves (Figures 4, 8, 9), making it impossible to assess whether observed differences between methods are statistically reliable or within sampling noise.

What evidence exists in the paper. The single-benchmark, single-model limitation is transparent in the experimental setup (Section 4, Section 5, Section 7). The paper does not include any experiments on other benchmarks (e.g., GSM8K for math, HumanEval for code) or with other base model families. The n=500 test set size is stated in Section 4, and the per-bin sample size (~100) and per-fold sample size (~50) are implicit from the five-bin two-fold cross-validation protocol described in Section 3.2.

Mitigation status. The paper does not claim generalizability beyond MATH or PaLM 2-S*. The limitation is implicit in the experimental scope rather than explicitly discussed as a concern. No future work on cross-benchmark or cross-model validation is suggested. Practitioners considering deployment in other domains or with other model families would need to replicate the key experiments—particularly the difficulty-dependent behavior of search vs. revisions and the verifier over-optimization threshold—before applying the compute-optimal framework.

6.5 Revisions and PRM Search Are Never Combined, Potentially Leaving Gains on the Table

The assumption or constraint. The paper studies two complementary axes of test-time compute—modifying the proposal distribution through iterative revisions (Section 6) and optimizing selection through PRM-guided search (Section 5)—but treats them as independent mechanisms. All experiments evaluate either search against a PRM or sequential revisions, never both combined. The authors acknowledge this explicitly in Section 8:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The paper identifies that these two mechanisms have complementary, difficulty-dependent strengths: revisions excel on easy problems where local refinement suffices, while PRM search helps most on medium-difficulty problems where global exploration is needed (Section 7 takeaway). Since they address different aspects of the inference pipeline—revisions improve the proposal distribution (generating better candidates), while PRM search improves candidate selection (finding the best among generated candidates)—there is strong reason to believe that combining them would yield gains beyond either method alone: using beam search over revision model outputs, or using the PRM to guide which revision branches to pursue, could break through the individual performance ceilings documented in Figures 3 and 7. The current results therefore represent a lower bound on what a fully integrated system could achieve.

The practical consequence is that a practitioner implementing the compute-optimal framework based on this paper would not know whether to deploy revisions, search, or both for any given difficulty tier, because the relative performance of the combined approach is unknown. The optimal policy derived from this paper (Figures 4, 8) might be suboptimal compared to a policy that includes a "search + revisions" option.

What evidence exists in the paper. The complementary difficulty-dependent behavior is demonstrated empirically: Figure 3 (right) shows beam search outperforming best-of-N on medium problems (bins 3–4) but degrading on easy problems (bin 1); Figure 7 (right) shows sequential revisions performing best on easy problems (bin 2) while a balanced sequential-parallel ratio is optimal on hard problems (bins 3–4). The two mechanisms target different difficulty regimes, suggesting combination would be beneficial, but no combined experiment exists. The paper's acknowledgment in Section 8 confirms this is a known gap, not an oversight.

Mitigation status. The paper explicitly flags this as future work (Section 8). The framework and infrastructure needed for the combination—a revision model as proposal distribution, a PRM as verifier, and a search algorithm—are all developed in the paper, so the combination is a natural next step. However, the lack of combined results means the paper's headline 4× efficiency improvement (compute-optimal vs. best-of-N) might understate what is achievable with full integration of both mechanisms.

6.6 Sequential Revisions Introduce Latency That Parallel Strategies Avoid

The assumption or constraint. The paper measures test-time compute in units of "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency. Sequential revisions are inherently serial—each revision depends on the previous one, requiring the full autoregressive generation of one answer before the next can begin. Parallel best-of-N sampling can execute all N generations simultaneously given sufficient hardware.

The consequence. A strategy that allocates, for example, 128 generations as "64 sequential × 2 parallel" takes roughly 64× longer wall-clock time than one that runs 128 parallel samples simultaneously. The paper's difficulty-dependent policy (Section 3.2) favors sequential-heavy strategies on easy problems (where Figure 7 right shows purely sequential is optimal or nearly optimal) and even on medium problems (where a sequential-to-parallel ratio of 2:1 to 8:1 is optimal at 128 generations, Figure 7 left). These strategies would impose substantial latency penalties in interactive or real-time applications.

The practical implication is that the compute-optimal policy derived from a FLOPs-only metric may be latency-suboptimal for use cases where response time matters. A policy optimized for FLOPs-efficiency-per-unit-accuracy might select sequential revisions for easy problems, but a latency-constrained deployment might prefer parallel best-of-N for those same problems because 64 parallel generations complete in the time of 1 generation, while 64 sequential revisions take 64 generation-times regardless of total FLOPs. The paper provides no guidance on this tradeoff.

What evidence exists in the paper. The latency issue is entirely unaddressed. The paper defines compute budget in terms of "generations" (Section 3.1, Section 5.3) and does not discuss wall-clock time, hardware parallelism assumptions, or latency constraints. The revision model's sequential nature is inherent in its design (Section 6.1) but is not discussed as a deployment consideration. All reported efficiency gains (4×, 4.1×) are in terms of generation count, not time-to-solution.

Mitigation status. No mitigation is proposed. The paper does not mention latency as a constraint or suggest how the compute-optimal framework could be extended to incorporate latency budgets. A practitioner deploying this system would need to independently measure the latency implications of sequential-heavy strategies and potentially modify the allocation policy to respect response-time requirements, which could reduce the effective efficiency gains.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces scribble-based interaction as a first-class modality in unified image editing and generation models, shifting the field's conception of spatial user input from an implementation detail (how to generate a mask) to a primary communication channel on equal footing with text and reference images. This is not a paradigm shift in the sense of upending fundamental model architectures—DreamOmni3 is built directly on DreamOmni2 and FLUX Kontext—but it is a meaningful reframing of the interaction design space for AI creation tools.

The reframing has three concrete dimensions:

First, from mask generation to scribble interpretation. Prior work treated user-drawn annotations as an intermediate step toward generating a binary mask, which was then fed to an inpainting or region-conditioned model. In this paradigm, the mask is the real input; the drawing is a noisy approximation that must be cleaned up. DreamOmni3 inverts this: the scribble is kept in RGB space, fed directly to the model as pixels, and interpreted in visual context. The model learns that a rough red circle around a bag means "attend to the bag, not the exact circular boundary." This shifts the computational burden from precision (the user must draw an accurate mask, or the system must extract one) to interpretation (the model must understand intent from approximate visual cues). The practical consequence is lower interaction friction—users can draw loosely and the model compensates—but also a different set of research priorities: improving scribble understanding (handling varied drawing styles, abstraction levels, ambiguous annotations) becomes more valuable than improving mask extraction.

Second, establishing RGB-space annotation as a scalable alternative to binary masks for multi-region editing. The paper identifies a genuine scaling bottleneck in mask-based approaches: each additional edit region requires a separate mask channel and a language-based correspondence mechanism, leading to combinatorial complexity as the number of edits grows. DreamOmni3's solution—color-coded scribbles in a single RGB image, with the joint input scheme preserving original pixels—handles an arbitrary number of regions without adding input channels. This is not just an architectural convenience; it changes what multi-region editing tasks are practical to request. A user can simultaneously say "make the bag in the red circle purple, delete the person in the green circle, and add a window in the blue box" without the model architecture or prompt complexity growing with each additional edit. The baseline results (Tables 1 and 2) show that existing unified models without scribble training completely fail on these tasks (DreamOmni2: 17.50% editing, 4.65% generation), confirming that spatial reasoning from RGB annotations is not an emergent capability and requires dedicated design. This establishes spatial annotations in RGB space as a distinct research subproblem within multimodal generation, analogous to how tool use or chain-of-thought reasoning became recognized subproblems in language models.

Third, demonstrating that scribble-specific training produces qualitatively different behavior from general-purpose visual reasoning. The comparison with GPT-4o is instructive not because DreamOmni3 outperforms it (on editing, they are comparable at ~57-58% human evaluation), but because their failure modes differ. GPT-4o exhibits color cast artifacts (yellowing), background pixel inconsistency, and scribble retention in outputs—symptoms of a system using general reasoning to approximate spatial editing without dedicated training. DreamOmni3 avoids these artifacts because its training data explicitly teaches that scribbles are instructions to be interpreted and discarded, not visual content to be preserved or blended. The generation results (53.49% human for DreamOmni3 vs. 39.53% for GPT-4o) amplify this pattern: scribble retention is a dominant failure mode for the untrained commercial model, while DreamOmni3 cleanly removes annotations from its outputs. This finding suggests that the field should not assume spatial reasoning capabilities will emerge naturally from scaling general-purpose multimodal models—dedicated training data and architectural accommodations (joint input, shared encodings) produce more robust behavior for this specific interaction paradigm, even when the general-purpose model is far larger and more capable overall.

Reconciling the inpainting vs. instruction-editing tension. The paper resolves a practical contradiction in the editing literature: inpainting methods preserve non-edited regions perfectly but cannot reason about contextual effects (lighting, shadows, reflections) because they treat the masked region as a local texture-completion problem; instruction-based editing methods handle contextual effects but often fail to preserve non-edited region pixel consistency because they regenerate the entire image. DreamOmni3's joint input scheme offers a middle path: the model sees both the clean source pixels (enabling preservation) and the scribble annotations (enabling localized editing), and the shared encoding scheme tells it these are two views of the same image (enabling spatial correspondence). The 10-percentage-point improvement from adding joint input in editing (35.00% → 45.00%, Table 3) provides evidence that this architectural pattern addresses the core tension. This doesn't make inpainting obsolete—for tasks requiring exact pixel-level preservation with no contextual reasoning, inpainting may still be preferable—but it expands the design vocabulary for systems that need both localization and contextual awareness.

What research directions become more attractive. The paper makes several directions newly tractable or more compelling:

Richer scribble vocabularies become worth developing because the model architecture (colored scribbles in RGB space, joint input) naturally supports them. Arrow annotations, opacity-varying brush strokes, text written directly on the image, and pressure-sensitive line weights are all expressible in RGB space and could be added to the training data without architectural changes. Prior mask-based systems would require new input channels for each new annotation type; DreamOmni3's framework handles them as visual variations.
Interactive refinement loops become more natural because the scribble format supports incremental, visual communication. A user can draw a rough circle, see the result, then add more precise doodles to refine details—each interaction cycle uses the same input format. This contrasts with text-based refinement, where spatial corrections are linguistically awkward.
Cross-modal instruction understanding (scribble + text + reference image) emerges as a distinct capability worth benchmarking and optimizing, rather than being treated as a side effect of general multimodal training. The paper's task taxonomy (seven sub-tasks across editing and generation) provides a vocabulary for describing what specific cross-modal integration a model can perform.

What research directions become less attractive. The paper weakens the case for several approaches:

Mask extraction as a primary research investment for spatial editing. If scribbles can be processed directly in RGB space with architectures like joint input + shared encodings, improving the precision of mask extraction from user drawings becomes a secondary optimization rather than a fundamental capability. The paper's results suggest that interpretative robustness (handling rough, varied scribbles) is the more impactful lever.
VLM-only evaluation of spatial editing tasks. The paper's finding that Gemini and Doubao evaluations disagree with human judgments in specific ways—VLMs rate GPT-4o higher on generation (58.14% Gemini) than humans do (39.53%), likely because VLMs miss scribble retention artifacts—suggests that automated evaluation of spatially-conditioned generation requires specialized metrics beyond off-the-shelf VLMs. This is a caution against the trend of replacing human evaluation entirely with VLM judges for tasks where the failure modes are visual artifacts rather than semantic errors.
Training on clean, precise annotations as a default assumption. The paper's decision to use 30 hand-drawn imperfect scribble templates—and to use GPT-Image-1 for doodles rather than Canny edges—makes a specific argument that training data for human-interaction tasks should match the imperfection characteristics of human input. If this principle generalizes, it argues against the common practice of training on clean ground-truth annotations and hoping for robustness, instead advocating for deliberately noisy, human-style training data.

Follow-Up Research This Work Enables

Scribble style robustness: testing generalization beyond the 30-template training distribution. The paper trains on scribbles from a manually-created library of 30 squares and circles, but real users produce an unbounded variety of drawing styles—varying stroke width, opacity, precision, speed-dependent line quality, and abstraction level. A critical stress-test would collect scribbles from actual users (e.g., via a crowd-sourcing study where participants are asked to "circle the bag" or "draw a dog here" on given images) and evaluate DreamOmni3's success rate on these real human scribbles compared to its performance on the template-library scribbles from the benchmark. If performance degrades significantly (e.g., from 57.50% to below 40%), the template library is insufficient and the data synthesis pipeline needs to incorporate more diverse scribble sources. If performance holds, it validates the 30-template approach as capturing sufficient variation. This experiment matters because the paper's core claim—that scribble-based interaction is practical for real users—hinges on robustness to natural drawing variation.

Doodle generation training data: Canny edges vs. GPT-Image-1 ablations. The paper argues theoretically that GPT-Image-1-generated doodles better match human sketches than Canny edge detection, but it never empirically validates this choice. A direct experiment would train three variants of the doodle editing and doodle generation tasks: (a) using Canny edges extracted from target objects, (b) using GPT-Image-1 doodles (the paper's approach), and (c) using actual human-drawn doodles collected for a subset of objects. Evaluating these three variants on a test set of real human doodles would reveal whether the aesthetic correction from GPT-Image-1 actually improves generalization to human input, and whether the gap between Canny and human is as large as the paper assumes. A negative result—Canny-trained models performing comparably to GPT-Image-1-trained models on human doodles—would simplify future data pipelines significantly by removing the dependency on a proprietary model for training data generation.

Joint input + shared encodings applied to other annotation types. The joint input scheme with shared index and position encodings is presented as a general architectural pattern for feeding annotated images alongside clean originals, not as scribble-specific. A natural extension is testing this pattern with other annotation modalities: (a) depth maps overlaid on RGB images for depth-conditioned editing, (b) segmentation maps painted in different colors for semantic region editing, (c) edge maps for structure-preserving editing, or (d) text rendered directly on the image (e.g., "make this blue" written next to an object). If the shared encoding scheme generalizes, a single architecture could support multiple annotation types by training on mixed data—the model would learn to interpret any RGB-space annotation in context, with the instruction text disambiguating annotation type. The experiment would train a model on a mixture of annotation types and evaluate per-type performance compared to dedicated single-type models. If performance matches or exceeds dedicated models, it suggests the joint input pattern is a general solution for visual instruction following, not a scribble-specific hack.

Per-task scaling analysis: how much data does each sub-task need? The paper reports total dataset sizes (32K, 14K, 16K, 8K per sub-task) and aggregate performance, but provides no evidence about whether these dataset sizes are sufficient or excessive. A data scaling study would train DreamOmni3 on subsets of the full dataset—25%, 50%, 75%, and 100%—and measure the success rate curve per sub-task. This would answer: (a) Is performance still improving with more data (suggesting data collection is the bottleneck), or has it saturated (suggesting architecture or base model capacity is the bottleneck)? (b) Which sub-tasks are most data-hungry? Doodle editing (8K samples) might need more data to match the performance of instruction editing (32K samples); if so, future data collection should prioritize doodle data. (c) What is the minimum viable dataset size for acceptable performance? This matters for practitioners who cannot replicate the full 117K-sample pipeline.

Combining scribble editing with iterative refinement loops. The paper treats scribble-based editing as a single-shot task: provide scribbles and an instruction, get the output. A natural extension is interactive refinement: the user provides an initial scribble, the model outputs an edit, the user provides additional scribbles on the output to refine it, and the model edits again. This requires the model to handle scribbles on its own outputs (which may contain artifacts from the first edit) and to understand that new scribbles modify or correct previous edits. The experiment would be a multi-turn evaluation: a benchmark of editing tasks that require 2-4 refinement steps to achieve the desired result, measuring whether DreamOmni3 can interpret incremental scribble corrections or whether it treats each turn as an independent edit. This tests a capability—sequential visual reasoning across edit turns—that the paper's training data (constructed from single-step edits) does not explicitly teach. If the model fails, it motivates training on multi-turn scribble trajectories; if it succeeds, it demonstrates emergent sequential reasoning from the single-step training.

User study comparing scribble-based vs. text-only editing efficiency. The paper claims that scribble-based interaction "enhances creative usability" and "lowers the user barrier" (Section 1), but provides no empirical usability evidence. A controlled user study would give participants a set of image editing tasks (e.g., "change the color of the third vase from the left," "add a tree behind the house," "make this person's hairstyle match that person's") and randomly assign them to either a text-only interface (describe the edit in words) or a scribble + text interface (draw circles/boxes and provide a short instruction). Metrics would include: task completion time, editing accuracy (judged by whether the output matches the intended edit), number of attempts per task, and user satisfaction scores. The specific hypothesis is that scribble + text reduces completion time and increases accuracy for tasks requiring precise spatial specification, while being neutral or slightly worse for tasks where text alone is sufficient (e.g., global style changes). This experiment would ground the paper's usability claims in quantitative evidence and identify the specific editing scenarios where scribble input provides the most value.

Stress-testing the difficulty estimation cost with adaptive allocation strategies. The paper's compute-optimal framework requires an expensive difficulty estimation step (2048 samples per question) that is not amortized in reported efficiency gains. A practical follow-up would test adaptive difficulty estimation: start with a small budget (e.g., 4-8 parallel samples), use the verifier's score distribution on these samples as a coarse difficulty signal, and then allocate the remaining budget adaptively based on this signal. The experiment would sweep the initial estimation budget and measure whether adaptive allocation recovers most of the gains of oracle difficulty estimation while dramatically reducing total compute. If a 4-sample initial estimate followed by adaptive allocation achieves, say, 90% of the compute-optimal performance, the framework becomes practical for deployment without requiring a cheap standalone difficulty predictor. This experiment is a natural bridge between the paper's analytical results and real-world deployment constraints.

Practical Applications and Downstream Use Cases

Consumer-grade AI photo editing with spatial guidance. The most direct application is integrating scribble-based editing into photo editing applications aimed at non-expert users. Current AI photo editors (e.g., Photoshop's Generative Fill, Luminar Neo) require users to paint precise masks or use text prompts to describe edit locations—both of which impose cognitive and motor-skill barriers. DreamOmni3's approach—roughly circling a region and providing a short instruction—reduces the interaction to what a user would naturally do when pointing at a printed photo. The 57.50% human-evaluated editing success rate and the model's demonstrated robustness to imperfect scribbles (from the 30-template training) suggest it could handle casual user input without requiring careful brushwork. The joint input scheme's preservation of non-edited region pixels (evidenced by the +10.00 point improvement over single-image input, Table 3) means users don't need to worry about their rough circle bleeding edits into surrounding areas. The immediate deployment scenario is a mobile photo app where users can circle an object with their finger and say "remove this" or "make this red"—interacting at the speed of thought rather than the speed of precise mask painting or verbose text description.

E-commerce product image customization at scale. Online retailers increasingly offer product customization (e.g., "show this sofa in blue," "add a monogram here") but current solutions either require pre-rendered variants (expensive and inflexible) or text-based generation (imprecise for spatial specifications like logo placement). DreamOmni3's scribble-based multimodal editing (Table 1: 52.50% Gemini, 45.00% Doubao for all editing sub-tasks) combined with image fusion (inserting cropped objects from reference images into scribbled regions) enables a workflow where customers draw on a product image to indicate where they want a logo, monogram, or color change, and the system generates the customized variant. The 16K image fusion training samples mean the model has specific training on the "extract from reference, insert at scribble location" workflow. The generation capabilities (53.49% human success rate, Table 2) could further enable "design from scratch" scenarios where a customer scribbles a rough layout on a blank canvas—"sofa here, lamp there, window here"—and the model generates a photorealistic room scene. The practical benefit is reducing the content creation cost for product variants from manual rendering or photography to automated AI generation, with the scribble interface making the process accessible to non-technical merchandising staff.

Collaborative design tools mixing multiple modalities. Professional design workflows (interior design, architectural visualization, fashion design) frequently involve referencing multiple source images, making precise spatial annotations, and iterating on designs. DreamOmni3's support for scribble + text + multiple reference images (demonstrated in Figures 3 and 4 with up to three reference images and multiple colored scribbles) enables a workflow where a designer can say "take the sofa from image 1, the painting from image 2, place them at these locations, and make the wall color match the accent color in image 3"—all in a single instruction with visual annotations. The color-based scribble differentiation (red circle, blue box, green doodle) provides an intuitive mapping between reference content and target locations that avoids the text-to-coordinate translation burden. The LoRA-based compositional architecture means the system can retain all original text-only and image-only editing capabilities while adding scribble support, so designers don't need to switch between specialized tools for different editing modes. The practical value is reducing the number of software tools and interaction modes needed in a creative workflow—a single model handles text prompts, reference-based edits, and spatial annotations within one interface.

When to Prefer This Method

The paper explicitly positions scribble-based editing and generation against two alternative paradigms: text-only unified models (DreamOmni2, Omnigen2, GPT-4o) that lack spatial input channels, and mask-based editing methods (inpainting, MagicQuill, object insertion via compressed IDs) that require precise binary masks. The choice between these approaches depends on the specific editing scenario:

Prefer scribble-based interaction (DreamOmni3) when:

The edit requires precise spatial localization that is awkward to describe verbally—e.g., "the third person from the left, partially behind the table," or when multiple similar objects need disambiguation. The paper's baseline results show text-only models achieve only 17.50% human editing success on scribble tasks (DreamOmni2, Table 1), confirming that language alone is insufficient for these scenarios.
The user needs to specify multiple edit regions simultaneously with different operations. Scribbles differentiated by color (red circle = change color, blue box = insert object, green circle = delete) scale to an arbitrary number of regions without additional input channels, whereas binary masks require a separate mask per region and language-based mask-to-instruction linking.
Pixel-level consistency in non-edited regions is critical (e.g., product photography, archival image restoration). The joint input scheme's +10.00 point improvement over single-image scribble input (Table 3, Scheme 3 vs. Scheme 4) demonstrates that seeing the clean source image enables the model to preserve non-edited pixels, whereas text-only instruction models often regenerate the entire image with inconsistencies.
The user is not a skilled mask-drawing expert and needs rough, approximate annotations to be interpreted correctly. The training on 30 imperfect hand-drawn scribble templates (rather than precise geometric shapes) optimizes for interpretative robustness to loose input.
The workflow involves combining spatial annotations with reference images—e.g., "insert this object from image 2 at this location." The image fusion sub-task (16K training samples) specifically handles this pattern.

Prefer text-only models when:

The edit is global rather than localized—e.g., "make this image look like a watercolor painting" or "increase the overall brightness." Scribbles add unnecessary interaction overhead when there is no spatial component to the instruction.
The user's editing intent is primarily semantic or abstract and does not benefit from spatial disambiguation—e.g., "make this image more dramatic" or "change the season to autumn." The scribble provides no useful signal for these edits.
Latency or simplicity is paramount and the spatial information can be approximated through text—e.g., quick social media edits where "crop and brighten" is sufficient.

Prefer mask-based methods when:

Extreme pixel-level precision is required at the exact mask boundary—e.g., cutting out a product from a white background with hair-level detail. Mask-based methods that directly condition on the mask channel may achieve finer boundary control than scribble interpretation, which infers approximate regions from rough annotations.
The edit is a pure inpainting task where the masked region should be completely regenerated without preserving any original structure or color, and no contextual reasoning (lighting, shadows) is needed. DreamOmni3's training data, built from instruction-editing pairs, teaches contextual awareness that may be undesirable for local texture replacement.
The training data availability favors mask-based approaches—if the user has access to a large dataset of (image, mask, instruction, output) tuples and no DreamOmni2-style paired editing data for retrofitting scribbles, building a mask-based pipeline may be more straightforward than constructing a scribble synthesis pipeline from scratch.