Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation

ArXiv: 2410.13848

🎯 Pitch

A single vision encoder cannot serve two masters: the coarse semantics needed for image understanding inherently clash with the fine-grained details required for generation. Janus proves that simply separating the visual encoding pipelines allows a 1.3B model to beat specialized 7B models at their own tasks, showing that architectural conflict, not model scale, was the real bottleneck.

---

1. Executive Summary

This paper introduces Janus, an autoregressive framework that unifies multimodal understanding and generation by decoupling visual encoding into separate pathways—a SigLIP encoder for high-level semantic understanding and a VQ tokenizer for fine-grained spatial generation—while processing both through a single shared transformer. Evaluated on a 1.3B-parameter model using the DeepSeek-LLM backbone, Janus surpasses prior unified models across both task families, achieving 87.0 on POPE and 69.4 on MMBench for understanding while reaching 61% overall accuracy on GenEval and an FID of 8.53 on MSCOCO-30K for generation—outperforming task-specific models with substantially more parameters, including LLaVA-v1.5 (7B) on several understanding benchmarks and SDXL (2.6B) on compositional generation. The decoupling resolves the inherent conflict between the coarse semantic granularity required for understanding and the fine-grained spatial detail required for generation, establishing that a single transformer can excel at both tasks only when visual encoding is specialized per modality rather than shared.

2. Context and Motivation

The Core Problem: Unified Multimodal Models Sacrifice Understanding for Generation

The paper addresses a specific, empirically demonstrable problem in the design of multimodal foundation models: when a single visual encoder is used for both understanding and generation tasks, the encoder must compromise between the fundamentally different representational requirements of each task, and this compromise consistently degrades multimodal understanding performance.

To understand why this happens, we need to be precise about what "multimodal understanding" and "visual generation" demand from their visual representations. In multimodal understanding tasks—visual question answering, image captioning, diagram interpretation—the visual encoder's job is to extract high-level semantic features: object categories, visual attributes, spatial relationships, and scene-level meaning. These representations need to be invariant to low-level details like texture patterns, exact pixel values, or minor variations in lighting. The features must support complex semantic reasoning when processed by the language model, which means they benefit from being compressed, abstracted, and aligned with linguistic concepts. Think of this as asking "what is in this image and what does it mean?"

In visual generation tasks—text-to-image synthesis—the encoder faces an entirely different imperative. Converting images into discrete tokens that a language model can then predict autoregressively requires preserving fine-grained spatial structure, textural details, and global consistency. Every token the encoder produces needs to carry enough information that a decoder can faithfully reconstruct the original image. The representation must be "low-dimensional" in the sense that it operates at the pixel or patch level, encoding local features that collectively reconstruct the whole. Think of this as asking "exactly what pixels should be here, and with what precise color and texture?"

Section 1 frames this tension explicitly:

"...the representations required by multimodal understanding and generation tasks differ significantly. In multimodal understanding tasks, the purpose of the vision encoder is to extract high-level semantic information... Therefore, the granularity of the vision encoder's representation tends to mainly focus on high-dimensional semantic representation. By contrast, in visual generation tasks, the main focus is on generating local details and maintaining global consistency in the image. The representation in this context necessitates a low-dimensional encoding that is capable of fine-grained spatial structure and textural detail expression."

The consequence of forcing these conflicting requirements into a single encoder is not hypothetical—it is demonstrated directly in the paper's ablation study (Table 5). When a single VQ tokenizer encodes images for both tasks (Exp-A), multimodal understanding scores collapse: POPE drops to 60.1 (vs. 87.0 with decoupled encoders), MMB drops to 35.0 (vs. 69.4), and SEED-Bench falls to 34.9 (vs. 63.7). These are not minor degradations—understanding performance is roughly halved.

Why This Matters: The Gap Between Unified and Task-Specific Models

The problem is important for both practical and theoretical reasons. On the practical side, the vision of a single model that can both understand and generate images is compelling: it reduces model redundancy (no need for separate understanding and generation systems), enables tighter integration between comprehension and creation (a model could analyze an image, then generate a modified version based on its analysis), and opens the door to emergent cross-task capabilities that isolated systems cannot achieve. However, if unified models consistently underperform task-specific ones—particularly on understanding benchmarks—they will not see real-world adoption. The paper's Figure 1 makes this concrete by comparing Janus against both unified models (Show-o) and understanding-only models (LLaVA-Phi, InstructBLIP), showing that Janus not only beats unified models but matches or exceeds larger task-specific models. This changes the equation: a unified model no longer means an unavoidable quality trade-off.

On the theoretical side, the paper illuminates a fundamental design principle that had been overlooked in prior work: the visual representation space for understanding and generation should not be shared because the optimization pressures are orthogonal. This insight extends beyond the specific architecture choices in Janus—it suggests that future multimodal systems handling even more modalities (audio, point clouds, tactile signals) will similarly benefit from modality-specific encoders feeding into a shared reasoning backbone, rather than attempting to force all modalities through a single encoding scheme.

Prior Approaches and Where They Fall Short

The paper identifies three families of prior work, each with specific limitations:

Tool-use approaches (Emu, SEED-X, NExT-GPT). These models connect a multimodal understanding system to an external pretrained diffusion model. The LLM produces a conditioning signal (text or embeddings), and the diffusion model generates the image. Section 2.2 characterizes this precisely:

"...strictly speaking, this approach cannot be considered a truly unified model, because the visual generation functionality is handled by the external diffusion model, while the multimodal LLM itself lacks the capability to directly generate images."

The problem is architectural: the generative capability is not integrated into the model's own weights and representations. The LLM cannot learn from generation feedback, cannot iteratively refine images based on its own analysis, and the overall system's generative quality is bottlenecked by the external diffusion model. Moreover, as noted in Section 2.2, these approaches often achieve worse generation quality than using the diffusion model directly, because the LLM's conditioning signal is a lossy intermediate representation.

Single-encoder unified approaches (Chameleon, Show-o, VILA-U). These methods use a single transformer with a single visual encoder for both understanding and generation, processing discrete visual tokens autoregressively. Chameleon, perhaps the most prominent example, adopts a VQ tokenizer to encode images for both tasks. This is the approach the paper directly challenges. The failure mode is the granularity conflict described above: the VQ tokenizer that works well for generation (preserving enough detail for reconstruction) produces representations that are poorly suited for high-level semantic understanding. The paper's ablation (Table 5, Exp-A) confirms this empirically—a VQ tokenizer shared across tasks produces acceptable generation (8.72 FID on COCO) but catastrophic understanding performance.

The paper also considers a stronger single-encoder baseline (Exp-B) using a "semantic tokenizer"—a VQ tokenizer distilled from SigLIP to better capture semantic information while retaining reconstruction capability. This partially closes the understanding gap (POPE 82.4 vs. 60.1 for the plain VQ tokenizer) but still falls well short of the decoupled approach (POPE 87.0). Critically, when this semantic tokenizer is used for understanding-only training (Exp-C), its performance improves significantly (POPE 83.9, MMB 62.1, SEED 60.8)—confirming that even a stronger single encoder still imposes a trade-off between the two tasks. The representations that optimize generation reconstruction are suboptimal for semantic reasoning, and vice versa.

Understanding-only and generation-only models. On the understanding side, models like LLaVA, MobileVLM, and InstructBLIP use dedicated vision encoders (CLIP, SigLIP) with adaptors to feed into LLMs, achieving strong understanding performance but having zero generative capability. On the generation side, models like SDXL, LlamaGen, and DALL-E produce high-quality images but cannot understand them. The paper's contribution is showing that a unified model need not compromise—by decoupling the encoders, it can match or exceed both families simultaneously.

How Janus Positions Itself

The paper's central claim is not that shared encoders are theoretically impossible, but that decoupling is the correct architectural choice given current techniques. Section 1 states:

"To the best of our knowledge, we are the first to highlight the importance of decoupling visual encoding within the unified multimodal understanding and generation framework."

This is an accurate claim: prior unified approaches (Chameleon, Show-o, VILA-U, Transfusion) all use a single visual representation pathway. Some, like Transfusion, use different processing mechanisms (next-token prediction for text, diffusion for images), but the visual encoding itself is shared. Janus's innovation is recognizing that the encoding—not just the processing—needs to be task-specialized.

The paper positions decoupling not as a hack or workaround but as a flexibility advantage. Section 3.5 enumerates practical benefits: the understanding pathway can independently adopt stronger vision encoders (EVA-CLIP, InternViT), dynamic high-resolution techniques, or token compression; the generation pathway can independently adopt finer-grained tokenizers (MoVQGan), diffusion losses, or hybrid attention patterns. The shared transformer remains the integration point, but the encoders are free to optimize for their respective tasks without mutual interference.

The naming itself—Janus, the Roman god with two faces looking in opposite directions—encodes this philosophy (Section 1 footnote):

"Similarly, our model captures the inherent tension between vision tasks: understanding demands abstract, high-level semantic representations, while generation requires concrete, detailed information. By decoupling these processes into specialized encoders, our system mirrors Janus's dual nature, resolving this tension within a unified architecture."

The key theoretical move is reframing the problem: rather than asking "how can we make a single encoder serve two masters?", the paper asks "why should one encoder serve two masters when we can have two specialized encoders and one shared reasoning engine?" The shared transformer provides the unified processing—cross-modal attention, reasoning, instruction-following—while the decoupled encoders ensure that the information entering that transformer is appropriately formatted for its intended use. This is an architectural division of labor that aligns with the different informational demands of the two tasks.

3. Technical Approach

3.1 Reader Orientation

The paper designs and trains Janus, a single neural network that can both answer questions about images (multimodal understanding) and create images from text descriptions (visual generation), using two separate visual encoders that feed into one shared reasoning core. The problem is that previous unified models used a single visual encoder for both tasks, forcing it to compromise between extracting abstract semantic meaning for understanding and preserving fine-grained pixel details for generation; Janus solves this by giving each task its own specialized encoder while keeping a single transformer to process both, eliminating the trade-off without adding major architectural complexity.

3.2 Big-Picture Architecture (Diagram in Words)

Janus has five major components arranged in a Y-shaped architecture that converges at the center:

1. Text Tokenizer — the built-in tokenizer of the base language model (DeepSeek-LLM), which converts text into discrete token IDs and retrieves their corresponding embedding vectors.
2. Understanding Encoder — a SigLIP-Large-Patch16-384 vision encoder that takes an image and produces high-dimensional semantic features (a 2D grid of feature vectors capturing object categories, attributes, and scene-level meaning).
3. Generation Encoder — a VQ (Vector Quantized) tokenizer with a codebook of 16,384 entries that converts an image into a sequence of discrete IDs, each ID representing a patch of the image and collectively encoding fine-grained spatial and textural information.
4. Adaptors — two separate two-layer MLPs (one for understanding, one for generation) that project the encoder outputs into the unified embedding space of the shared transformer. These are the "translation layers" that make different visual representations compatible with the language model.
5. Shared Autoregressive Transformer — the DeepSeek-LLM (1.3B parameters) that processes the concatenated sequence of text, understanding, and generation features, plus two output heads: the built-in text prediction head for understanding tasks, and a randomly initialized image prediction head for generation tasks.

Information flows as follows: a prompt enters the system → if it contains an image for understanding, the SigLIP encoder extracts semantic features, the understanding adaptor projects them, and the resulting tokens are concatenated with text tokens → if the task is image generation, the text prompt is tokenized, the transformer autoregressively predicts image token IDs, and the generation adaptor maps codebook embeddings back to the input space for the next prediction step → the image head converts the predicted token distribution into the final discrete image token sequence, which is decoded by the VQ tokenizer's decoder into pixels.

The architecture is fully autoregressive and requires no special attention masks — all modalities are flattened into a single 1D sequence processed left-to-right by the same causal attention mechanism.

3.3 Roadmap for the Deep Dive

• First, the three-stage training procedure, because it reveals the key design philosophy: the encoders are frozen throughout, the adaptors and image head are trained first to establish cross-modal connections, then the LLM is unfrozen for unified pretraining, and finally supervised fine-tuning makes the model conversational. The order matters because each stage builds on the previous one.
• Second, the training objective (cross-entropy loss), which is deceptively simple but requires understanding which tokens contribute to the loss and why different tasks are not weighted differently — this simplicity is a deliberate design choice.
• Third, the inference procedure, particularly classifier-free guidance for image generation, because this is the only place where Janus departs from standard autoregressive decoding and the mechanism is non-obvious.
• Fourth, the decoupling design itself — why SigLIP for understanding and VQ for generation, why separate adaptors, and what the ablation studies in Table 5 reveal about the necessity of this separation. This connects directly to the motivation established in Section 2.
• Fifth, the flexibility and extensibility claims in Section 3.5, which are not just aspirational but reveal specific architectural decisions (frozen encoders, separate adaptors, unified transformer) that make Janus a platform rather than a point solution.
• Finally, the specific hyperparameters and training configurations from Table 1 and Section 4.1, because these concretize what "we trained Janus" means and enable reproducibility assessment.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an architectural innovation paper whose core contribution is the observation that visual encoding must be decoupled by task type within a unified multimodal model, and the demonstration that this decoupling enables a small (1.3B) autoregressive model to match or exceed both larger unified models and task-specific models on both understanding and generation benchmarks.

---

Three-Stage Training Procedure (Section 3.2)

The training strategy reflects a careful decomposition of the learning problem into three phases, each with a specific goal, a specific set of frozen vs. trainable parameters, and a specific data mixture. The progression moves from establishing basic cross-modal connections, to large-scale multimodal pretraining, to instruction-following alignment.

Stage I: Training Adaptors and Image Head

The goal is to create an initial conceptual bridge between visual and linguistic elements in the shared embedding space, enabling the LLM to "see" entities in images and have a rudimentary ability to generate visual tokens — all without modifying the LLM's core language capabilities or the pretrained visual encoders.

What is frozen: both the understanding encoder (SigLIP), the generation encoder (VQ tokenizer), and the entire LLM (DeepSeek-LLM 1.3B). What is trainable: only the understanding adaptor, the generation adaptor, and the randomly initialized image prediction head. This is intentionally a small parameter set — two two-layer MLPs and one output head — so training is fast and the LLM's language knowledge is preserved intact.

The data mixture is a 1:0:1 ratio of multimodal understanding data to pure text data to visual generation data (Table 1), meaning equal amounts of understanding and generation data with no standalone text. Specifically, this includes approximately 1.25 million image-text paired captions from ShareGPT4V for understanding, formatted as "<image><text>" where the image features are provided and the text is the target, and approximately 1.2 million samples from ImageNet-1k for generation, formatted as text-to-image pairs using category names: "<category_name><image>" where the category name is the input and the image tokens are the target.

The training runs for 10,000 steps with a batch size of 256, using AdamW optimizer with $\beta_1 = 0.9$ and $\beta_2 = 0.95$, a learning rate of $1.0 \times 10^{-3}$ with cosine decay, zero weight decay, gradient clipping at 1.0, and 300 warm-up steps (Table 1). The learning rate is notably high ($10^{-3}$) compared to later stages, appropriate for training randomly initialized adaptor weights from scratch.

Stage II: Unified Pretraining

The goal shifts to large-scale multimodal learning: the model must now acquire broad capabilities in both understanding and generation using diverse data, and for this the LLM itself needs to be updated.

What changes from Stage I: the LLM is now unfrozen and trained along with the adaptors and image head. The visual encoders remain frozen — this is a critical design choice maintained throughout all three stages. The paper never trains the SigLIP encoder or the VQ tokenizer as part of Janus; they are used as fixed feature extractors. This is what enables the "flexibility" discussed in Section 3.5: any improved encoder can be swapped in without retraining the entire model.

The data mixture expands substantially to a 2:3:5 ratio of understanding data to pure text data to visual generation data (Table 1). The pure text data comes from the DeepSeek-LLM pretraining corpus, ensuring the model maintains its language capabilities. The multimodal understanding data now includes interleaved image-text data from WikiHow and WIT, image caption data from multiple sources (Datacomp, ImageNet, OpenImages, and others), table and chart data from DeepSeek-VL, and re-captioned images using an open-source multimodal model. The visual generation data uses image-caption pairs from multiple datasets plus 2 million in-house samples, with a filtering step based on aesthetic scores and image sizes that retains only 20% of images from certain sources.

A specific curriculum strategy is employed for visual generation: ImageNet-1k samples appear only during the first 120,000 training steps, while images from other, more diverse datasets appear only in the later 60,000 steps. The paper explicitly motivates this by referencing PixArt-$\alpha$: the model first learns "basic pixel dependencies" on the simpler, object-centered ImageNet images before progressing to "complex scene understanding" with diverse open-domain images. This prevents the model from being overwhelmed by scene complexity before it has mastered the fundamental mapping from text concepts to visual patterns.

During text-to-image data processing, the paper applies a caption truncation trick: with 25% probability, only the first sentence of the caption is used as the conditioning text. This encourages the model to develop strong generation capabilities for short, concise descriptions — an important practical capability since users often provide brief prompts.

Training hyperparameters for Stage II (Table 1): 180,000 total steps, batch size 512, learning rate $1 \times 10^{-4}$ (constant schedule — no decay), zero weight decay, gradient clipping at 1.0, 5,000 warm-up steps. The constant learning rate at an order of magnitude lower than Stage I reflects the need for stable, sustained learning across a much longer training horizon. The batch size doubling from 256 to 512 improves throughput for the larger data volume.

Stage III: Supervised Fine-Tuning

The goal is to make the pretrained model conversational and instruction-following, capable of engaging in dialogue while retaining both understanding and generation abilities.

What changes from Stage II: all parameters are fine-tuned except the generation encoder, which remains frozen. The understanding encoder, adaptors, LLM, and image head are all trainable. The generation encoder is the only component excluded — the paper does not elaborate on why, but a reasonable inference is that the VQ tokenizer's codebook and encoder are already optimized for reconstruction and further training on the smaller, instruction-formatted dataset could destabilize the discrete token space, which would directly harm generation quality.

The data mixture shifts to a 7:3:10 ratio of understanding to pure text to visual generation data (Table 1), with instruction-formatted examples. For text understanding, data comes from LLaVA-OneVision. For multimodal understanding, instruction tuning data is drawn from multiple sources including VQAv2, GQA, IconQA, ScreenQA, and KVQA. For visual generation, a subset of image-text pairs from earlier stages (Datacomp, JourneyDB, PixelProse) plus 4 million in-house samples are used.

All data is formatted using a consistent chat template: "User: <Input Message> \n Assistant: <Response>", with multi-turn dialogues created by repeating this pattern. This is standard for instruction-tuned LLMs and ensures the model learns the conversational format. Only the assistant's responses are supervised — the user prompts and system messages are masked from the loss computation.

Training hyperparameters for Stage III (Table 1): 24,000 steps, batch size 256, learning rate $2.0 \times 10^{-5}$ (constant schedule), weight decay 0.1 (the first stage to use weight decay — providing regularization for fine-tuning), gradient clipping at 1.0, zero warm-up steps. The learning rate is two orders of magnitude below Stage I and five times lower than Stage II, appropriate for fine-tuning a mostly-trained model where large weight updates would be destructive. The introduction of weight decay at this stage helps prevent overfitting to the relatively smaller instruction-tuning dataset.

A critical design choice articulated in Section 3.2: "To ensure Janus's proficiency in both multimodal understanding and generation, we don't fine-tune separate models for a certain task. Instead, we use a blend of pure text dialogue data, multimodal understanding data and visual generation data, ensuring versatility across various scenarios." This means Janus is not two separate fine-tuned variants (one for understanding, one for generation) but a single model that handles both tasks. This is what makes it a true unified model — the same weights serve both purposes.

---

Training Objective (Section 3.3)

Janus is trained with a single, simple objective: standard autoregressive cross-entropy loss over the sequence of tokens. There are no auxiliary losses, no contrastive terms, no perceptual losses, and no task-specific loss weighting.

$\mathcal{L} = -\sum_{i=1} \log P_\theta(x_i \mid x_{<i})$

where $P_\theta(\cdot \mid \cdot)$ is the conditional probability distribution over the vocabulary (for text tokens) or over the image codebook (for image tokens) parameterized by the model weights $\theta$, $x_i$ is the $i$-th token in the sequence, and $x_{<i}$ are all tokens preceding position $i$.

What it computes: For each position in the sequence, the model predicts a probability distribution over all possible next tokens. The cross-entropy $-\log P_\theta(x_i \mid x_{<i})$ measures how surprised the model is by the actual token that appears — it is the negative log-likelihood of the ground-truth token under the model's predicted distribution. Summing over all positions $i$ gives the total sequence-level loss, which is then averaged (or summed, depending on implementation) to produce a single scalar that gradient descent minimizes.

Operationally: Given a concatenated sequence like [image_features, text_tokens] for understanding or [text_tokens, image_tokens] for generation, the model processes left to right, and at each position computes the loss only on the target tokens for that task. For pure text understanding and multimodal understanding, the loss is computed on the text sequence — the model learns to predict the text response given the image and text prompt. For visual generation, the loss is computed only on the image token sequence — the model learns to predict the discrete image token IDs given the text prompt.

Why this form: Cross-entropy is the standard maximum-likelihood objective for discrete token prediction, and it is the same loss used for language model pretraining. The paper explicitly states: "To keep the design simple, we have not assigned different loss weights to different tasks." This is a deliberate simplicity choice — no hyperparameter tuning for loss balancing, no risk of one task's loss dominating the other, and full compatibility with standard LLM training infrastructure. The alternative would be to weight understanding and generation losses differently (e.g., a $\lambda$ coefficient multiplying the generation loss), but this introduces a hyperparameter that would need to be tuned and might not transfer across data mixtures. The paper's empirical results show that equal weighting works, so the added complexity is unnecessary.

An important subtlety: the loss is computed on all target tokens simultaneously within a batch because the paper uses "sequence packing" (Section 4.1) — multiple training examples are concatenated into a single long sequence separated by attention mask boundaries, improving training efficiency. The cross-entropy loss naturally handles this because it sums over all positions independently; the masking ensures that tokens from different examples do not attend to each other.

---

Classifier-Free Guidance for Image Generation (Section 3.4)

During inference for image generation, Janus departs from standard autoregressive sampling by employing classifier-free guidance (CFG) , a technique borrowed from diffusion models that sharpens the conditioning signal.

For each token to be generated, the logit (the pre-softmax score for each possible image token ID) is computed as:

$l_g = l_u + s(l_c - l_u)$

where $l_u$ is the unconditional logit (the model's prediction when no text prompt is provided — or more precisely, when the text condition is replaced with a pad token), $l_c$ is the conditional logit (the model's prediction given the actual text prompt), and $s$ is the guidance scale, set to a default of 5 for all evaluations.

What it computes: The guided logit $l_g$ is the unconditional logit plus a scaled difference between the conditional and unconditional logits. When $s = 1$, the guided logit equals the conditional logit (standard conditional generation). When $s > 1$, the model amplifies the features that the text prompt activates relative to the un-prompted baseline. The term $l_c - l_u$ isolates the effect of the text conditioning — it is positive for tokens that the text prompt makes more likely and negative for tokens that the text prompt suppresses. Multiplying by $s$ exaggerates this effect, steering generation more aggressively toward text-consistent outputs.

Operationally: At each autoregressive step, the model performs two forward passes: one with the actual text prompt to obtain $l_c$, and one with the text replaced by a pad token to obtain $l_u$. The two logit vectors are combined using the formula above to produce $l_g$, which is then converted to a probability distribution via softmax and sampled from as usual. This doubles the inference cost per token for generation (two forward passes instead of one).

Why this form: Classifier-free guidance is well-established in diffusion models as a way to improve sample quality and prompt adherence without training a separate classifier. The paper adopts it because autoregressive image generation faces the same challenge: the model needs to produce images that are not just visually plausible but specifically faithful to the text prompt. The guidance scale $s$ provides a tunable knob — higher values produce images more tightly coupled to the prompt but can introduce artifacts (over-saturation, repetitive patterns), while lower values produce more diverse but potentially less relevant images. The default of $s = 5$ is inherited from prior work on autoregressive image generation (LlamaGen) and is not tuned in this paper.

The unconditional generation capability is trained during Stage II: with 10% probability, the text condition in text-to-image training data is replaced with a pad token. This teaches the model what images look like in the absence of text conditioning, which is necessary for the $l_u$ term in the CFG formula to be meaningful. Without this, the unconditional logits would be undefined or random.

CFG is used only for image generation, not for text or multimodal understanding, where standard temperature-based sampling from the conditional distribution is sufficient.

---

The Decoupling Design: Why SigLIP and VQ, and Why It Matters (Section 3.1 and Ablation in Section 4.5)

The core architectural decision in Janus is assigning two different visual encoders to two different tasks. This section explains the specific encoder choices and the empirical evidence that decoupling is necessary.

Understanding Encoder: SigLIP-Large-Patch16-384. SigLIP (Sigmoid Loss for Language Image Pre-training) is a contrastive vision-language model similar to CLIP but trained with a sigmoid loss rather than softmax over a batch. The "Large-Patch16-384" variant operates on 384×384 pixel images, splits them into 16×16 pixel patches, and produces a feature vector per patch. The output is a 2D grid of high-dimensional semantic features — specifically, a $(384/16) \times (384/16) = 24 \times 24 = 576$ feature vectors, each encoding the semantic content of its corresponding image region. These features are flattened into a 1D sequence of 576 tokens and passed through the understanding adaptor (a two-layer MLP) to project them into the LLM's input embedding space.

SigLIP is chosen because it is optimized for semantic understanding — its training objective (matching images to text descriptions) encourages features that capture object identities, attributes, and scene-level meaning rather than pixel-level details. This aligns exactly with what multimodal understanding needs: the model must reason about what objects are present, their relationships, and their semantic properties.

Generation Encoder: VQ Tokenizer with Codebook Size 16,384. The VQ (Vector Quantized) tokenizer is taken from LlamaGen and operates as follows: an image is passed through a convolutional encoder that downsamples it by a factor of 16 in each spatial dimension, producing a grid of continuous feature vectors. Each feature vector is quantized by finding its nearest neighbor in a learned codebook of 16,384 entries — it is replaced by the codebook vector that minimizes Euclidean distance. The index of that codebook entry becomes the discrete token for that spatial position. For a 384×384 input, this produces a $(384/16) \times (384/16) = 24 \times 24 = 576$ grid of discrete token IDs. These IDs are flattened into a 1D sequence, and the generation adaptor (another two-layer MLP) maps the codebook embeddings for each ID into the LLM's input space.

The VQ tokenizer is chosen because it preserves fine-grained spatial information necessary for reconstruction — the codebook is large enough (16,384 entries) to capture diverse visual patterns, and the downsampling factor of 16 is a balance between compression (fewer tokens to predict) and fidelity (enough tokens to reconstruct details). A smaller codebook would lose visual diversity; a larger one would be harder for the LLM to learn to predict.

Why two separate adaptors? The understanding adaptor maps from the SigLIP feature dimension to the LLM's embedding dimension; the generation adaptor maps from the VQ codebook embedding dimension to the LLM's embedding dimension. These are different source spaces (continuous semantic features vs. discrete codebook embeddings), so separate adaptors are necessary simply because the input dimensionalities and semantics differ. But beyond this mechanical necessity, separate adaptors allow each pathway to learn its own optimal projection — the understanding adaptor learns to preserve semantic information in a form the LLM can reason about, while the generation adaptor learns to preserve visual detail in a form the LLM can predict autoregressively.

The ablation evidence (Table 5). The paper's ablation studies provide the empirical justification for decoupling. When a single VQ tokenizer (Exp-A) encodes images for both understanding and generation, understanding performance collapses: POPE 60.1, MMB 35.0, SEED-Bench 34.9. The generation performance is acceptable (8.72 FID on COCO) because VQ tokenizers are designed for generation. This confirms the asymmetry: a generation-optimized encoder is terrible for understanding but a shared encoder will default to generation-compatible representations.

When a stronger "semantic tokenizer" is used (Exp-B) — a VQ tokenizer distilled from SigLIP to capture both semantic and reconstruction information — understanding improves (POPE 82.4, MMB 52.7, SEED 54.9) but still falls significantly short of the decoupled model (POPE 87.0, MMB 69.4, SEED 63.7). Crucially, when this same semantic tokenizer is used for understanding-only training without generation (Exp-C), performance jumps to POPE 83.9, MMB 62.1, SEED 60.8. The gap between Exp-B and Exp-C is the trade-off: even a stronger single encoder sacrifices understanding capability when forced to also support generation. The representations that optimize for both tasks are a compromise, not a Pareto improvement.

The decoupled model (Exp-D) achieves both strong understanding (matching or exceeding Exp-E, the understanding-only baseline) and strong generation (matching Exp-F, the generation-only baseline). This is the paper's central empirical result: decoupling eliminates the trade-off.

---

Flexibility and Extensibility (Section 3.5)

The decoupled architecture is not just a fix for the current encoder limitations — it is presented as a platform design that enables future improvements without architectural changes.

For multimodal understanding: The SigLIP encoder can be replaced with any stronger vision encoder (EVA-CLIP, InternViT) without affecting generation, because the two pathways are independent. Dynamic high-resolution techniques — where an image is split into crops, each encoded separately, and the features merged — can be applied to the understanding pathway alone, since generation uses a fixed-resolution VQ tokenizer. Token compression methods like pixel shuffle can reduce the 576 understanding tokens to fewer tokens (saving LLM computation) without affecting generation, which needs its full token grid for reconstruction quality.

For visual generation: Finer-grained tokenizers like MoVQGan can be substituted to preserve more image details. Loss functions specifically designed for image generation — such as diffusion loss, which has shown promise in recent autoregressive image models — can be used without changing the understanding pipeline. Hybrid attention patterns combining causal (autoregressive) and bidirectional attention for image tokens can be applied specifically to the generation pathway to reduce accumulated errors during autoregressive decoding, a known weakness of next-token prediction for images.

For additional modalities: The architecture naturally extends to new input types — 3D point clouds, tactile signals, EEG brain recordings — by adding new encoders with their own adaptors, all feeding into the same shared transformer. The transformer acts as a universal reasoning engine that learns to integrate information from whatever modalities are provided, without architectural changes. This is the "generalist" vision: Janus could become a model that simultaneously understands text, images, 3D shapes, touch, and brain signals, with each modality having its own specialized encoder.

The critical design choice that enables this flexibility is that encoders are frozen during all three training stages. The adaptors are the only trained interface between encoders and the LLM. This means swapping an encoder requires only retraining its corresponding adaptor (a small MLP), not the entire model. If encoders were fine-tuned during training, swapping one would require full retraining because the LLM would have adapted to the specifics of that encoder's feature space.

---

Hyperparameters and Training Infrastructure (Section 4.1)

All images are resized to 384×384 pixels, but the resizing strategy differs by task. For multimodal understanding, the long side is resized to 384 and the short side is padded with background color (RGB 127, 127, 127) to reach 384 — this preserves the original aspect ratio and avoids distortion that could confuse semantic interpretation. For visual generation, the short side is resized to 384 and the long side is cropped to 384 — this produces a square image that matches the VQ tokenizer's expected input size, but may crop out content. This asymmetry reflects the different requirements: understanding benefits from seeing the full image context, while generation needs a fixed-size grid for the tokenizer.

The base language model is DeepSeek-LLM (1.3B) with a maximum supported sequence length of 4096 tokens. This is a relatively small LLM by modern standards, which makes Janus's competitive performance against 7B models particularly notable — the decoupled visual encoding appears to compensate for the smaller language model in understanding tasks.

Sequence packing is used during training to improve efficiency: multiple training examples are concatenated into a single sequence up to the 4096-token limit, with appropriate attention masking to prevent cross-example attention. This maximizes GPU utilization by avoiding padding waste.

The training was conducted using HAI-LLM, a distributed training framework built on PyTorch, on a cluster of 16 nodes each with 8 Nvidia A100 40GB GPUs (128 GPUs total). The entire three-stage training process took 7 days. This is a relatively modest compute budget for a multimodal model, enabled by the small model size and the fact that the visual encoders are frozen (their forward passes are computed but their gradients are not).

The three-stage progression, the specific learning rate schedule (high → medium → low), the curriculum strategy for image generation data, and the frozen-encoder design all reflect a careful decomposition of the learning problem into manageable phases. The model does not attempt to learn everything simultaneously from scratch; instead, it builds capabilities incrementally, with each stage's frozen components providing a stable foundation for the newly trainable ones.

4. Key Insights and Innovations

Innovation 1: Decoupling as a Resolution of Representational Conflict, Not Merely an Architectural Variant

Prior unified multimodal models (Chameleon, Show-o, VILA-U, Transfusion) operated under a shared implicit assumption: that a single visual encoder could—or should—serve both understanding and generation tasks within one model. This assumption persisted across fundamentally different processing mechanisms (autoregressive next-token prediction in Chameleon, combined AR and diffusion in Transfusion), suggesting it was treated as architectural default rather than a decision requiring justification. The paper's central conceptual move is diagnosing why this default is wrong, and the diagnosis is what makes the contribution distinctive.

The paper identifies a representational granularity conflict: multimodal understanding demands high-level semantic features that abstract away from pixel-level detail (object categories, attributes, scene meaning), while visual generation demands low-level spatial and textural encodings that preserve reconstruction fidelity. These are not merely different—they are in tension. A representation optimized for semantic abstraction discards the texture and fine structure that generation needs; a representation optimized for reconstruction fidelity carries noise and low-level variation that complicates semantic reasoning. The paper's key insight is that this tension is inherent to the task definitions, not an artifact of particular encoder architectures.

What elevates this beyond an architectural preference is the ablation evidence in Table 5: when a single encoder (VQ tokenizer) serves both tasks, understanding performance collapses to roughly half that of the decoupled model across every benchmark (POPE 60.1 vs. 87.0, MMB 35.0 vs. 69.4). More revealingly, even a stronger single encoder—the semantic tokenizer distilled from SigLIP that explicitly attempts to capture both semantic and reconstructive information—still imposes a measurable trade-off. Exp-B (semantic tokenizer, unified training) achieves POPE 82.4 and MMB 52.7, but when the same encoder is used for understanding-only training in Exp-C, performance jumps to POPE 83.9 and MMB 62.1. The gap between Exp-B and Exp-C is the cost of compromise: roughly 1.5 points on POPE and nearly 10 points on MMB, sacrificed because the encoder must also support generation. The decoupled model (Exp-D) eliminates this gap entirely, matching or exceeding both the understanding-only (Exp-E) and generation-only (Exp-F) baselines.

The theoretical significance is that the paper establishes a design principle rather than a one-off fix: within any unified multimodal system, representations that cross the understanding-generation boundary will incur a performance penalty proportional to the mismatch between the encoder's inductive biases and the task's information requirements. This principle has predictive power—it explains why prior unified models consistently underperformed task-specific ones on understanding benchmarks (the gap is not due to insufficient scale or training, but to representational mismatch), and it implies that future unified systems handling additional modalities (audio, point clouds, tactile) should similarly use per-modality encoders rather than seeking a universal encoding scheme.

Innovation 2: The Shared Transformer as Universal Reasoning Engine with Modality-Specialized Peripherals

The architectural contribution is not the decoupling itself—using separate encoders for different inputs is standard in multimodal understanding models (LLaVA uses a vision encoder plus a text tokenizer; no one argues those should be shared). Rather, the innovation is the architectural division of labor: the visual encoders are frozen feature extractors that never receive gradient updates during Janus training, while all cross-modal learning happens in the shared transformer. This design transforms Janus from a specific model into a platform: the transformer learns to reason over whatever features the encoders provide, and the encoders can be independently upgraded without retraining the entire system.

What distinguishes this from prior work is the combination of three properties: (a) encoders are task-specialized rather than shared, (b) encoders are frozen during all training stages, and (c) the interface between each encoder and the transformer is a small trainable adaptor (a two-layer MLP). Prior multimodal understanding models freeze vision encoders but have no generation pathway and thus no need to manage representational conflict. Prior unified models train or fine-tune their visual encoders alongside the LLM, coupling the encoder choice to the full training pipeline. Janus's design means that swapping the SigLIP encoder for EVA-CLIP or InternViT requires retraining only the understanding adaptor—a trivial cost compared to retraining the full model. Swapping the VQ tokenizer for MoVQGan requires retraining only the generation adaptor. Adding a new modality (e.g., audio) requires training one new encoder and one new adaptor, with the transformer remaining unchanged.

This is a fundamental shift in how to think about unified multimodal architectures: rather than seeking a monolithic model where all components are jointly optimized, Janus demonstrates that a modular design—frozen specialized encoders, small trainable interfaces, shared reasoning core—achieves superior performance while being more extensible. The transformer becomes a universal integration layer that learns to attend across modality-specific feature sequences, regardless of their origin. This reframes the unification problem from "how do we compress all modalities into one representation space?" to "how do we teach a reasoning engine to work with multiple specialized representation spaces simultaneously?"

The practical significance extends beyond the reported benchmarks: this architecture makes it feasible to continuously improve a deployed Janus model by upgrading individual encoders (e.g., when a new state-of-the-art vision encoder is released) without retraining from scratch, and to add new modalities incrementally without disrupting existing capabilities. The frozen-encoder design also has training efficiency implications—the visual encoders' parameters are not stored in optimizer state, reducing memory requirements during the 180,000-step Stage II training.

Innovation 3: The Semantic Tokenizer Ablation as a Controlled Demonstration of the Trade-Off

The paper constructs a particularly incisive ablation that deserves recognition as a methodological contribution. Rather than simply comparing shared vs. decoupled encoders, the authors build a semantic tokenizer—a VQ tokenizer augmented with a semantic decoder branch supervised by a frozen SigLIP teacher (detailed in Appendix A)—that explicitly attempts to capture both the semantic information needed for understanding and the reconstructive information needed for generation within a single encoder. This is a best-effort attempt to make the shared-encoder approach work, using a stronger architecture than the standard VQ tokenizer used in Chameleon or the baseline Exp-A.

The results (Table 5, Exp-B vs. Exp-C) provide a clean controlled experiment: the same encoder architecture, trained on the same data, shows a significant difference in understanding performance depending on whether it must also support generation. The semantic tokenizer achieves POPE 82.4 when trained for both tasks (Exp-B) but POPE 83.9 when trained for understanding only (Exp-C). This gap cannot be attributed to encoder capacity, training data, or optimization—it is purely the cost of the dual-task objective pulling the learned representations toward a compromise that serves neither task optimally.

What makes this an intellectual contribution is that it rules out the most obvious alternative hypothesis: that shared encoders fail only because they are poorly designed for understanding, and a better shared encoder would solve the problem. The semantic tokenizer is a better shared encoder—it substantially improves over the VQ baseline (POPE 60.1 → 82.4)—but it still underperforms the decoupled approach by a meaningful margin, and it still shows an internal trade-off when generation is removed. This suggests the trade-off is fundamental: any single encoder optimizing for both semantic abstraction and reconstruction fidelity will reach a Pareto frontier where further improvement on one axis degrades the other. Decoupling sidesteps this frontier entirely by assigning each task its own encoder, each optimized for its own objective without mutual interference.

This ablation transforms the paper's contribution from an empirical claim ("our model works better") into a diagnostic insight ("shared visual encoding for understanding and generation imposes an inescapable trade-off"). It is the strongest evidence that decoupling is not merely beneficial but necessary for eliminating the performance gap between unified and task-specific models.

Innovation 4: Unified Training Without Task-Specific Compromise

A subtler but important conceptual contribution is the demonstration that a single model trained on both understanding and generation data can match the performance of models trained exclusively on one task. Table 5 shows that the decoupled Janus (Exp-D, unified training) achieves POPE 87.0 and MMB 69.4, which is statistically comparable to the understanding-only variant (Exp-E, POPE 85.9, MMB 70.6). On the generation side, Exp-D achieves COCO FID 8.53, comparable to the generation-only variant (Exp-F, FID 8.92). This means unified training does not degrade either capability—a finding that prior unified models could not demonstrate because their shared-encoder architectures already imposed performance penalties.

This is significant because it challenges an unstated assumption in the multimodal literature: that unifying understanding and generation in a single model necessarily involves a performance trade-off, and the goal is to minimize rather than eliminate the degradation. Models like Chameleon, Show-o, and VILA-U all show understanding performance well below comparably-sized task-specific models—the field implicitly accepted this as the cost of unification. Janus shows that the cost is not inherent to unification per se, but to the specific architectural choice of shared visual encoding. With decoupled encoders, unified training becomes essentially "free" in performance terms—the model gets generation capability without sacrificing understanding, and vice versa.

The training procedure deserves partial credit here. The three-stage schedule (adaptor alignment → unified pretraining → instruction tuning) with frozen encoders throughout means that the LLM is the only component that must serve both tasks, and the LLM's role is fundamentally about sequence modeling and reasoning—capabilities that transfer across modalities. The decoupling ensures that the task-specific information extraction happens before the LLM sees the data, so the LLM receives features already optimized for their intended use. The LLM learns to switch between understanding-mode processing (reasoning over semantic image features to produce text) and generation-mode processing (predicting discrete visual token sequences conditioned on text) based on the sequence structure, without needing to internally disentangle conflicting representational formats.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses a diverse collection of datasets rather than a single unified benchmark. For multimodal understanding pretraining data, Stage II uses interleaved image-text data from WikiHow and WIT; image caption data from Datacomp, ImageNet-1k, OpenImages, and several other sources; table and chart data from DeepSeek-VL; and re-captioned images produced by an open-source multimodal model. For visual generation pretraining, image-caption pairs are drawn from multiple datasets including Datacomp, JourneyDB, PixelProse, and 2 million in-house samples. Supervised fine-tuning in Stage III uses instruction data from LLaVA-OneVision for text, plus VQAv2, GQA, IconQA, ScreenQA, and KVQA for multimodal understanding, and a subset of the generation pretraining data plus 4 million in-house samples for visual generation. Evaluation datasets are separate from training: for understanding, the paper evaluates on VQAv2 (test set), GQA, POPE, MME-Perception, MMBench, SEED-Bench, MM-Vet, and MMMU; for generation, it uses MSCOCO-30K, MJHQ-30K, and GenEval. Each of these is a standard benchmark with publicly available test splits.

• Base model. All experiments use DeepSeek-LLM (1.3B parameters) as the base language model, with a maximum supported sequence length of 4096 tokens. For the understanding encoder, the paper uses SigLIP-Large-Patch16-384. For the generation encoder, it uses the VQ tokenizer from LlamaGen with a codebook size of 16,384 and a downsampling factor of 16. The authors argue in Section 4 that this model scale is representative of what can be trained and deployed practically, and the 1.3B parameter count sits in a regime where test-time compute and architectural choices matter substantially—larger models might mask the representational conflict through sheer capacity, making the decoupling benefit harder to isolate.

• Metrics. For multimodal understanding, the paper reports standard accuracy or accuracy-like metrics on each benchmark: POPE uses F1-score (binary classification of object presence), MME-Perception reports a composite score aggregating multiple perception subtasks, MMBench and SEED-Bench report accuracy (percentage of correctly answered multiple-choice questions), VQAv2 reports test-set accuracy on open-ended visual questions, GQA reports accuracy on compositional visual reasoning questions, MM-Vet reports a composite score evaluating integrated vision-language capabilities, and MMMU reports accuracy on multi-discipline multimodal understanding questions spanning college-level subjects. For visual generation, the paper uses Fréchet Inception Distance (FID) on MSCOCO-30K and MJHQ-30K—FID measures the distributional distance between generated and real images in Inception feature space, with lower values indicating higher quality and diversity. On GenEval, the paper reports accuracy across six compositional subtasks (Single Object, Two Objects, Counting, Colors, Position, Color Attribution) plus an overall average—this measures whether the model correctly follows specific compositional instructions in text-to-image generation, not just overall image quality.

• Baselines. The paper compares against three families of models. For understanding-only models, baselines include LLaVA-v1.5-Phi-1.5 (1.3B), MobileVLM (1.4B and 2.7B variants), MobileVLM-V2 (1.4B and 2.7B), LLaVA-Phi (2.7B), LLaVA-v1.5 (7B), InstructBLIP (7B and 13B), Qwen-VL-Chat (7B), and IDEFICS-9B. For generation-only models, baselines include LlamaGen (0.8B), LDM (1.4B), SDv1.5 (0.9B), SDv2.1 (0.9B), PixArt-α (0.6B), DALL-E 2 (6.5B), SDXL (2.6B), DALL-E (12B), GLIDE (5B), GigaGAN (0.9B), Imagen (34B), and RAPHAEL (3B). For unified understanding and generation models, baselines include DreamLLM (7B, using external diffusion), LaVIT (7B, using external diffusion), Emu (13B, using external diffusion), NExT-GPT (13B, using external diffusion), Show-o (1.3B), Gemini-Nano-1 (1.8B), LWM (7B), VILA-U (7B), Chameleon (7B and 34B), and Emu3-Chat (8B). The paper is careful to mark which unified models use external pretrained diffusion models with a † symbol in its tables (Tables 2, 3, 4), distinguishing true unified models from tool-use approaches.

• Generation budget / compute accounting. The paper measures compute primarily through model size (parameter count) and reports performance at a fixed model scale (1.3B) rather than sweeping compute budgets. For image generation inference, the paper notes that classifier-free guidance with scale s = 5 requires two forward passes per token (one conditional, one unconditional), effectively doubling generation inference cost compared to standard autoregressive sampling, but this cost is not formally accounted for in any FLOPs comparison. Training compute is reported in qualitative terms (7 days on 16 nodes of 8× A100 40GB, totaling 128 GPUs) rather than in FLOPs or GPU-hours. There is no formal FLOPs-matched comparison between Janus and larger models—the comparison is purely on final task performance at given parameter scales.

• Cross-validation / statistical protocol. The paper does not report cross-validation or statistical significance testing. Results in Tables 2, 3, and 4 are point estimates on standard benchmark test sets without confidence intervals or error bars. The ablation studies in Table 5 similarly report single-number results. The paper's claims about outperforming baselines are based on absolute performance differences without formal statistical testing—for example, Janus achieving 87.0 on POPE vs. LLaVA-v1.5's 85.9 (Table 2) is a 1.1-point absolute difference whose statistical reliability cannot be assessed from the reported data.

Main Quantitative Results

Multimodal Understanding Performance (Table 2)

The headline result for understanding is that Janus (1.3B) achieves the best overall results among models of similar scale and is competitive with much larger task-specific models. The specific numbers: POPE 87.0, MME-Perception 1338.0, MMBench 69.4, SEED-Bench 63.7, VQAv2 (test) 77.3, GQA 59.1, MMMU 30.5, MM-Vet 34.3 (Table 2).

Compared to the previous best unified model at similar scale, Show-o (1.3B), Janus shows substantial improvements: MME-Perception improves from 948.4 to 1338.0—a 41% relative increase—and GQA improves from 48.7 to 59.1—a 21% relative increase. The paper attributes this to decoupling eliminating the representational conflict, but an alternative interpretation is simply that SigLIP is a stronger encoder than whatever Show-o uses for understanding; the paper does not provide an ablation isolating the encoder quality effect from the decoupling effect.

Compared to larger task-specific models, Janus holds up remarkably well. On POPE, Janus achieves 87.0 vs. LLaVA-v1.5 (7B) at 85.9—a 1.1-point advantage—and vs. InstructBLIP (13B) at 78.9—an 8.1-point advantage. On MMBench, Janus scores 69.4 vs. LLaVA-v1.5 (7B) at 64.3—a 5.1-point advantage. On SEED-Bench, 63.7 vs. LLaVA-v1.5 (7B) at 58.6—a 5.1-point advantage. On MM-Vet, 34.3 vs. LLaVA-v1.5 (7B) at 31.1—a 3.2-point advantage. These are striking results: a 1.3B model outperforming a well-regarded 7B model on multiple benchmarks. However, on VQAv2 (77.3 vs. 78.5), GQA (59.1 vs. 62.0), and MMMU (30.5 vs. 35.4), LLaVA-v1.5 (7B) maintains an edge, suggesting Janus's advantage is not uniform across all understanding tasks.

The paper emphasizes these results as evidence that decoupling eliminates the understanding penalty incurred by unified models, but a more nuanced reading is warranted: Janus's understanding performance is strong for its size, but it is not uniformly above all larger models, and the comparison to Show-o—the most direct baseline—confounds encoder quality with the decoupling effect.

Visual Generation Performance (Tables 3 and 4)

On GenEval (Table 3), Janus achieves an overall accuracy of 61%, which the paper highlights as surpassing "the previous best unified model Show-o (53%) and some popular generation-only methods, e.g., SDXL (55%) and DALL-E 2 (52%)." The breakdown by compositional subtask reveals a distinctive strength profile. Janus excels at Position (0.46 vs. SDXL's 0.15 and Show-o's 0.11) and Color Attribution (0.42 vs. SDXL's 0.23 and Show-o's 0.28)—these are tasks measuring precise spatial and attribute binding. It also performs strongly on Two Objects (0.68 vs. SDXL's 0.74 and DALL-E 2's 0.66) and Single Object (0.97, matching SDXL and DALL-E 2). Its weakest subtask is Counting (0.30 vs. Show-o's 0.49 and SDXL's 0.39), where it underperforms several baselines. This profile suggests Janus's generation strength lies in compositional precision—correctly associating attributes with objects and placing them in specified spatial relationships—rather than in counting or numeracy.

On MSCOCO-30K (Table 4), Janus achieves an FID of 8.53, which the paper positions as surpassing unified models Show-o (9.24) and LWM (12.68), and being competitive with generation-only models like SDv1.5 (9.62) and PixArt-α (7.32), though not matching the best generation-only models like RAPHAEL (6.61) or Imagen (7.27). On MJHQ-30K, Janus achieves an FID of 10.10, compared to Show-o's 15.18, LWM's 17.77, and VILA-U's 7.69 (at 384 resolution) and 12.81 (at 256 resolution). The MJHQ-30K result is particularly notable because it evaluates aesthetic quality on a curated set of high-quality images—Janus's 10.10 FID suggests reasonable aesthetic quality for a 1.3B model, though not at the level of larger generation-only models.

An important caveat for these generation comparisons: image resolution varies across models. Janus generates at 384×384, while SDXL generates at 1024×1024 (Figure 4 caption). The FID metric is computed on images resized to a common resolution (typically 299×299 for Inception features), so the resolution difference is partially normalized, but higher native resolution typically provides more detail that can survive downsampling. The paper does not discuss this as a confounding factor.

Ablation Studies: Decoupling and Unified Training (Table 5)

The ablation in Table 5 is the paper's central empirical contribution, so it deserves careful treatment. The table reports five experiments plus a generation-only baseline, each measuring POPE, MMBench, SEED-Bench, MMMU, and COCO-FID.

Exp-A (VQ Tokenizer, Unified Training): This is the baseline shared-encoder approach analogous to Chameleon. Results: POPE 60.1, MMB 35.0, SEED 34.9, MMMU 24.7, COCO-FID 8.72. The understanding numbers are dramatically lower than the decoupled model (Exp-D: POPE 87.0, MMB 69.4, SEED 63.7, MMMU 30.5), confirming that a standard VQ tokenizer is severely suboptimal for understanding. The generation FID of 8.72 is acceptable—slightly better than the decoupled model's 8.53—which is consistent with VQ tokenizers being designed for generation.

Exp-B (Semantic Tokenizer, Unified Training): This is the stronger baseline using a VQ tokenizer distilled from SigLIP with an additional semantic decoder. Results: POPE 82.4 (+22.3 over Exp-A), MMB 52.7 (+17.7), SEED 54.9 (+20.0), MMMU 26.6 (+1.9), COCO-FID 7.11 (+1.61 better than Exp-A). The semantic tokenizer substantially closes the understanding gap but still falls well short of the decoupled Exp-D on all understanding metrics: POPE lags by 4.6 points, MMB by 16.7 points, SEED by 8.8 points, and MMMU by 3.9 points. The generation FID of 7.11 is notably better than Exp-D's 8.53, which the paper attributes to two factors: the semantic tokenizer producing "more semantically coherent" discrete IDs (providing better prediction targets), and the semantic tokenizer having significantly more parameters than the VQ tokenizer used in Exp-D.

Exp-C (Semantic Tokenizer, Understanding Only): This is the critical controlled experiment. Using the same semantic tokenizer but training only on understanding tasks (no generation data), performance improves to: POPE 83.9 (+1.5 over Exp-B), MMB 62.1 (+9.4), SEED 60.8 (+5.9), MMMU 27.5 (+0.9). The gap between Exp-B and Exp-C is the trade-off cost: the semantic tokenizer's representations degrade for understanding when it must also support generation. The degradation is most severe on MMBench (9.4 points) and SEED-Bench (5.9 points)—benchmarks that require deeper semantic reasoning—and milder on POPE (1.5 points, a task focused on object presence). This pattern is consistent with the paper's granularity thesis: generation forces the encoder to preserve low-level details that interfere with high-level semantic reasoning.

Exp-D (SigLIP + VQ, Decoupled, Unified Training): This is Janus. Results: POPE 87.0, MMB 69.4, SEED 63.7, MMMU 30.5, COCO-FID 8.53.

Exp-E (SigLIP, Understanding Only): Training the understanding pathway alone without generation data yields: POPE 85.9, MMB 70.6, SEED 64.8, MMMU 28.8. Comparing Exp-D to Exp-E, the differences are small: POPE is 1.1 points higher in Exp-D, MMB is 1.2 points lower, SEED is 1.1 points lower, and MMMU is 1.7 points higher. These are within what could reasonably be attributed to training noise given the single-run reporting. The paper interprets this as evidence that unified training does not degrade understanding—a claim that is supported for these four benchmarks, though the absence of statistical testing means we cannot rule out small systematic effects.

Exp-F (VQ Tokenizer, Generation Only): Training the VQ tokenizer for generation alone yields COCO-FID 8.92, compared to Exp-D's 8.53. The decoupled model actually achieves slightly better generation FID than the generation-only baseline—a modest 0.39 improvement. The paper does not explain this, but possibilities include: the unified training providing additional visual knowledge that transfers to generation, the diverse training data in Exp-D improving general visual understanding that benefits generation, or simply training noise.

The key findings from this ablation matrix are: (1) a shared VQ encoder catastrophically fails at understanding (Exp-A vs. Exp-D), (2) a stronger shared encoder partially recovers but still imposes a measurable trade-off (Exp-B vs. Exp-C), (3) decoupling eliminates this trade-off entirely (Exp-D matches Exp-E on understanding and Exp-F on generation). These three points together form the evidential backbone of the paper's core claim.

Qualitative Results (Figures 4, 5, 7, 8, 9)

Figure 4 provides side-by-side visual comparisons of Janus against SDXL and LlamaGen on four text prompts. The paper claims Janus demonstrates "superior instruction-following capabilities" and "accurately captures most details." The examples are visually compelling—Janus's outputs appear faithful to complex prompts involving spatial relationships ("Sydney Opera House sitting next to Eiffel tower"), detailed descriptions ("aged, with deep-set wrinkles... youthful, exuding vigor"), and fantastical scenes—but qualitative cherry-picking is inherent to such figures. The paper does not report any systematic human evaluation or automated metric comparing instruction-following fidelity beyond the GenEval numbers.

Figure 5 shows qualitative multimodal understanding on humorous memes, comparing Janus against Chameleon and Show-o. Janus correctly identifies the text in the meme and interprets its emotional tone, while Chameleon hallucinates nonexistent text ("I'm not arguing, I'm just explaining why I'm right") and Show-o misidentifies objects and colors. These examples illustrate the failure modes of shared-encoder models: Chameleon's VQ tokenizer loses the fine-grained text information in the image, and Show-o makes basic perceptual errors (dog color). The paper attributes these to the decoupled encoder capturing finer detail, but without systematic evaluation of text reading accuracy or hallucination rates, these remain illustrative anecdotes.

Figure 7 shows additional text-to-image generations across diverse prompts; Figure 8 demonstrates emergent multilingual generation (Chinese, French, Japanese) despite training only on English text-to-image data; Figure 9 shows understanding across varied contexts (scientific charts, artwork, LaTeX formulas, code generation from plots). These figures demonstrate breadth but are inherently non-systematic.

Ablation Studies and Robustness Checks

Shared vs. decoupled encoder architectures (Table 5): This is the central ablation discussed extensively above. The finding is that a single VQ tokenizer (Exp-A) achieves tolerable generation (FID 8.72) but collapses on understanding (POPE 60.1, MMB 35.0, SEED 34.9); a stronger semantic tokenizer (Exp-B) partially recovers understanding but still shows a trade-off when compared to its understanding-only variant (Exp-C); the decoupled model (Exp-D) matches both task-specific baselines (Exp-E and Exp-F). The non-obvious finding is that even a sophisticated attempt to build a dual-purpose encoder (the semantic tokenizer with both reconstruction and semantic losses) cannot eliminate the trade-off—the representations must compromise.

Unified vs. task-specific training (Table 5, comparing Exp-D to Exp-E and Exp-F): The finding is that unified training with decoupled encoders imposes negligible performance cost on either task. On understanding, Exp-D is within ±2 points of Exp-E on all four metrics; on generation, Exp-D slightly outperforms Exp-F (FID 8.53 vs. 8.92). This demonstrates that the transformer can learn to serve both tasks without mutual interference, provided the input representations are appropriately specialized.

Effect of caption truncation during generation training: The paper mentions (Section 4.2, Stage II) that during text-to-image training, only the first sentence of a caption is used with 25% probability. No ablation measuring the effect of this design choice is reported—we do not know whether generation quality or brevity-handling would differ without it.

Effect of ImageNet curriculum in Stage II: The paper uses ImageNet-1k only during the first 120K of 180K training steps, with other datasets appearing later. No ablation comparing this curriculum against uniform mixing is reported. The stated motivation comes from PixArt-α, but whether it actually matters for Janus is untested.

Semantic tokenizer design (Appendix A): The semantic tokenizer used in the ablation includes a 12-layer ViT semantic decoder with 12 attention heads and hidden dimension 768, supervised by a frozen SigLIP teacher with cosine similarity loss (weight 0.25) alongside standard RGB reconstruction loss. This is a non-trivial architecture choice—the 12-layer ViT decoder adds substantial capacity beyond the base VQ tokenizer—and different architectural choices for the semantic decoder might yield different trade-off characteristics. The paper does not ablate the semantic decoder depth, the loss weight, or the choice of teacher model.

Classifier-free guidance scale (Section 3.4): The default guidance scale s = 5 is used for all generation evaluations. No sweep over guidance scales is reported—we do not know whether Janus's generation quality is sensitive to this hyperparameter, or whether the optimal scale differs from what prior work found for generation-only autoregressive models.

Understanding encoder choice: The paper uses SigLIP-Large-Patch16-384. No ablation comparing alternative understanding encoders (CLIP, EVA-CLIP, InternViT) is reported, despite Section 3.5 explicitly listing these as compatible alternatives. The paper's claim that the decoupled architecture allows encoder upgrades is theoretical—no upgrading experiment is performed.

Negative result: ReST^EM-style training not attempted for Janus. Unlike the reference paper discussed in prior sections, Janus does not experiment with reinforcement learning or iterative self-improvement for the generation pathway. This is not a weakness per se—the paper's scope is architectural—but it means we do not know whether the VQ-based generation pathway would benefit from techniques like those used in the reference paper's revision model.

Critical Assessment

Claim from the executive summary: "Janus surpasses prior unified models and matches or exceeds task-specific models."

This claim is supported with qualifications. For multimodal understanding (Table 2), Janus does surpass all unified models at comparable scale (Show-o at 1.3B, Gemini-Nano-1 at 1.8B) by substantial margins, and it outperforms several task-specific models with larger parameter counts (LLaVA-v1.5 7B on POPE, MMBench, SEED-Bench, MM-Vet). However, the comparison to LLaVA-v1.5 is not a clean model-scale comparison because LLaVA-v1.5 uses a different base LLM (Vicuna-7B vs. DeepSeek-LLM 1.3B) and a different vision encoder (CLIP-ViT-L vs. SigLIP-L). The performance difference could stem from the LLM, the vision encoder, the training data mixture, or the training procedure—the paper's ablation establishes that decoupling matters, but it does not isolate how much of the advantage over LLaVA comes from architecture vs. other factors.

For visual generation (Tables 3 and 4), Janus surpasses the previous best unified model (Show-o) on GenEval (61% vs. 53%) and on COCO-FID (8.53 vs. 9.24) and MJHQ-FID (10.10 vs. 15.18). It also matches or exceeds several generation-only models (SDXL on GenEval overall, SDv1.5 and DALL-E 2 on COCO-FID). However, Janus operates at 384×384 resolution while SDXL generates at 1024×1024—the comparison conflates model quality with resolution. On GenEval, the resolution difference may matter less because compositional accuracy is measured at instance level, but on FID, higher native resolution typically improves the metric even after resizing. The paper does not address this.

Claim: "Decoupling eliminates the conflict between understanding and generation."

This claim is well-supported by the ablation evidence in Table 5, but only within the specific encoder choices tested. The experiment shows that SigLIP produces representations that work well for understanding, that the VQ tokenizer produces representations that work well for generation, and that training them jointly in a unified model does not degrade either. However, the claim would be more precisely stated as "for these specific encoders and this specific model scale, decoupling eliminates the trade-off." The paper does not test whether decoupling remains beneficial at larger model scales—it is possible that larger LLMs could learn to disentangle conflicting representations internally, making decoupling less important. The paper also does not test whether alternative encoder pairs might produce different results—for instance, a stronger joint encoder (using a more sophisticated architecture than the semantic tokenizer) might narrow or close the trade-off gap.

Claim: "Janus is flexible and extensible."

This is primarily a design claim rather than an empirically validated one. The architecture certainly supports encoder swapping and modality addition in principle, but the paper provides zero experiments demonstrating this: no alternative encoder is tested, no additional modality is added, no dynamic high-resolution technique is applied, no token compression is evaluated. The claim is best understood as stating a design philosophy whose practical benefits are yet to be demonstrated.

Genuine weaknesses in the experimental design:

• No statistical testing or confidence intervals. All results are point estimates on single evaluation runs. With test sets of 500–30,000 examples, some of the reported differences (e.g., POPE 87.0 vs. 85.9) may not be statistically significant. The paper cannot distinguish real performance differences from evaluation noise.

• Single model family, single scale. All experiments use DeepSeek-LLM 1.3B as the base. We do not know whether the benefits of decoupling generalize to other LLM families (Llama, Qwen, Gemma) or to other scales (300M, 7B, 13B). At 7B+, the representational conflict might be less severe because a larger transformer has more capacity to internally disentangle mixed representations. The paper's claims about decoupling being a general design principle are supported only at one scale.

• Understanding vs. generation trade-off is only shown for one encoder pair. The paper demonstrates the trade-off for a VQ tokenizer and a semantic tokenizer, then shows decoupling with SigLIP + VQ eliminates it. Alternative encoder combinations are not explored. Could a different generation encoder (e.g., a diffusion-based autoencoder rather than VQ) produce representations that are less harmful to understanding when shared? Could a different understanding encoder produce features that also work well for generation? The paper's claim that the trade-off is "inherent" requires testing more than two shared-encoder architectures.

• No resolution ablation for generation. Janus generates at 384×384. The paper does not report how generation quality scales with resolution, or whether the decoupled architecture imposes any resolution constraints compared to generation-only models. This matters because practical text-to-image applications typically require higher resolutions.

• Missing baseline: Janus with a single SigLIP encoder adapted for generation. The paper shows that VQ tokenizers are bad for understanding (Exp-A), but does not test whether SigLIP—adapted with a generation-specific output head—could serve both tasks. This would test whether the granularity problem is symmetric: is a semantically-rich encoder better at generation than a reconstruction-focused encoder is at understanding? The semantic tokenizer (Exp-B) partially addresses this but is a VQ-based architecture; a direct SigLIP-to-generation pathway is not tested.

• No compute-matched comparison. The paper compares Janus at 1.3B total parameters against models with up to 34B parameters, but there is no FLOPs-matched training comparison. We do not know whether a 7B model with a shared encoder, trained with the same total FLOPs as Janus, would outperform or underperform. This is particularly important because the paper argues that decoupling is necessary—but if a larger shared-encoder model achieves the same performance at similar total compute, the architectural argument weakens.

• Training data not fully specified. The paper lists data sources by name (Datacomp, JourneyDB, PixelProse, WikiHow, WIT, etc.) but does not report the total number of training examples in each stage, the dataset sizes, or the data distribution across sources. The 2M and 4M in-house data samples are completely opaque. This makes it impossible to assess whether Janus's performance advantage over baselines comes from architecture or from data scale/quality.

Experiments that would have strengthened the paper:

1. Scale sweep: Train Janus at 300M, 1.3B, and 7B scales with both decoupled and shared (Exp-A) encoders, showing whether the decoupling benefit grows, shrinks, or stays constant with scale.

2. Encoder swap experiments: Replace SigLIP with CLIP-ViT-L or EVA-CLIP in the understanding pathway, and measure whether the adaptor-only retraining is sufficient to maintain performance—this would directly validate the flexibility claim.

3. Resolution scaling for generation: Train and evaluate Janus at 256, 384, and 512 resolution for generation, reporting the FID-resolution trade-off curve.

4. Compute-matched comparison: Train a 7B shared-encoder model (like Chameleon) with the same total GPU-hours as Janus-1.3B and compare on both understanding and generation benchmarks. This would test whether the decoupling advantage is architectural or simply an efficient use of a given compute budget.

5. Systematic evaluation of fine-grained understanding: Given the paper's claim that decoupling improves fine-grained perception, a targeted evaluation on tasks like text reading (OCR), object counting, and spatial relationship judgment would provide more direct evidence than the meme examples in Figure 5.

6. Ablation of the understanding adaptor architecture: The adaptors are two-layer MLPs. Testing whether deeper adaptors, attention-based adaptors, or linear projections affect performance would clarify whether the adaptor design matters or whether the encoder choice dominates.

The paper's experimental evidence is strongest for its central causal claim: shared visual encoding imposes a trade-off between understanding and generation, and decoupling eliminates it (Table 5). The evidence is more suggestive for the broader claims about matching task-specific models—the comparisons are not controlled for model family, training data, or compute budget, so performance differences are confounded. The flexibility and extensibility claims remain entirely aspirational, untested by any experiment in the paper.

6. Limitations and Trade-offs

6.1 Resolution Constraints for Visual Generation

The assumption or constraint. Janus generates images at a fixed resolution of 384×384 pixels, with all images resized to this dimension during both training and inference. For visual generation training data, "the short side is resized to 384, and the long side is cropped to 384" (Section 4.1). This is substantially lower than the output resolution of leading generation-only models: SDXL generates at 1024×1024, and LlamaGen at 512×512 (Figure 4 caption). The paper does not discuss any mechanism for native higher-resolution generation, nor does it test whether the architecture can scale to higher resolutions.

The consequence. For practical text-to-image deployment, 384×384 is likely insufficient. Real-world applications (creative tools, design assistance, content creation) typically require at least 1024×1024 for usable outputs. The FID and GenEval metrics reported in Tables 3 and 4 are computed on images resized to standard evaluation resolutions (e.g., 299×299 for Inception features in FID), which masks resolution differences—higher native resolution typically provides more detail that can survive downsampling, so comparing Janus's 384×384 FID of 8.53 against SDXL's 1024×1024 FID of 9.62 (Table 4) potentially understates SDXL's practical advantage. Additionally, the cropping strategy for generation data (short side resized, long side cropped) means the model never learns to handle non-square aspect ratios for generation—it can only produce square images, which is another practical constraint not discussed.

What evidence exists in the paper. The resolution is specified in Section 4.1, and Figure 4's caption explicitly states the resolution difference ("The image resolutions for SDXL, LlamaGen, and ours are 1024 × 1024, 512 × 512, and 384 × 384, respectively."). No resolution scaling experiment is reported—we do not know whether generation quality degrades or improves at different resolutions, whether the VQ tokenizer's downsampling factor of 16 imposes a fundamental constraint, or whether the 4096-token maximum sequence length limits how many image tokens can be processed.

Mitigation status. Not addressed. The paper's discussion of future improvements for visual generation (Section 3.5) mentions "finer-grained encoders" and "a combination of AR and parallel methods" but does not mention resolution scaling. The architecture does not appear to have any inherent mechanism for dynamic resolution generation—unlike diffusion models, which naturally support variable resolution, the VQ tokenizer processes fixed-size grids. Higher-resolution generation would require either a larger token grid (increasing sequence length quadratically and potentially exceeding the LLM's 4096-token limit) or architectural modifications to support multi-scale generation, neither of which is discussed.

---

6.2 Single Model Scale and Single LLM Family

The assumption or constraint. All experiments use DeepSeek-LLM at 1.3B parameters as the base language model. The paper argues this model is "representative" (Section 4, though the quoted framing about representativeness comes from the executive summary context rather than the paper itself—the paper simply selects this model as its base without providing explicit justification for generalizability). No experiments are conducted with larger LLM variants (e.g., 7B, 13B) or with different LLM families (Llama, Qwen, Gemma, Mistral).

The consequence. The central finding—that decoupling visual encoding eliminates the understanding-generation trade-off—is validated only at one scale with one model architecture. There are at least two reasons this matters for practitioners. First, scale dependence: a larger LLM has greater representational capacity and might be able to internally disentangle the conflicting signals from a shared encoder, making decoupling less necessary. If this is the case, the architectural insight is scale-specific rather than universal, and practitioners training larger models might reasonably choose simpler shared-encoder architectures. Second, LLM family dependence: the specific interaction between SigLIP features, VQ tokenizer embeddings, and the LLM's internal representations depends on the LLM's pretraining—a different LLM might require different adaptor architectures, different learning rates, or might not benefit from decoupling to the same degree. The paper provides no evidence that the approach transfers.

What evidence exists in the paper. None. The paper contains no scale sweep (e.g., 300M, 1.3B, 7B Janus variants), no experiment with a different base LLM, and does not even mention this as a limitation or avenue for future work. All Tables (2, 3, 4, 5) and all Figures report results from a single model configuration.

Mitigation status. Not addressed. The paper describes Janus as though the design principle is established, but without scale or model diversity experiments, the generalization claim remains untested. This is particularly significant because the comparison baselines in Table 2 use different LLM families (Vicuna for LLaVA-v1.5, Phi for LLaVA-Phi, the Qwen LLM for Qwen-VL-Chat), so the performance advantage over task-specific models could partially stem from DeepSeek-LLM being a stronger base model at 1.3B rather than from the decoupled architecture per se.

---

6.3 Flexibility and Extensibility Claims Are Untested

The assumption or constraint. Section 3.5 lays out an ambitious vision for Janus as a platform: the understanding encoder can be upgraded to EVA-CLIP or InternViT, dynamic high-resolution techniques can be applied, token compression can be used, the generation encoder can be switched to MoVQGan, diffusion losses can be employed, and new modalities (point clouds, tactile signals, EEG) can be added—all without architectural changes. The paper's framing suggests these are practical near-term possibilities enabled by the decoupled design.

The consequence. None of these extensions is demonstrated. A practitioner considering Janus cannot know whether swapping the SigLIP encoder for EVA-CLIP would require retuning the understanding adaptor hyperparameters, or whether the adaptor architecture (two-layer MLP) is sufficient for a different encoder's feature space, or whether adding a third modality would cause training instability or cross-modal interference. The claim that "the straightforward architecture of Janus allows for easy integration with additional encoders" (Section 3.5) is a design assertion, not an empirical finding. This matters because the value proposition of Janus over task-specific models includes this flexibility—if upgrading the understanding encoder requires substantial re-engineering or produces degraded performance, the practical advantage diminishes.

What evidence exists in the paper. None. There is no experiment demonstrating any of the listed extensions. The frozen-encoder design does make encoder swapping mechanically possible (only the adaptor needs retraining), but whether the resulting performance is competitive, stable, or requires careful hyperparameter tuning is unknown. The paper does not even test a different SigLIP variant or a different VQ tokenizer as a minimal validation of the flexibility claim.

Mitigation status. Not addressed. The paper presents these extensions as "Possible Extensions" (Section 3.5 title) without performing any. The section is aspirational rather than evidential. Adding even one experiment—for example, comparing Janus with SigLIP vs. Janus with CLIP-ViT-L on understanding benchmarks, retraining only the adaptor—would have transformed this from a design claim into an empirical demonstration.

---

6.4 Computational Overhead of Dual Encoders and Classifier-Free Guidance

The assumption or constraint. Janus uses two separate visual encoders (SigLIP and VQ tokenizer) with separate adaptors, and during image generation inference, it performs two forward passes per token for classifier-free guidance with scale s = 5 (Section 3.4). The paper does not report the parameter counts of these encoders, their inference FLOPs, or the total compute cost relative to a shared-encoder unified model or a task-specific model of comparable performance.

The consequence. The headline results in Tables 2–4 report performance at a given parameter count (1.3B for the LLM), but this excludes the visual encoder parameters from the model size comparison. The SigLIP-Large-Patch16-384 encoder has approximately 430M additional parameters; the VQ tokenizer from LlamaGen adds further parameters (encoder, decoder, codebook—the paper does not specify the count). The total parameter count for Janus is therefore substantially higher than 1.3B when all components are included. This matters for two practical reasons: (1) the comparison against models like LLaVA-v1.5 (7B) in Table 2 is not a clean "smaller model beats larger model" story—Janus's effective parameter count is larger than 1.3B, and (2) during inference, both encoders are used (SigLIP for understanding, VQ tokenizer encoder for generation input, VQ decoder after prediction), and CFG doubles the autoregressive generation cost, so the per-example inference compute differs by task and may be substantially higher than a pure understanding or pure generation model.

Additionally, the training cost includes forward passes through both frozen encoders for all multimodal data, plus storing their activations for the backward pass through the adaptors—this is cheaper than training the encoders but not free, and the paper does not quantify the computational overhead relative to a single-encoder unified model.

What evidence exists in the paper. None for parameter counts of visual encoders. None for inference FLOPs comparison. The paper reports that training took "7 days on a cluster of 16 nodes, each equipped with 8 Nvidia A100 (40GB) GPUs" (Section 4.1), which provides a qualitative sense of total cost, but without a baseline (how long would a shared-encoder variant take? how long would a 7B task-specific model take?), this number is not actionable. The CFG guidance scale s = 5 is mentioned (Section 3.4), and its effect on inference cost (2× forward passes per generated token) is mathematically implied but not explicitly quantified or compared.

Mitigation status. Not addressed. The paper does not acknowledge the parameter count discrepancy or the inference compute overhead. The CFG cost is inherent to the generation approach (and shared with other autoregressive image generation models like LlamaGen, which also use CFG), but the dual-encoder overhead is specific to Janus's architecture. A simple accounting of total parameters (LLM + SigLIP + VQ encoder + VQ decoder + adaptors + image head) would help practitioners assess whether the performance benefits justify the additional parameters, but this is not provided.

---

6.5 Incomplete Training Data Specification

The assumption or constraint. The paper describes training data in Section 4.2 by listing data source names and categories, but does not provide quantitative details essential for reproducibility and fair comparison against baselines. The total number of training examples in each stage is not reported. The 2 million in-house visual generation samples in Stage II and 4 million in-house samples in Stage III are completely opaque—their content, quality, and domain are unspecified. The filtering procedure for visual generation data ("we filter based on aesthetic scores and image sizes, resulting in 20% remaining" for certain sources) is mentioned without specifying the aesthetic score threshold, the image size threshold, or the scoring model used.

The consequence. Assessing whether Janus's performance advantage over baselines stems from architecture or from data volume/quality is impossible. If Janus's pretraining data mixture is substantially larger or higher-quality than what Chameleon, Show-o, or VILA-U used, the architectural claims are confounded. The in-house data's role is particularly opaque: it comprises the majority of generation training samples (2M out of the listed datasets plus in-house, and 4M out of the fine-tuning data), so its quality directly affects the generation results reported in Tables 3 and 4. A practitioner attempting to reproduce Janus's results or adapt the approach to their own domain cannot replicate these data sources and cannot estimate how much performance degradation to expect when using only publicly available data.

The data ratios in Table 1 (2:3:5 for Stage II, 7:3:10 for Stage III) specify the mixing proportions but not the absolute volumes—without the total dataset size, these ratios cannot be converted into concrete training step counts per data type.

What evidence exists in the paper. The paper lists data sources by name in Section 4.2 but provides no dataset sizes, no download links for composite datasets, no documentation of the re-captioning process, and no characterization of the in-house data. The statement "we filter based on aesthetic scores and image sizes, resulting in 20% remaining" (Section 4.2, Stage II) is typical of the level of detail provided—qualitative but not quantitative.

Mitigation status. Not addressed. The paper does not flag data opacity as a limitation, does not provide a table of dataset sizes, and does not release the in-house data or the filtered/augmented versions of public datasets. This is a common limitation in industrial multimodal model papers but is particularly acute here because the paper's central claim is an architectural one—without controlling for data, the architecture-data confound is unresolvable.

---

6.6 GenEval Weakness on Counting and No Systematic Evaluation of Fine-Grained Understanding

The assumption or constraint. Janus's visual generation strength on GenEval (Table 3) is concentrated in spatial and attribute-binding subtasks (Position 0.46, Color Attribution 0.42)—tasks where the model must place objects in specified relationships or correctly bind visual attributes to objects. However, its Counting score (0.30) is notably weak, falling below Show-o (0.49), SDv2.1 (0.44), SDXL (0.39), and DALL-E 2 (0.49). Similarly, while the paper claims decoupling enhances "fine-grained multimodal understanding ability" (Section 4.6) and provides qualitative examples of accurate text reading in memes (Figure 5), there is no systematic evaluation of fine-grained perceptual capabilities—object counting, OCR accuracy, spatial relationship precision, or attribute binding in understanding tasks.

The consequence. The weakness on Counting suggests that Janus's autoregressive approach to generation, combined with the VQ tokenizer's discrete representations, struggles specifically with numeracy—the model cannot reliably produce images with exact object counts. This is a known challenge for autoregressive image generation (each token prediction depends only on previous tokens, making it hard to maintain global constraints like exact counts), and the paper's results confirm that Janus does not resolve it. For applications requiring precise compositional control (e.g., "three apples on a table"), this is a practical failure mode. On the understanding side, without systematic fine-grained evaluation, the qualitative examples in Figure 5 are merely illustrative—the paper claims decoupling improves fine-grained perception but provides no benchmark evidence measuring this.

What evidence exists in the paper. The Counting score of 0.30 in Table 3 is a clear quantitative indicator. For understanding, Figure 5 provides anecdotal examples (correctly reading text in a meme), but no benchmark like TextVQA (text reading), TallyQA (counting), or spatial reasoning probes is evaluated. The standard multimodal understanding benchmarks used (POPE, MMBench, SEED-Bench, VQAv2) test a mix of general visual reasoning and recognition, not specifically fine-grained perceptual accuracy.

Mitigation status. Not addressed for generation—the paper does not discuss the low Counting score or propose mechanisms to improve numeracy. For understanding, the paper does not acknowledge the absence of fine-grained evaluation; it treats the qualitative examples as sufficient evidence for the claim that "the decoupled vision encoder significantly enhances Janus's fine-grained multimodal understanding ability compared to the shared encoder used by Chameleon and Show-o" (Section 4.6). This claim extrapolates from two meme examples to a general capability without systematic measurement.

7. Implications and Future Directions

How This Work Changes the Landscape

Janus introduces a diagnostic reframing rather than a paradigm shift: it identifies a specific architectural failure mode in unified multimodal models—the representational granularity conflict between understanding and generation—and provides a clean, mechanically simple solution (decoupled encoders) that eliminates it. The magnitude of the contribution lies not in proposing a fundamentally new learning algorithm or training objective, but in pinpointing why prior unified models consistently underperformed task-specific ones and demonstrating that the fix is architectural rather than a matter of scale or data.

The paper's most significant conceptual move is converting an implicit design assumption into an explicit, testable hypothesis. Prior work (Chameleon, Show-o, VILA-U, Transfusion) treated "one visual encoder for both tasks" as natural or even necessary—an architectural default rather than a decision. Janus asks: what if this default is wrong? The ablation in Table 5 provides the cleanest answer to date: a shared VQ tokenizer collapses understanding performance (POPE 60.1 vs. 87.0, MMB 35.0 vs. 69.4), and even a carefully designed dual-purpose semantic tokenizer (Exp-B) leaves 5–10 points on the table compared to task-specialized encoders, with the gap traceable directly to the generation objective pulling representations away from semantic abstraction (Exp-B vs. Exp-C). This transforms the debate from "how can we make shared encoders work better?" to "should we even try to make shared encoders work for understanding and generation?"

The paper also reconciles a latent tension in the multimodal literature. On one side, the unified model community (Chameleon, Show-o) argued that a single encoder could serve both tasks with sufficient scale. On the other side, task-specific models consistently showed that specialized encoders (CLIP/SigLIP for understanding, VQ/VAE for generation) outperform shared ones. Janus synthesizes these positions: shared processing (transformer) does work, but shared encoding does not—and the two are separable. This is a more nuanced and practically useful resolution than either extreme.

The research directions that become more attractive after this work are those that treat the transformer as a universal reasoning engine with modality-specialized peripherals, rather than seeking a grand unified representation. The directions that become less attractive are those attempting to design a single visual encoder that simultaneously optimizes semantic extraction and reconstruction fidelity—Table 5 strongly suggests this is a compromised objective whose Pareto frontier is inherently bounded.

However, the paper does not establish that decoupling is necessary at all scales. A critical open question is whether larger LLMs (7B, 13B, 70B) could internally disentangle conflicting signals from a shared encoder, making the decoupling benefit diminish with scale. If so, Janus's contribution would be scale-specific rather than a universal design principle—an important caveat the paper does not address.

Follow-Up Research This Work Enables

Scale sweep: does the decoupling benefit shrink with larger LLMs? Train Janus variants at 300M, 1.3B, 3B, and 7B parameters, each with both decoupled (Exp-D) and shared VQ tokenizer (Exp-A) configurations, using identical training data and compute budgets (matched by steps). Measure the performance gap between decoupled and shared configurations on understanding benchmarks at each scale. The key question: does the gap (e.g., POPE 87.0 vs. 60.1 at 1.3B) narrow proportionally with scale? If a 7B shared-encoder model achieves POPE 82+ simply through greater representational capacity, the architectural argument weakens—larger models could brute-force through the representational conflict. If the gap persists, decoupling becomes a scaling law rather than a small-model fix.

Encoder swap stress-test. Validate the flexibility claim in Section 3.5 by substituting the SigLIP understanding encoder with CLIP-ViT-L, EVA-CLIP, and InternViT, retraining only the understanding adaptor (two-layer MLP) on the same data, and measuring whether understanding performance is maintained or degraded. If adaptor-only retraining recovers comparable performance across all encoders, the frozen-encoder platform design is validated and practitioner confidence in modular upgrades increases. If performance degrades (suggesting the LLM has overfit to SigLIP-specific features despite the adaptor), the flexibility claim is weaker than presented and encoder choices are more coupled than the architecture implies. A strong follow-up would also test whether deeper adaptors (3–4 layer MLPs, or cross-attention-based adaptors) can recover performance when the encoder changes, establishing adapter capacity requirements for future modality additions.

Dynamic resolution understanding with decoupled encoders. The paper mentions (Section 3.5) that dynamic high-resolution techniques can be applied to the understanding pathway without affecting generation. Implement a dynamic high-resolution scheme (e.g., LLaVA-NeXT-style image splitting into multiple 384×384 crops, encoding each with SigLIP, and concatenating features) on the understanding pathway only, while keeping generation at fixed 384×384. Measure whether this improves performance on fine-grained understanding benchmarks (TextVQA for OCR, DocVQA for document understanding, InfoVQA for infographics) without any generation degradation. This is a direct test of the claim that decoupling enables task-specific optimizations without cross-task interference.

Multi-encoder extension to a third modality. Add an audio encoder (e.g., Whisper encoder or CLAP) with its own adaptor to Janus, training only the new adaptor in Stage I and incorporating audio data (speech recognition, audio captioning) into the Stage II data mixture. Measure whether the transformer can learn to process audio without degrading existing understanding and generation performance. This tests the "universal reasoning engine" claim explicitly: if adding a third modality causes performance regression on vision tasks, the transformer's capacity (1.3B parameters, 4096 context length) may be a bottleneck, and the platform claim would be capacity-constrained rather than architecturally general. A negative result here would set important boundaries on when modality addition is "free."

Understanding-to-generation feedback loop. A natural integration the paper does not explore: use the understanding pathway to critique and refine generated images. After the model generates an image autoregressively, feed the generated image back through the SigLIP understanding encoder and ask the model whether the image matches the prompt (e.g., "Does this image contain exactly three apples? Answer yes or no."). If the model detects a mismatch, condition a second generation pass on both the original prompt and the critique. Measure whether this iterative refinement improves GenEval scores, particularly on Counting (Janus's weakest subtask at 0.30). This would demonstrate that the decoupled architecture enables capabilities impossible in shared-encoder models—there, the understanding encoder would see its own generated representations rather than decoded pixels, making self-critique meaningless.

Training data ablation to isolate architectural vs. data contributions. Janus's performance advantage over baselines is confounded with data quality (2M + 4M in-house samples, re-captioned public datasets, aesthetic filtering). Train a Janus model using only the publicly documented data sources (excluding in-house data) and compare against the reported numbers. If performance drops substantially (e.g., COCO-FID rising from 8.53 to 12+, understanding benchmarks dropping by 5+ points), the architectural contribution is smaller than it appears—data quality and volume account for much of the gain. If performance degrades only modestly, the architectural advantage is robust and replicable. This is essential for the academic community to build on the work.

Practical Applications and Downstream Use Cases

Single-model deployment for multi-purpose visual AI assistants. A Janus-style model can serve as the sole backend for applications that require both understanding and generating images—for instance, a design assistant that can critique a user's uploaded image ("the lighting in this photo is too harsh, try shooting during golden hour") and then generate an example image demonstrating the suggestion. The unified architecture eliminates the need to orchestrate separate understanding and generation models (and their associated latency, serialization, and API costs). The paper's numbers make this credible at modest scale: a 1.3B model achieving 87.0 POPE and 69.4 MMB for understanding alongside 8.53 COCO-FID generation can plausibly run on a single GPU without model swapping. For organizations currently maintaining separate LLaVA-like and SDXL-like deployments, a Janus-based system reduces operational complexity and model serving costs, provided the 384×384 generation resolution is acceptable for the use case.

Efficient fine-tuning for domain-specific multimodal applications. Because the encoders are frozen and the adaptors are small (two-layer MLPs), adapting Janus to a new domain (e.g., medical imaging where you need to both interpret X-rays and generate annotated visualizations) requires fine-tuning only the adaptors and the LLM on domain data, without retraining the visual backbones. This is substantially cheaper than fine-tuning a full shared-encoder model where the visual encoder must also adapt. The paper's three-stage training recipe establishes a clear protocol: if you have domain-specific understanding data (image-report pairs) and generation data (condition-to-image pairs), you can conduct a shortened Stage II followed by Stage III fine-tuning, with the frozen SigLIP and VQ tokenizer providing strong visual priors. The risk is that the VQ tokenizer, trained on natural images, may not faithfully reconstruct domain-specific visual features (e.g., medical anomalies), but the generation adaptor can partially compensate.

On-device or edge deployment with parameter-efficient unified capability. A 1.3B LLM with frozen encoders is within the realm of on-device deployment (the total parameter count including visual encoders is likely 2–3B, fitting within 8–16GB of memory with quantization). A mobile application that needs both scene understanding (for accessibility: "describe what's in front of me") and on-device image generation (for creative tools or visual communication) could run Janus without cloud round-trips. The paper's demonstration that a 1.3B model can match or exceed 7B task-specific models on several understanding benchmarks (POPE 87.0 vs. LLaVA-v1.5 7B at 85.9) makes a practical case for preferring a smaller unified model over larger task-specific ones in resource-constrained settings. The main deployment barrier is inference latency for image generation: autoregressive decoding of 576 image tokens at 384×384 resolution, especially with CFG requiring two forward passes per token, may be too slow for interactive use without aggressive optimization (speculative decoding, token parallelism, or reduced CFG steps). This is not benchmarked in the paper and would need measurement.

Data generation for self-improvement pipelines. The unified architecture enables a closed-loop data generation workflow: the understanding pathway can evaluate candidate images against criteria, and the generation pathway can produce new candidates. For instance, in a synthetic data pipeline for training object detectors, the model could generate images from text specifications, evaluate whether the generated images contain the specified objects using its own understanding capability, and filter or re-generate as needed—all within a single model without external verifiers. This is analogous to the self-improvement loops envisioned in the reference paper on test-time compute but applied to multimodal data generation. The GenEval results suggest where this would work (Position 0.46, Color Attribution 0.42) and where it would fail (Counting 0.30), providing a concrete performance envelope for pipeline designers.