Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

ArXiv: 2408.12570

🎯 Pitch

Jamba-1.5 models are the only open-weight LLMs that maintain full accuracy up to 256K context length—dramatically outperforming all others on long-context benchmarks while slashing KV cache memory by 10×. This breakthrough is enabled by a hybrid architecture that scales to 94B active parameters without sacrificing the efficiency gains over pure Transformers.

---

1. Executive Summary

This paper introduces Jamba-1.5, two new instruction-tuned large language models based on the hybrid Transformer-Mamba mixture-of-experts architecture, released in two sizes: Jamba-1.5-Large (94B active / 398B total parameters) and Jamba-1.5-Mini (12B active / 52B total parameters). The models are evaluated across academic benchmarks (MMLU, GPQA, HumanEval), chatbot benchmarks (Arena-Hard, WildBench), and long-context evaluations (RULER, ∞BENCH), where they achieve comparable quality to similarly-sized state-of-the-art open-weight models while providing substantially higher throughput and lower latency — particularly at long contexts, with an order-of-magnitude reduction in KV cache memory (e.g., 9GB vs. 80–88GB for comparably-sized Transformer models at 256K tokens). To enable cost-effective deployment of the Large model on 8×80GB GPUs at 256K-token contexts, the paper introduces ExpertsInt8, a novel calibration-free quantization technique that quantizes MoE and MLP weights to INT8 and dequantizes inside the fused MoE kernel, matching FP8 latency on H100 GPUs while remaining available on A100s where FP8 is unavailable. On RULER, both Jamba-1.5 models are the only publicly available or proprietary models confirmed to maintain an effective context length of 256K tokens, with Jamba-1.5-Large achieving 93.9% accuracy at 256K compared to Gemini-1.5-pro's 65.1%, establishing that the hybrid Transformer-Mamba architecture scales to 94B active parameters while preserving the long-context efficiency that distinguishes it from pure Transformer baselines — though the efficiency gains over comparably-sized models narrow on standard short-context academic benchmarks, where quality is roughly equivalent rather than superior.

2. Context and Motivation

The Core Problem: Transformer Efficiency Breaks Down at Long Contexts

The fundamental problem that Jamba-1.5 addresses is straightforward to state but difficult to solve: Transformer-based large language models become prohibitively expensive at long context lengths, and existing alternatives that improve efficiency either sacrifice quality or haven't been demonstrated to work at the scale of frontier models (tens to hundreds of billions of parameters).

This isn't a niche concern — it's a structural bottleneck that affects virtually every deployment scenario where models need to process long documents, maintain extended conversations, or reason over large codebases. The root cause is the Transformer's self-attention mechanism, which computes pairwise interactions between every token in the context. For a sequence of length $L$, the attention computation scales as $\mathcal{O}(L^2)$ in time and stores a key-value (KV) cache of size $\mathcal{O}(L)$ per layer per head. When $L$ reaches 128K or 256K tokens — lengths that are increasingly demanded by real applications — the KV cache alone can consume hundreds of gigabytes of GPU memory, as Table 1 in the paper quantifies:

LLaMA-3.1-70B requires 80GB of KV cache at 256K context; LLaMA-3.1-405B requires 252GB.

These numbers mean that serving large Transformer models at long contexts requires distributing the model across many GPUs, driving up infrastructure costs and latency. Worse, some models — like LLaMA-3.1-405B — literally cannot fit on an 8×80GB GPU machine at contexts above roughly 100K tokens, as Figure 4 reveals (the LLaMA-3.1-405B latency line truncates at 64K). This creates a hard ceiling: you can have a powerful model or a long context, but not both on affordable hardware.

The real-world impact is immediate. Documents in legal discovery, scientific literature review, and enterprise knowledge management routinely exceed 100K tokens. Conversational agents that maintain long histories, coding assistants that operate on entire repositories, and multi-document summarization systems all need to process contexts that stress or break Transformer architectures. Without architectural innovation, deploying these applications means either (a) accepting degraded quality from context truncation, (b) paying for expensive multi-node GPU clusters, or (c) waiting for hardware improvements that aren't keeping pace with the demand for longer contexts.

The Prior Solution Landscape: Trade-offs Everywhere

The field has developed several approaches to the long-context efficiency problem, each with significant limitations that Jamba-1.5 aims to overcome or sidestep.

State-space models (SSMs) like Mamba [13] replace the quadratic self-attention mechanism with a recurrent computation that scales linearly with sequence length ($\mathcal{O}(L)$). Mamba achieves this through a selective state-space formulation where the state transition matrices are input-dependent, allowing the model to selectively retain or discard information at each time step. The key efficiency win: Mamba maintains a fixed-size hidden state (the "state space") rather than a growing KV cache, so memory usage stays constant regardless of context length. At inference time, this provides dramatic throughput and latency advantages over Transformers at long contexts.

However, SSMs face a fundamental challenge: because information must pass sequentially through the state from token to token, they have more difficulty than Transformers with tasks requiring precise token-level retrieval from arbitrary positions in the past — what the paper calls "needle-in-a-haystack" style tasks. A Transformer's explicit attention can directly attend to any previous token with a single operation; Mamba must encode that token's information into the state at the moment it's processed and hope it's preserved through potentially hundreds of thousands of subsequent steps. The paper's predecessor, Jamba [24], explicitly found that this retrieval limitation is real and motivated the hybrid approach.

Mixture-of-Experts (MoE) [8, 34] addresses a different axis of the efficiency problem: the cost per token (independent of context length). In an MoE Transformer, each token is processed by only a subset of the feed-forward network parameters (the "experts"), selected by a learned routing mechanism. This means the model can have a very large total parameter count (providing capacity) while keeping the per-token computation roughly proportional to a much smaller "active" parameter count. MoE models like Mixtral 8x7B and Mixtral 8x22B have demonstrated that this sparsity approach works well in practice, achieving strong quality with substantially lower FLOPs-per-token than dense models of comparable total size.

But MoE doesn't address the KV cache problem. A Mixtral model still uses full Transformer attention, so its KV cache grows linearly with sequence length regardless of how many experts are activated. At 256K tokens, Mixtral 8x22B still requires 56GB of KV cache (Table 1), and Mixtral 8x7B requires 32GB. MoE makes the model cheaper per token, but doesn't make it cheaper per context length — the memory wall remains.

Efficient attention variants — including grouped-query attention (GQA), multi-query attention, sliding window attention, and sparse attention patterns — reduce the KV cache size by sharing key-value heads or limiting the attention span. LLaMA-3.1 and Mistral-Large-2 both use GQA, which reduces KV heads (e.g., LLaMA-3.1-70B uses 8 KV heads for 64 query heads). This helps — it's why LLaMA-3.1-70B needs 80GB rather than multiples more — but it's a constant-factor improvement. At 256K contexts, even GQA'd Transformers consume memory that forces multi-node deployment or aggressive quantization.

Quantization techniques like GPTQ [9] and FP8 reduce memory by storing weights or activations at lower precision. However, as the paper notes in Section 3.1, these have practical drawbacks: GPTQ requires calibration (which "can take hours or days and can be unstable"), and FP8 is only available on H100 GPUs, not the widely-deployed A100 generation. More fundamentally, quantization reduces the memory footprint of weights but doesn't directly address the KV cache, which scales with context length independently of weight precision.

Prior hybrid architectures. The Jamba predecessor [24] introduced the specific hybrid recipe that Jamba-1.5 builds on: interleaving Transformer attention layers with Mamba SSM layers, in a ratio of 1:7 (one attention layer for every seven Mamba layers), and using MoE for the feed-forward blocks. The key insight driving this design is a division of labor between architectural components: attention layers provide precise token-level retrieval and long-range information pooling (handling the "needle-in-a-haystack" tasks that Mamba struggles with), Mamba layers provide efficient sequence-level processing and state tracking (reducing the memory cost of the bulk of the layers), and MoE provides high model capacity at reduced per-token FLOPs. The original Jamba paper demonstrated this at a smaller scale; subsequent work by other groups [6, 37] confirmed the benefits of hybrid Transformer-SSM designs up to roughly 8B parameters.

The Specific Gap: Does the Hybrid Architecture Scale?

This is the precise gap that Jamba-1.5 addresses. Prior to this release, the hybrid Transformer-Mamba-MoE architecture had been validated at modest scales — the original Jamba, models in the 1–8B range from concurrent work — but no one had demonstrated that the approach continues to work at the 94B-active-parameter scale where frontier open-weight models operate. This scaling question matters because architectural benefits don't always transfer linearly. The concerns are:

Quality cliff. At some model scale, the reduced attention capacity (only 1/8 of layers use attention) might become a binding constraint on quality, preventing the model from matching dense Transformers on reasoning, knowledge, or instruction-following tasks. The paper needs to show that 1:7 attention-to-Mamba ratio is sufficient not just at 12B active parameters (Mini) but at 94B active parameters (Large).

Mamba-1 vs. Mamba-2 in hybrid settings. Mamba-2 [6] was introduced as a faster, higher-quality successor to Mamba-1, with the ability to use much larger state sizes. The natural engineering impulse would be to upgrade Jamba to Mamba-2. But the paper found this intuition is wrong in the hybrid context — Figure 1 shows that Mamba-1-Attention outperforms Mamba-2-Attention at both 350M and 1.3B scale. The authors hypothesize that "some of the advantages of Mamba-2 over Mamba-1, in particular the ability to use a much larger state size, are less significant when we have full attention layers interleaved between the Mamba layers, as they can pool information from the entire context." This is a non-obvious finding that affects architectural decisions for anyone building hybrid models.

Long-context retention through post-training. Instruction tuning typically uses relatively short examples (conversational turns, QA pairs, coding tasks). The paper's stated tension in Section 5.3 is that "most of the available post-training datasets consist of relatively short examples," creating a risk that the model's pre-training long-context capabilities get degraded or forgotten during the fine-tuning phase that gives it conversational and instruction-following skills. Demonstrating that a hybrid architecture can survive this post-training process and still achieve 256K effective context length on RULER — with 93.9% accuracy at 256K — is a genuine technical contribution, not just a scale-up exercise.

Deployment feasibility at scale. The paper's claim that Jamba-1.5-Large fits on a single 8×80GB machine at 256K contexts depends on both the architectural efficiency and the novel ExpertsInt8 quantization technique. If either component fails at scale — if the KV cache somehow grows faster than expected, or if the quantization introduces unacceptable quality degradation — the deployment story collapses. Section 3.1 and the Figure 2 latency benchmarks are designed to validate that the claimed deployment scenario is real, not theoretical.

How This Paper Positions Itself

The paper positions Jamba-1.5 not as a breakthrough in raw benchmark quality — the results on standard academic benchmarks (Table 2) show rough parity with LLaMA-3.1-70B and Mistral-Large-2, not dominance — but as an efficiency-for-quality trade-off that is dramatically favorable at long contexts. The claim is: you can have a model that matches LLaMA-3.1-70B or Mistral-Large-2 on most short-context tasks (within a few percentage points), while being an order of magnitude cheaper to serve at long contexts and being the only open-weight model that actually works at 256K tokens.

This is a pragmatic, deployment-focused positioning. The paper doesn't claim architectural superiority in the abstract — it claims that the hybrid design enables a specific deployment scenario (single-machine 256K-context serving of a frontier-scale model) that competing architectures cannot match. The evidence for this claim comes from three interconnected demonstrations:

1. The architectural scaling evidence (Section 2, Tables 1 and 4): Jamba-1.5-Large operates at 94B active parameters with only 9GB of KV cache at 256K, compared to 80–88GB for comparable Transformers, and achieves 93.9% on RULER at 256K where LLaMA-3.1-70B drops to 66.6% at 128K and can't reach 256K at all.

2. The quantization evidence (Section 3.1, Figure 2): ExpertsInt8 makes the Large model fit on 8×80GB GPUs without calibration overhead or quality loss, matching FP8 latency where available and outperforming GPTQ where it's not.

3. The post-training evidence (Section 5.3, Tables 2, 3, 4, 5): The model acquires competitive instruction-following and conversational capabilities through supervised fine-tuning while retaining its pre-training long-context performance — countering the concern that post-training on short examples would degrade long-context abilities.

The paper explicitly connects to the broader trend of "efficient architectures for long contexts" in the field, citing concurrent hybrid Transformer-SSM work [6, 37] that validates the general approach at smaller scales, and positions Jamba-1.5-Large as the demonstration that these benefits persist to the frontier scale. The reference to Mamba-2 in particular serves a dual purpose: it acknowledges the state of the art in SSM design while establishing that the hybrid setting has different dynamics than pure SSM scaling, making Jamba-1.5's specific design choices (Mamba-1, 1:7 ratio) non-obvious even to researchers familiar with the latest Mamba developments.

The paper also positions itself in the broader landscape of open-weight model releases, implicitly competing with the LLaMA-3.1, Mistral-Large-2, and Mixtral families. The abstract's emphasis on the 256K effective context length being "the largest amongst open-weight models" and the model weights being "publicly available under the Jamba Open Model License" frames the release as filling a specific ecosystem gap: open-weight models that can actually process very long contexts without requiring multi-node GPU clusters.

3. Technical Approach

3.1 Reader orientation (approachable technical breakdown)

Jamba-1.5 is a family of two instruction-tuned large language models — one at 12B active parameters (Mini) and one at 94B active parameters (Large) — designed to process very long sequences (up to 256K tokens) efficiently on modest GPU hardware. The core problem the system solves is the Transformer's quadratic memory cost at long contexts: Jamba-1.5 replaces most of the expensive self-attention layers with a cheaper recurrent state-space mechanism while sprinkling in just enough attention to maintain retrieval accuracy, then makes the whole thing fit on a single 8-GPU machine through a novel calibration-free quantization technique that targets the dominant weight matrices.

3.2 Big-picture architecture (diagram in words)

The architecture has four major components, layered into 9 repeated blocks:

Information flows as a standard decoder: input tokens enter the first block, pass through a sequence of layers (Mamba, Mamba, MoE-MLP, Mamba, Attention, Mamba, Mamba, MoE-MLP, Mamba — following the 1:7 attention-to-Mamba ratio and every-other-layer MoE pattern), with each layer producing a hidden state of dimensionality 8192. The next block receives this hidden state, and so on for 9 blocks (72 total layers: $9 \times 8 = 72$). At generation time, the Mamba layers maintain a small recurrent state (not a growing KV cache), while only the attention layers require a KV cache — and since attention appears in only 1/8 of the layers with 8 KV heads, the total KV cache at 256K tokens is just 9GB for the Large model.

The post-training pipeline (Section 5.3) sits alongside the architecture: it takes the pre-trained base model and fine-tunes it on a mixture of short conversational data, skill-specific synthetic data, and long-context data to produce an instruction-tuned model that retains its pre-training long-context capabilities.

3.3 Roadmap for the deep dive

• First, the Jamba hybrid block structure — the exact composition of layers within each block, the 1:7 attention-to-Mamba ratio, and the MoE configuration — because this is the architectural blueprint that everything else depends on.
• Second, the Mamba-1 vs. Mamba-2 decision (Figure 1), since it's a non-obvious design choice where the "better on paper" component (Mamba-2) actually underperforms in the hybrid setting.
• Third, ExpertsInt8 quantization — how it works mechanically, what makes it different from existing approaches, and why it's essential for the deployment story.
• Fourth, the Activation Loss mechanism (Section 3.2), which solves a numerical stability problem that emerged at scale.
• Fifth, the training infrastructure and three-stage pipeline (pre-training, mid-training, post-training), focusing on the specific design choices for retaining long-context capabilities through instruction tuning.
• Sixth, the synthetic data generation pipelines for post-training, since the paper provides unusually concrete detail about how specific capabilities (table QA, document QA, tool use, steerability) were engineered.

3.4 Detailed, sentence-based technical breakdown

This is primarily a model release and architectural scaling paper. Its core idea is that the specific hybrid recipe introduced in the original Jamba — 1:7 attention-to-Mamba ratio with interleaved MoE — scales to 94B active parameters without hitting a quality cliff, and that the resulting efficiency gap versus pure Transformers widens at long contexts rather than narrowing, enabling deployment scenarios (single-machine 256K-context serving) that competing architectures cannot match.

---

The Jamba Hybrid Block Structure

The fundamental building block of Jamba-1.5 is the Jamba block, repeated 9 times to form the full model. Each block contains exactly $l = 8$ layers. The internal composition of these 8 layers follows a fixed pattern defined by two ratios: the attention-to-Mamba ratio $a:m = 1:7$ and the MoE frequency $e = 2$ (meaning MoE replaces the standard MLP every 2 layers).

Layer ordering. Within each 8-layer block, the paper specifies the $1:7$ ratio — one attention layer for every seven Mamba layers — but does not spell out the exact interleaving pattern in prose. The original Jamba paper [24] placed the attention layer at regular intervals within each block (specifically as the $8^{\text{th}}$ layer in each block, meaning the first block has seven Mamba layers then one attention layer, the second block has seven Mamba layers then one attention layer, and so on). This means attention layers occur exactly once per block, at the block boundary, creating a regular cadence of attention layers spaced 8 layers apart throughout the 72-layer network.

MoE integration. The MoE module replaces the standard feed-forward network (two-layer MLP with activation) in specific layers. With $e = 2$, every other layer uses MoE instead of a single MLP. Within a block of 8 layers, this means 4 layers use MoE. The MoE module has $n = 16$ experts, each being a full feed-forward network. At each token, a learned router selects the top $K = 2$ experts, and the token's output is the weighted sum of those two experts' outputs:

$\text{MoE}(x) = \sum_{i \in \text{TopK}(x, 2)} g_i(x) \cdot \text{Expert}_i(x)$

where $x$ is the token's hidden state (dimension 8192), $\text{TopK}(x, 2)$ selects the indices of the 2 experts with the highest routing scores, and $g_i(x)$ is the softmax-normalized routing weight for expert $i$.

What this computes: for each token at an MoE layer, the router — a small learned linear transformation — produces 16 scores (one per expert). The top-2 scoring experts are activated, their outputs are computed (each expert is a standard FFN), and those outputs are combined with the softmax-normalized routing weights. The token only pays the FLOPs cost of 2 experts rather than 16, giving an 8× reduction in FFN computation per MoE layer.

Why this form: the combination of $e = 2$ and $n = 16, K = 2$ was "found optimal in our work on Jamba [24]." The interleaved pattern (MoE every 2 layers rather than every layer) balances total capacity against the communication overhead of expert routing. Placing MoE too frequently increases the all-to-all communication cost in distributed training without proportional quality gains; placing it too infrequently reduces the model's total parameter count and capacity. The specific choice of 16 experts with top-2 routing is a standard configuration from the MoE literature (Switch Transformers [8], Mixtral) that the prior work validated for the Jamba architecture.

Hidden state dimensionality and attention configuration. The model uses a hidden state dimension of $d_{\text{model}} = 8192$. The attention layers use 64 query heads and 8 key-value heads — this is a grouped-query attention (GQA) configuration with a grouping ratio of $64/8 = 8$ query heads per KV head. GQA reduces the KV cache size by a factor of 8 compared to standard multi-head attention (which would have 64 separate KV heads), and by a factor of roughly $64/8$ compared to having fewer attention layers — so the KV cache savings compound: fewer attention layers (1 in 8) × grouped KV heads (8 instead of 64) yields the dramatic 4–10× reduction in total KV cache seen in Table 1.

Total model dimensions. With 9 blocks of 8 layers each, the model has 72 transformer layers total. Among these 72 layers, $9 \times 1 = 9$ are attention layers and $9 \times 7 = 63$ are Mamba layers. The MoE layers appear in $\lfloor 72 / 2 \rfloor = 36$ layers (every other layer across all blocks), and the remaining 36 layers use standard single-MLP feed-forward networks. The total parameter count of 398B comes from: 94B active parameters (the parameters actually used for any given token — the non-MoE parameters plus 2/16 of the MoE parameters) plus 304B "inactive" parameters (the other 14/16 of the MoE expert parameters that sit in memory but aren't used for a given token). This is why Table 1 lists "Available params: 398B" and "Active params: 94B."

---

The Mamba-1 vs. Mamba-2 Decision (Figure 1)

A natural engineering question when scaling the Jamba architecture is: should we upgrade from Mamba-1 to Mamba-2? Mamba-2 [6] was introduced as a faster, improved successor that reformulates state-space models through the lens of "structured state space duality," connecting SSMs to attention through a matrix formulation. Among its advantages: the ability to use much larger state dimensions (the "state size" that determines how much information each Mamba layer can retain in its recurrent memory), and faster computation through optimized structured matrix operations.

The paper tested this explicitly at two smaller scales — 350M and 1.3B parameter models trained for 100B tokens each — comparing four configurations:

• Mamba-1 alone (pure Mamba-1, no attention)
• Mamba-2 alone (pure Mamba-2, no attention)
• Mamba-1-Attention (hybrid with Mamba-1 + attention, following the Jamba pattern)
• Mamba-2-Attention (hybrid with Mamba-2 + attention)

Figure 1 shows the results. In isolation (without attention), Mamba-2 outperforms Mamba-1 at both scales — this replicates the finding from Dao and Gu [6] that Mamba-2 is a better standalone architecture. However, in the hybrid setting, Mamba-1-Attention outperforms Mamba-2-Attention at both scales, and the hybrid architecture (both variants) outperforms pure Mamba-2.

The paper's hypothesis for this reversal is explicit:

"We hypothesize this is because some of the advantages of Mamba-2 over Mamba-1, in particular the ability to use a much larger state size, are less significant when we have full attention layers interleaved between the Mamba layers, as they can pool information from the entire context."

What this means operationally: in a pure Mamba model, the recurrent state is the only mechanism for passing information across long distances — so a larger state dimension directly improves long-range recall. But in the hybrid architecture, the attention layers act as periodic "global context refreshers." They can directly attend to any token in the past and inject that information back into the hidden state, which the subsequent Mamba layers can then carry forward. This reduces the burden on the Mamba state: it doesn't need to perfectly preserve every potentially relevant detail from hundreds of thousands of tokens ago, because the attention layers periodically re-establish access to that information.

Consequence: Jamba-1.5-Large uses Mamba-1, not Mamba-2. This is a non-obvious design choice — the "better" component in isolation is not better in context — and it matters for anyone reproducing or extending the architecture. The paper also notes that the hybrid architecture "outperform[s] pure Mamba-2," confirming that the addition of attention layers provides benefits beyond what a stronger SSM alone can achieve.

---

ExpertsInt8 Quantization (Section 3.1)

The deployment claim — that Jamba-1.5-Large fits on a single machine with 8 80GB GPUs at 256K context — depends on compressing the model weights. Even with the efficient architecture, 398B total parameters in 16-bit precision would require $398 \times 10^9 \times 2 \text{ bytes} = 796 \text{GB}$ just for the weights, exceeding the $8 \times 80 = 640\text{GB}$ total GPU memory before accounting for the KV cache and activations. Quantization is not optional.

The paper observes a structural property that motivates the approach: "over 85% of the model weights are in the MoE layers, and over 90% are in MoE or MLP layers." This means that if you can quantize the MoE and MLP weights effectively, you capture the vast majority of the memory savings without touching the attention or Mamba parameters (which are proportionally small).

The core mechanism. ExpertsInt8 works as follows:

1. At model loading time (which "only takes a few seconds"): the MoE expert weight matrices and the standard MLP weight matrices are quantized from BF16 (16-bit) to INT8 (8-bit). This is done statically — no calibration data is run through the model, unlike GPTQ which requires hours of calibration on representative inputs to determine per-channel scaling factors.

2. Storage: the weights are stored in GPU memory in INT8 format (1 byte per parameter instead of 2), cutting the memory footprint of these layers roughly in half.

3. During inference, inside the GPU kernel: the INT8 weights are dequantized back to BF16 on-the-fly within the computation kernel, specifically inside the fused_moe kernel in vLLM [18]. The dequantization step happens as the weights are moved from GPU High Bandwidth Memory (HBM) to the GPU's on-chip SRAM (the fast local memory near the compute units). Since the weights travel over the memory bus in INT8 (half the size), they take less time to move, and the dequantization — a simple scaling and casting operation — is performed in SRAM at negligible cost.

4. Activations remain in BF16: unlike FP8 quantization approaches that quantize both weights and activations, ExpertsInt8 keeps the large activation tensors in BF16. This avoids the precision issues that motivate the Activation Loss (Section 3.2) and means the technique works on A100 GPUs (which lack FP8 support) since INT8 arithmetic is universally available.

The paper has contributed this modified fused_moe kernel to the vLLM open-source project.

Why this approach over alternatives.

• Versus GPTQ: GPTQ [9] quantizes weights and then calibrates per-channel scaling factors by passing representative data through the model, which "can take hours or days and can be unstable." ExpertsInt8 avoids calibration entirely — it uses a simple, presumably symmetric quantization scheme (scaling each weight matrix by its absolute maximum value or similar) that takes seconds. Figure 2 shows ExpertsInt8 substantially outperforming GPTQ in latency on both H100 and A100 GPUs.

• Versus FP8: FP8 (8-bit floating point) is supported natively on H100 GPUs and provides good quality, but it is "only available on H100 GPUs" — the widely-deployed A100 generation lacks FP8 support. ExpertsInt8 works on A100s, making it deployable on existing infrastructure. On H100s, Figure 2 shows ExpertsInt8 matches FP8 latency (the lines overlap for Jamba-1.5-Large on 8×H100), so there is no penalty for using the simpler technique.

• Versus no quantization: without quantization, the Large model cannot fit on a single machine at 256K contexts. Figure 2(c) shows the "None" (unquantized BF16) line for Jamba-1.5-Large on 8×H100 has reasonable latency (since the weights fit), but the memory capacity constraint — not latency — is the binding issue.

Performance evidence (Figure 2). The paper measures end-to-end latency (seconds per token) at varying batch sizes with 1024-token contexts and 128-token decoding. For Jamba-1.5-Mini on 2×H100 (Figure 2a), ExpertsInt8, FP8, and GPTQ all cluster together at moderate batch sizes, with GPTQ showing slightly worse latency at larger batches. For Jamba-1.5-Mini on 2×A100 (Figure 2b), where FP8 is unavailable, ExpertsInt8 significantly outperforms GPTQ across all batch sizes (e.g., at batch size 40, roughly 1 s/t vs. 1.5 s/t). For Jamba-1.5-Large on 8×H100 (Figure 2c), ExpertsInt8 and FP8 are indistinguishable, showing the technique "matches FP8 in latency." The Mixtral comparisons (Figures 2d, 2e) serve as cross-validation that the kernel works on standard MoE architectures too.

---

Activation Loss (Section 3.2)

During Jamba-1.5-Large's pre-training, the team observed a numerical stability issue: "certain activations, namely outputs of specific experts as well as the output of the last Mamba layers, were gradually increasing in magnitude for certain input tokens, eventually reaching values as high as $4 \times 10^6$." This is not a training divergence problem (the model continued to train fine in BF16, which can represent values up to roughly $3.4 \times 10^{38}$), but it creates an inference problem: some quantization libraries and inference frameworks support only FP16 for activations, and FP16 has a maximum representable value of 65,504 ($2^{16} \approx 6.5 \times 10^4$). Activations reaching $4 \times 10^6$ would overflow FP16, producing infinity or NaN values.

The solution. The paper introduces an auxiliary loss term added to the standard language modeling cross-entropy loss during training:

$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{LM}} + \alpha \cdot \frac{1}{N} \sum_{i=1}^{N} a_i^2$

where $\mathcal{L}_{\text{LM}}$ is the standard next-token prediction loss, $\alpha$ is a configurable scalar weight, $N$ is the number of activations being regularized, and $a_i$ are the individual activation values (the outputs of specific experts and the last Mamba layers that exhibited the growth).

What it computes: for each forward pass, the mean squared value of the problematic activations is computed and added — scaled by $\alpha$ — to the language modeling loss. This penalizes the model for producing activations with large magnitudes, creating a training signal that pushes them back toward smaller values.

Why this form: mean squared error is the simplest penalty that grows quadratically with magnitude — small activations (near zero) incur negligible penalty, while large activations incur rapidly increasing cost. An L1 penalty ($|a_i|$) would provide a weaker gradient for very large values; the quadratic form ensures the gradient $2\alpha a_i$ scales with the magnitude, providing stronger corrective signal precisely where it's needed.

Practical properties observed:

• The paper found that "this auxiliary loss has no affect on the training even with $\alpha$ values up to at least $10^{-3}$" — meaning the language modeling loss was not degraded by the regularization.
• For Jamba-1.5-Large, they used $\alpha = 10^{-5}$, which "was enough to reduce the activations to an acceptable range (2K-3K max)" — a reduction of roughly three orders of magnitude from the $4 \times 10^6$ peaks.
• The effect is described as "almost instant," meaning the loss could be added "only towards the end of the training without any affect on training speed and quality." This is important because it means the team didn't need to retrain from scratch with the loss active — they could identify the problem late in training and fix it with a minimal intervention.
• Validation: "we ran our full evaluation suite on the model using FP16 activations and obtained the same results as the BF16 evaluations without any nans/overflows."

Why this matters beyond Jamba-1.5: as models scale, numerical stability issues like this become more common. The Activation Loss technique is simple, cheap, and can be applied reactively (when you observe a problem) rather than proactively (requiring architectural changes from the start). The finding that it doesn't affect training quality at $\alpha \leq 10^{-3}$ suggests it's a safe addition to any large-scale training run, and the paper explicitly positions it as an observation for the community to explore further.

---

Training Infrastructure and Three-Stage Pipeline (Sections 5.1–5.3)

The training of Jamba-1.5-Large follows a three-stage process designed to build general capabilities, then specialize for long contexts, then instruct-tune while preserving those long-context capabilities.

Stage 1: Pre-training. The model is trained on "an in-house dataset last updated in March 2024," consisting of "a mixture of publicly available web documents, code, books and scientific articles." The pre-processing pipeline includes "parsing, quality filters, and deduplication," with an in-house parser used "to extract text and formatting" (suggesting they process raw web documents rather than relying on pre-cleaned datasets like CommonCrawl dumps). The exact data mixture "was determined through various ablations" — the paper doesn't specify the ablation methodology or the final proportions.

Crucially, the pre-training data includes multilingual data "with emphasis on the following languages: English, Spanish, French, Portuguese, Italian, Dutch, German, Arabic, and Hebrew." This is notable because the post-training phase uses "only a very small fraction of non-english data, for a few languages and only for specific skills," yet the models perform well on the multilingual MMLU benchmark (Table 6). The paper hypothesizes that "the models are able to use the learned knowledge from that phase when being post-trained mostly in English" — an instance of cross-lingual transfer from pre-training to post-training.

Infrastructure. Training used NVIDIA H100 GPUs with the team's "in-house proprietary framework, which includes FSDP, tensor parallelism, sequence parallelism, and expert parallelism." FSDP (Fully Sharded Data Parallelism) shards model parameters across GPUs, tensor parallelism splits individual layers across GPUs, sequence parallelism distributes the sequence dimension for operations like layer norm, and expert parallelism distributes MoE experts across GPUs (each GPU hosts a subset of the 16 experts). For expert parallelism, the team "adapted MegaBlocks" [10], a framework for efficient sparse MoE training that avoids the padding inefficiencies of traditional all-to-all communication approaches.

Stage 2: Mid-training. After pre-training, the model undergoes "a short phase of mid-training with a high proportion of long documents to emphasize its long-range capabilities." This phase sits between pre-training and post-training. The paper doesn't specify the exact duration, data composition, or proportion of long documents, but the purpose is clear: prime the model's long-context processing capabilities before the instruction tuning phase, which will use mostly short examples. This is a design choice that acknowledges the tension identified in Section 5.3 — that post-training data skews short and could erode long-context ability.

Stage 3: Post-training (supervised fine-tuning). The post-training process "aims to achieve two objectives simultaneously: (i) provide the model with various skills and conversational capabilities; (ii) retain capabilities from pre-training and especially the long-context capabilities from mid-training." These objectives are "partly conflicting, since most of the available post-training datasets consist of relatively short examples."

The approach is supervised fine-tuning (SFT) only — no reinforcement learning from human feedback (RLHF), no direct preference optimization (DPO), no proximal policy optimization (PPO). The paper explicitly states: "while preference tuning algorithms like PPO or DPO improve alignment between model outputs and human intent, we found that the combination of careful synthetic data generation, data filtering, and supervised fine-tuning is crucial for obtaining a strong post-trained model." This is a significant design choice: they're claiming that well-constructed SFT data can achieve competitive alignment without the complexity and instability of preference-based methods.

The SFT data mix includes:

• "High-quality conversational data" (presumably human-written or curated dialogue examples)
• "Skill-specific data" (task-specific training examples for capabilities like function calling, table understanding, steerability)
• "Long-context data" (examples with long inputs to maintain the long-range capabilities from mid-training)

The paper emphasizes that "mixing these different types of data aims to retain long-context capabilities and acquire desired skills." The evaluation results — particularly the 93.9% on RULER at 256K (Table 4) — validate that this mixing strategy works: the model didn't lose its long-context performance despite being fine-tuned primarily on short examples.

Hyperparameter selection. The paper states they "picked our final training recipes (data mix and hyperparameters) based on a battery of mostly internal automatic metrics." The specific hyperparameters (learning rate, batch size, number of epochs, optimizer) are not disclosed, which is a limitation for reproducibility. The two Jamba-1.5 models "are fine-tuned with the same control tokens and formatting template," provided as a HuggingFace-compatible tokenizer and chat template.

Observation on fine-tuning cost. The paper notes a practical benefit of the hybrid architecture: "our efficient Jamba architecture lowers the cost of fine-tuning on long contexts, allowing us to experiment more with a given budget. Thus we could experiment with multiple different training recipes at the post-training stage." This is a meta-observation: the architectural efficiency doesn't just help at inference time but also accelerates the research iteration cycle, enabling more experiments within a fixed compute budget. This is a secondary benefit that's easy to overlook but practically significant for teams developing these models.

---

Synthetic Data Generation Pipelines (Section 5.3)

The paper provides unusually concrete detail about how post-training data was generated synthetically — a departure from many model release papers that treat data generation as a black box. All pipelines follow the same four-stage pattern:

"(i) Sample or generate prompts in a target distribution; (ii) Generate responses from language models; (iii) Filter or rank responses by quality according to automatic validation and scoring; and (iv) Post-edit to remove artifacts and fit desired formatting."

The paper then details four specific pipelines, each targeting a different capability:

Table-based QA. Building on the team's prior work on table understanding [20], this pipeline:

1. Generates "tabular data and accompanying question-answer pairs" — synthesizing both the table structure and the questions about it.
2. Converts the tables "into natural language paragraphs using a language model" — translating structured data into prose descriptions.
3. Generates training examples covering "extraction, aggregation, and attribution tasks vis-a-vis text corresponding to specific rows or columns in a given table."

The result is a dataset that teaches the model to reason about structured information presented in natural language — answering questions like "what was the total revenue in Q3?" from a paragraph that describes a financial table.

Document QA. Given source documents:

1. A language model generates question-answer pairs "for both single and multiple paragraphs" — some questions require reasoning across paragraph boundaries.
2. These examples are "embedded within longer context by adding similar texts" — the question-answer pairs are placed inside much longer documents, forcing the model to locate the relevant information within a larger context. This directly targets the long-context understanding capability, teaching the model to attend to specific information in a "needle-in-haystack" style but with naturalistic content.

Tool use. Starting from the open-source Glaive function-calling dataset [1]:

1. The dataset is "filtered with various heuristics and validations on the output schemas" — removing examples with malformed function definitions or invalid parameter assignments.
2. To support parallel function calling (invoking multiple functions simultaneously), the pipeline generates "multiple valid parameter assignments for each function in Glaive," then samples "subsets of these valid parameter assignments, for the same function and across different functions, to generate user requests corresponding to the set of function calls."
3. A "function-calling language model" responds to these generated user requests, and only responses where the function calls matched the original parameter assignments are retained.

The key insight here: by synthesizing user requests that require multiple function calls, then validating that the model's generated function calls exactly match the intended calls, the pipeline creates training data for a capability (parallel tool use) that's scarce in human-generated data.

Steerability. This pipeline targets instruction-following with multiple constraints:

1. The team "defined a set of instructions that can be easily validated and synthesized prompts that include a generic document-drafting task with one or more constraints added to it." Example: "write a cover letter for a job application to Company X, mentioning skills A and B, in a formal tone, under 300 words."
2. A language model generates completions for these constrained prompts.
3. Rejection sampling is applied: responses are validated against the fine-grained instructions (did it mention skill A? is it under 300 words? is the tone formal?) using automatic checks and a "general-purpose reward model."
4. To teach the model to respect system message instructions, the pipeline reformats multiple prompts that share a fine-grained instruction into a multi-turn conversation, with the shared instruction moved to the system message. This teaches the model that system messages contain persistent constraints that apply across multiple turns.

The innovation here is the combination of synthetic constraint generation (creating novel combinations of constraints), automatic validation (checking constraint satisfaction without human annotation), and structural reformatting (moving instructions to system messages to teach the appropriate behavior).

Why synthetic data at this scale: the paper's approach reflects the broader trend in foundation model development [7] of using model-generated training data to scale post-training beyond what human annotation budgets allow. The specific contribution is the detailed description of individual pipelines and the emphasis on automatic validation — each pipeline has a verifiable correctness criterion (tables can be checked, function calls can be pattern-matched, constraints can be automatically tested) that enables high-quality filtering without human review. This is what the paper means by "careful synthetic data generation, data filtering, and supervised fine-tuning" being "crucial" — the quality comes from the validation and filtering, not just from using a strong generator model.

The paper also makes a notable observation about multilingual transfer: even though the post-training phase included "only a very small fraction of non-english data, for a few languages and only for specific skills," the resulting models perform well on multilingual benchmarks (Table 6), with Jamba-1.5-Mini averaging 64.30 across 7 languages on multilingual MMLU vs. LLaMA-3.1-8B's 56.83. The hypothesis is that the multilingual knowledge embedded during pre-training survives the English-dominated post-training phase and can be accessed by the instruction-tuned model — an empirical finding about cross-lingual transfer that the paper presents as a point for community investigation.

4. Key Insights and Innovations

Innovation 1: The Hybrid Architecture Scales — and the Scaling Behavior Is Non-Monotonic

The most significant intellectual contribution of this paper is an empirical finding that contradicts the default assumption of the efficient architecture literature: adding attention to Mamba doesn't just help at small scales — it helps more at large scales, and the "better" SSM component (Mamba-2) is actually worse in the hybrid setting.

The dominant narrative in efficient sequence modeling has been progressive. Transformers are expensive, so we develop SSMs. SSMs are efficient but struggle with retrieval, so we develop better SSMs (Mamba-1 → Mamba-2, with larger state sizes and faster kernels). The natural engineering trajectory is to keep upgrading the SSM backbone and, if hybridization helps, to use the best available SSM. The implicit assumption is that components that are better in isolation will be better in combination.

Figure 1 flatly contradicts this. Mamba-2 does outperform Mamba-1 as a pure architecture (replicating Dao and Gu [6]), but Mamba-1-Attention outperforms Mamba-2-Attention at both 350M and 1.3B scales. The paper's diagnostic explanation — that attention layers reduce the burden on the Mamba state, making Mamba-2's larger state size less necessary — is not just a hypothesis but a conceptual reframing: in a hybrid architecture, the components are not additive but substitutive. The attention layers don't augment the Mamba state — they partially replace its function, changing which Mamba properties matter. This is a fundamental design insight for hybrid architectures: you don't just pick the best components from each paradigm and combine them. You need to understand how the combination changes the requirements on each component, which may invert the ranking you'd get from standalone benchmarks.

The scaling dimension reinforces this. Prior hybrid Transformer-SSM work [6, 37] had validated the approach up to roughly 8B parameters. Jamba-1.5 demonstrates it at 94B active parameters — more than an order of magnitude larger — and the evidence shows that the efficiency gap widens rather than narrows. Compare Table 1: at 256K context, Jamba-1.5-Large uses 9GB of KV cache vs. 80GB for LLaMA-3.1-70B and 88GB for Mistral-Large-2 (both roughly comparable active parameter counts). The KV cache ratio is not a constant factor — it's a function of the attention-to-Mamba ratio (1:7) and the grouped-query configuration (8 KV heads), both of which scale independently of model size. As models grow, the absolute memory savings grow proportionally.

This is a conceptual shift from how the field typically thinks about architectural innovation. Most architecture papers claim "our method is better" in terms of quality-per-parameter or quality-per-FLOP on standard benchmarks. Jamba-1.5's claim is different: quality parity at short contexts, order-of-magnitude efficiency at long contexts, and the unique ability to actually operate at 256K tokens on affordable hardware. The RULER results (Table 4) make this concrete: LLaMA-3.1-70B achieves 88.4% at 64K and 66.6% at 128K but cannot reach 256K at all on 8 GPUs; Jamba-1.5-Large achieves 95.4% at 64K, 95.1% at 128K, and 93.9% at 256K. The quality doesn't just stay acceptable — it barely degrades across a 64× range of context lengths. This suggests something deeper than just "cheaper attention": the hybrid design may be more robust to context scaling even at lengths where pure Transformers haven't yet hit the memory wall.

---

Innovation 2: Calibration-Free Quantization That Exploits MoE Sparsity Structure

ExpertsInt8 is not just "another quantization technique" competing with GPTQ and FP8 on the same playing field. Its intellectual contribution is in identifying and exploiting a structural property of MoE models that existing quantizers don't leverage: the extreme concentration of parameters in specific layer types.

The observation (Section 3.1) is that MoE and MLP layers contain over 90% of the total parameters. This is not true of dense Transformer models, where attention weights, embedding tables, and MLPs are more evenly balanced. In an MoE model, the expert feed-forward networks dominate the parameter count because the 16 experts collectively store a massive weight matrix that is activated sparsely. Standard approaches to quantization (GPTQ, AWQ, SmoothQuant) treat all linear layers uniformly — they may apply different bit-widths to different layers, but they don't fundamentally change their strategy based on which layers dominate the memory.

ExpertsInt8 makes a simple but powerful design choice: only quantize the layers where it matters most (MoE and MLP), leave everything else in BF16, and optimize the dequantization by fusing it directly into the existing MoE kernel rather than introducing a separate dequantize-compute pipeline. The result is a technique that is simultaneously faster to apply (no calibration — seconds vs. hours), faster at inference (dequantization hidden inside the memory transfer), and more broadly deployable (works on A100s where FP8 is unavailable) than general-purpose alternatives.

The intellectual move here is co-designing the quantization strategy with both the model architecture and the inference framework. The insight that "over 85% of weights are in MoE layers" only matters because the vLLM fused_moe kernel provides a natural insertion point for on-the-fly dequantization. This kernel already fuses the expert routing, weight loading, and computation into a single operation to avoid multiple GPU kernel launches. By embedding the int8→BF16 conversion inside this existing fusion, ExpertsInt8 achieves what the paper calls "negligible overhead" — the dequantization computation is amortized over the much larger cost of loading weights from HBM and computing the expert outputs. A general-purpose quantizer that treats all layers uniformly would miss this optimization opportunity.

Figure 2 demonstrates the practical consequence: on H100 GPUs, ExpertsInt8 matches FP8 latency; on A100 GPUs (where FP8 is unavailable), it substantially outperforms GPTQ. The contribution is not just a new quantization algorithm — it's a demonstration that quantization strategies should be architecture-aware and deployment-aware, inverting the typical approach where quantization is treated as a post-hoc compression step applied uniformly to a trained model.

---

Innovation 3: Post-Training Long-Context Retention as an Explicit Design Objective

Most instruction-tuned model releases treat long-context capability as something that's acquired during pre-training and hopefully survives post-training. Jamba-1.5 treats it as an active design objective that must be engineered against the natural tendency of instruction tuning to degrade it.

The paper explicitly frames this as a conflict in Section 5.3: post-training datasets consist "of relatively short examples" (conversational turns, QA pairs, coding tasks), but the model must retain the ability to process 256K-token contexts. The typical approach — just throw in some long-context training examples and hope — is replaced by a structured three-stage pipeline where mid-training (Stage 2, explicitly designed to "emphasize long-range capabilities") serves as a buffer between pre-training and the short-context-dominated post-training phase.

This is significant because it identifies a failure mode that's easy to overlook. When a model's RULER score drops from 90% to 50% at 128K after instruction tuning, it's tempting to attribute this to the base model's limitations or to assume the task is too hard. The Jamba-1.5 results demonstrate that the drop is not inevitable — it can be engineered away by (a) mid-training with long documents, (b) including long-context examples in the post-training mix, and (c) having an architecture (the hybrid Mamba-attention design) that may be more resistant to context-length degradation because it doesn't rely exclusively on the attention mechanism's KV cache for long-range information access.

The RULER results (Table 4) provide the evidence that this engineering works. Jamba-1.5-Large's accuracy drops from 96.7% at 4K to 93.9% at 256K — a decline of less than 3 percentage points over a 64× increase in context length. Compare Gemini-1.5-pro (96.7% → 65.1%, a 31.6-point drop) or LLaMA-3.1-70B (96.5% → 66.6% at 128K, and it can't reach 256K). The Jamba models' RULER curves are remarkably flat — they're not just "better at long context," they're specifically better at maintaining their short-context performance when the context grows.

The conceptual contribution here is framing post-training as a capability preservation problem rather than just a capability acquisition problem. The paper's data mixing strategy (conversational + skill-specific + long-context data) and the explicit mid-training stage for long documents are concrete realizations of this framing, and the results demonstrate it works. This is a lesson that transfers to any model family, not just hybrid architectures: if you want your instruction-tuned model to retain long-context capabilities, you need to actively preserve them during post-training, not just hope they survive.

---

Innovation 4: The "Careful SFT" Thesis — Preference Optimization May Be Unnecessary at Scale

The paper makes a provocative claim about post-training methodology that challenges the dominant RLHF/DPO paradigm. In Section 5.4, it states:

"while preference tuning algorithms like PPO or DPO improve alignment between model outputs and human intent, we found that the combination of careful synthetic data generation, data filtering, and supervised fine-tuning is crucial for obtaining a strong post-trained model."

This is a substantive methodological claim, not a casual observation. The paper is arguing that when SFT data is constructed with sufficient care — through multiple synthetic generation pipelines with automatic validation and rejection sampling (Section 5.3) — the additional complexity of preference optimization may be unnecessary for achieving competitive quality.

The evidence for this claim is indirect but notable: Jamba-1.5-Large achieves 65.4 on Arena-Hard (Table 3, exceeding LLaMA-3.1-70B's 55.7 and competitive with models that likely used extensive RLHF), 81.5 on IFEval instruction following (Table 2), and strong safety metrics (RealToxicity 6.7, TruthfulQA 58.3). These are precisely the dimensions where preference optimization is supposed to help — instruction following, alignment with human intent, harmlessness — and the model performs well using only SFT.

The intellectual move is to reframe the debate from "SFT vs. RLHF" to "data quality × training algorithm." The implicit argument is that the gains typically attributed to RLHF/DPO might actually be achievable through better data construction — specifically, synthetic data pipelines with verifiable correctness criteria and rejection sampling (which is itself a form of preference signal, but applied at the data filtering stage rather than at the training objective stage). The four detailed synthetic data pipelines (table QA, document QA, tool use, steerability) each include an automatic validation step that serves the same function as a reward model — distinguishing good from bad outputs — but embeds that signal into the training examples rather than into the loss function.

This is a "negative result with implications" innovation. It doesn't prove that RLHF is unnecessary — the comparison isn't controlled (we don't see the same data with and without RLHF). But it provides a strong existence proof: you can build a competitive instruction-tuned model at the 94B-active-parameter scale without preference optimization, if you're willing to invest heavily in synthetic data engineering. For the broader community, this suggests that data pipeline investment may have higher marginal returns than algorithmic sophistication for post-training — a hypothesis that deserves controlled testing but is consistent with the Jamba-1.5 results.

---

Innovation 5: Activation Loss as a Reactive Numerical Stability Tool

The Activation Loss technique (Section 3.2) is, in isolation, a small engineering fix. But its intellectual contribution is the concept of reactive, late-training numerical stabilization — the observation that large-scale training can develop numerical pathologies that are invisible during most of the run and can be fixed with a minimal, targeted intervention rather than requiring architectural redesign or restart.

The specific pathology — activation magnitudes in specific experts and final Mamba layers growing to 4×10⁶ — is likely specific to the Jamba architecture and training setup. But the category of problem is universal: as models scale, interactions between components that were benign at smaller scales can produce numerical instabilities that only manifest after billions of training tokens. The dominant approach in the field is to prevent these through careful initialization, gradient clipping, and normalization — all proactive measures. The Activation Loss represents a complementary reactive strategy: train normally, monitor for pathologies, and apply a cheap auxiliary loss to fix them when they appear.

The key empirical finding that makes this a methodological contribution rather than a one-off hack: the loss "has no affect on the training even with α values up to at least 10⁻³" and "reduced the activations almost instantly, allowing it to be added only towards the end of the training without any affect on training speed and quality." This means the technique has a very wide safety margin — you can add it late, set α conservatively, and be confident it won't degrade the model. The validation that FP16 inference matches BF16 results after the fix confirms that the problem was solved, not just masked.

For practitioners training large models, this is an innovation in training methodology: it provides a tool for addressing a category of problem that isn't well-covered by existing best practices, and it comes with empirical calibration (α = 10⁻⁵ worked for a 94B-active-parameter model; values up to 10⁻³ were tested to be safe). The paper explicitly presents this as an observation "for the community to look further into," positioning it as a hypothesis-generating contribution rather than a closed solution.

5. Experimental Analysis

Evaluation Methodology

• Datasets and benchmarks. The paper evaluates on a wide battery spanning four categories: (1) Academic benchmarks: MMLU [16] (5-shot), MMLU-Pro [38] (5-shot), GPQA [31] (0-shot), ARC-Challenge [5] (0-shot), BBH [35] (3-shot), HumanEval [4] (pass@1), GSM8K (5-shot), IFEval [42] (0-shot), and BFCL v1 [40] (0-shot). (2) Chatbot benchmarks: Arena-Hard [22] (500 challenging user queries, GPT-4-Turbo judge) and WildBench [25] (GPT-4-Turbo judge with length-bias mitigation). (3) Long-context benchmarks: RULER (13 synthetic tasks including needle-in-a-haystack variants, variable tracking, aggregation, and QA at lengths up to 256K tokens) and ∞BENCH (novel-length QA tasks EN.QA and EN.MC, average length ~100K tokens). (4) Safety and multilingual: RealToxicity [12] (average toxicity), TruthfulQA [26] (0-shot), and multilingual MMLU [19] (covering Spanish, Portuguese, French, German, Arabic, Italian, Dutch). The long-context evaluations are the paper's primary differentiator; the academic and chatbot benchmarks establish quality parity with existing models.

• Base models. Two model sizes within the same Jamba-1.5 family: Jamba-1.5-Mini (12B active / 52B total parameters) and Jamba-1.5-Large (94B active / 398B total parameters). Both are instruction-tuned versions of the Jamba hybrid Transformer-Mamba-MoE architecture, with the Mini being an updated version of the original Jamba release [24] and the Large representing the first demonstration of this architecture at frontier scale. The family-internal comparison (Mini vs. Large) allows assessment of scaling behavior within the same architecture.

• Metrics. Accuracy is the primary metric across all benchmarks: exact match for MMLU, MMLU-Pro, ARC-C, GSM8K; pass@1 for HumanEval; task-specific scoring for GPQA, BBH, IFEval, BFCL, RULER, and ∞BENCH. Arena-Hard uses GPT-4-Turbo as judge producing a win-rate score; WildBench uses a length-bias-mitigated GPT-4-Turbo judge. RealToxicity reports average toxicity score (lower is better). Latency is measured as seconds per token (s/t) for end-to-end generation; throughput is output tokens per second (t/s). KV cache memory is reported in GB at 256K context with 16-bit precision.

• Baselines. The paper compares against the following open-weight models, grouped by approximate active parameter count: Against Jamba-1.5-Mini (~12B active): LLaMA-3.1-8B [7], Gemma-2-9B, Mixtral 8x7B, and Mistral Nemo 12B. Against Jamba-1.5-Large (~94B active): LLaMA-3.1-70B [7], Mistral-Large-2 (123B active), and (for throughput/latency only) LLaMA-3.1-405B. For long-context evaluations (RULER, Table 4), a broader set of models is compared including Gemini-1.5-pro, GPT-4-1106-preview, Qwen2-72B, Command-R+, Command-R, Yi-34B, Phi-3-mini, Phi-3-medium, Mixtral 8x22B, Mixtral 8x7B, Mistral Nemo 12B, and DBRX. Baseline results are either "taken from official sources or evaluated by us, as indicated in the table" (Table 2 note), with the RULER baselines sourced from the RULER GitHub repository. The paper notes two cases where they "failed to obtain good results": Mistral-Large-2 on ARC-C (despite multiple attempts) and LLaMA-3.1 models on GSM8K with strict evaluation (flexible evaluation results are also reported).

• Generation budget and compute accounting. For the throughput and latency analysis (Figures 3, 4), measurements use batch size 1 (single-query latency) with output length fixed at 512 tokens, varying the total context length from 4,096 to 262,144 tokens. Hardware configuration: Jamba-1.5-Mini and its comparables tested on 2×A100 80GB GPUs; Jamba-1.5-Large and its comparables tested on 8×A100 80GB GPUs. For the quantization benchmarks (Figure 2), context length is 1,024 tokens with 128-token decoding, tested on 2×H100, 2×A100, or 8×H100 depending on model size. The "effective context length" on RULER is determined by the longest context where a model maintains accuracy above a threshold (the paper doesn't specify the exact threshold, but the conventional RULER threshold is typically >85% across tasks, consistent with the reported results showing Jamba-1.5-Large at 93.9% at 256K).

• Cross-validation / statistical protocol. The paper does not describe any cross-validation, statistical significance testing, or confidence intervals. Results are reported as single-run evaluation scores. For benchmarks where the model's own evaluation is reported (marked with † or ⋄ in Tables 2, 3, 5), the evaluation methodology follows the standard protocol for each benchmark (e.g., LM Evaluation Harness for MMLU, the RULER evaluation script). The GSM8K evaluation uses both "strict" and "flexible" evaluation to accommodate LLaMA-3.1's poor strict-mode performance, but this is a per-benchmark adaptation rather than a statistical protocol. The absence of error bars, multiple random seeds, or statistical testing is a limitation — all reported differences of 1–3 percentage points should be interpreted as within the range of evaluation noise for these benchmarks (particularly on the 500-question RULER and Arena-Hard sets).

Main Quantitative Results

Standard Academic Benchmarks (Table 2)

Headline finding: Jamba-1.5 models achieve rough parity with comparably-sized state-of-the-art open models on standard short-context academic benchmarks, with no single model dominating across the board. The efficiency advantages of the Jamba architecture do not come at a quality cost on these metrics, but they also do not produce quality gains — the models are competitive, not superior.

Jamba-1.5-Large vs. LLaMA-3.1-70B vs. Mistral-Large-2 (123B active):

• MMLU (5-shot): Jamba-1.5-Large scores 80.0, versus 83.6 for LLaMA-3.1-70B and 82.5 for Mistral-Large-2 — a deficit of 2.5–3.6 percentage points to the best competitor.
• MMLU-Pro (5-shot): 48.3 vs. 53.0 (LLaMA) and 54.2 (Mistral) — a larger gap of 4.7–5.9 points, suggesting harder reasoning tasks may expose the reduced attention capacity more than standard MMLU.
• GPQA (0-shot): 36.9 vs. 36.0 (LLaMA) and 40.7 (Mistral) — essentially tied with LLaMA, behind Mistral by 3.8 points.
• ARC-Challenge (0-shot): 93.0 vs. 94.8 (LLaMA) and 65.0 (Mistral, which the paper notes "fails to obtain good scores on ARC-C despite multiple attempts"). Against the functioning baseline (LLaMA), Jamba trails by 1.8 points.
• BBH (3-shot): 65.5 vs. 69 (LLaMA) and 70.8 (Mistral) — a consistent 3.5–5.3 point deficit.
• HumanEval (pass@1): 71.3 vs. 80.5 (LLaMA) and 92 (Mistral) — a substantial code generation gap, with Jamba trailing LLaMA by 9.2 points and Mistral by 20.7 points. This is the largest relative weakness across all benchmarks and may reflect the reduced attention capacity being more detrimental for code generation tasks that require precise token-level reasoning.
• GSM8K (5-shot): 87.0 vs. 71.5/94.2 (LLaMA, strict/flexible) and 91.0 (Mistral). Under strict evaluation, Jamba significantly outperforms LLaMA (87.0 vs. 71.5) but trails Mistral (91.0). Under flexible evaluation, LLaMA jumps to 94.2, surpassing Jamba. The evaluation sensitivity on this benchmark makes cross-model comparison unreliable.
• IFEval (0-shot): 81.5 vs. 87.5 (LLaMA) and 87.8 (Mistral) — trailing by ~6 points on instruction following.
• BFCL (0-shot): 85.5 vs. 84.8 (LLaMA) and 85.1 (Mistral) — essentially tied, with Jamba slightly ahead of both.
• RealToxicity (avg toxicity): 6.7 for Jamba-1.5-Large vs. unreported for LLaMA-3.1-70B and Mistral-Large-2. The paper does not provide comparative safety numbers for the larger baselines.
• TruthfulQA (0-shot): 58.3 vs. 60.7 (LLaMA) and 50.4 (Mistral) — between the two baselines, trailing LLaMA by 2.4 points but ahead of Mistral by 7.9.

Pattern: Across 10 academic benchmarks (excluding the problematic ARC-C for Mistral and GSM8K evaluation sensitivity), Jamba-1.5-Large is numerically ahead of the best competitor on 1 benchmark (BFCL, by 0.4–0.7 points), essentially tied on 2 (GPQA with LLaMA, GSM8K depending on evaluation), and behind on 7. The deficits range from 1.8 points (ARC-C vs. LLaMA) to 9.2 points (HumanEval vs. LLaMA) to 20.7 points (HumanEval vs. Mistral). The average deficit to the best-performing competitor across the clean benchmarks is roughly 4–6 percentage points. This represents a quality parity-minus result: the model is in the same competitive tier but consistently slightly behind the state-of-the-art on short-context academic metrics.

Jamba-1.5-Mini vs. LLaMA-3.1-8B vs. Gemma-2-9B:

• MMLU (5-shot): 69.7 (Jamba), 69.4 (LLaMA), 71.3 (Gemma) — effectively a three-way tie within 1.6 points.
• MMLU-Pro (5-shot): 39.8 vs. 38.0 (LLaMA) vs. 39.0 (Gemma) — essentially tied, with Jamba slightly ahead.
• GPQA (0-shot): 32.3 vs. 27.0 (LLaMA) vs. 36.0 (Gemma) — Jamba between the two, trailing Gemma by 3.7 points.
• ARC-C (0-shot): 85.7 vs. 83.4 (LLaMA) vs. 68.4 (Gemma) — Jamba leads LLaMA by 2.3 points and substantially outperforms Gemma.
• BBH (3-shot): 53.4 vs. 51.0 (LLaMA) vs. 60.0 (Gemma) — trailing Gemma by 6.6 points.
• HumanEval (pass@1): 62.8 vs. 72.6 (LLaMA) vs. 40.2 (Gemma) — behind LLaMA by 9.8 points but substantially ahead of Gemma.
• GSM8K (5-shot): 75.8 vs. 75.2/83.7 (LLaMA, strict/flexible) vs. 68.6 (Gemma) — ahead of Gemma, competitive with LLaMA depending on evaluation strictness.
• IFEval (0-shot): 75.8 vs. 80.4 (LLaMA) vs. 74.3 (Gemma) — between the two, trailing LLaMA by 4.6 points.
• BFCL (0-shot): 80.7 vs. 76.1 (LLaMA) vs. not reported (Gemma lacks function-calling capabilities) — Jamba leads LLaMA by 4.6 points, the Mini model's largest advantage.
• RealToxicity: 8.1 (Jamba) vs. 8.2 (Gemma) — essentially tied; LLaMA not reported.
• TruthfulQA: 54.1 (Jamba), 51.5 (LLaMA), 50.2 (Gemma) — Jamba ahead of both by 2.6–3.9 points.

Mini pattern: The Mini model shows a more mixed profile than Large. It leads on some benchmarks (ARC-C, BFCL, TruthfulQA) and trails on others (HumanEval, BBH, IFEval). The average performance is highly competitive with the two baselines — there's no consistent deficit like the ~4–6 point gap observed for Large. This suggests the architectural trade-off (reduced attention for efficiency) may be more favorable at the 12B-active scale than at 94B-active, where the attention deficit becomes more binding for certain reasoning tasks.

---

Chatbot Evaluations (Table 3)

Headline finding: Jamba-1.5-Large achieves strong chatbot performance, surpassing LLaMA-3.1-70B on both Arena-Hard and WildBench, but trailing Mistral-Large-2 (which has ~30% more active parameters).

• Arena-Hard: Jamba-1.5-Large: 65.4; LLaMA-3.1-70B: 55.7; Mistral-Large-2: 70.4. Jamba leads LLaMA by 9.7 points and trails Mistral by 5.0 points.
• WildBench: Jamba-1.5-Large: 48.5; LLaMA-3.1-70B: 49.8; Mistral-Large-2: 56.3. Jamba essentially ties LLaMA (1.3-point deficit, within evaluation noise) and trails Mistral by 7.8 points.

Jamba-1.5-Mini: Arena-Hard 46.1 (substantially ahead of LLaMA-3.1-8B's 21.3 and Gemma-2-9B's 43.2); WildBench 42.4 (trailing Gemma's 42.7 but ahead of LLaMA's 33.6).

The Arena-Hard results are notably strong for both Jamba models — the Large model's 9.7-point lead over LLaMA-3.1-70B on a challenging, GPT-4-judged benchmark is the paper's clearest evidence of quality advantage over a direct competitor. The WildBench results are more modest but still competitive. Mistral-Large-2's consistent lead (~5–8 points) across both benchmarks is attributed to its larger active parameter count (123B vs. 94B).

---

Long-Context Evaluations (Tables 4, 5)

This is the paper's strongest experimental section and the primary evidence for its central claim that the Jamba architecture enables long-context performance that competing architectures cannot match.

RULER (Table 4):

Headline claim verified: Jamba-1.5-Mini and Jamba-1.5-Large are "the only ones with a confirmed effective length of 256K tokens" among all publicly available and proprietary models tested. The "Claimed Effective Length" column in Table 4 reports 256K for both Jamba models, compared to >128K (but not 256K) for Gemini-1.5-pro, 64K for GPT-4-1106-preview, 64K for LLaMA-3.1-70B, and 32K or lower for most others. The "Effective Length" column validates this: Jamba-1.5-Large achieves 256K effective length; Gemini-1.5-pro drops to 65.1% at 256K (the paper notes they "were unable to reproduce" Gemini's previously-reported 128K+ results and hypothesizes the model was updated).

Jamba-1.5-Large RULER trajectory (4K → 256K):

Key observations:

1. Near-flat scaling curve for Jamba: The Large model's accuracy drops only 2.8 percentage points from 4K to 256K (96.7 → 93.9). This is remarkably flat — the model is essentially unaffected by context length increases over a 64× range.

2. Dramatic degradation for Transformers: LLaMA-3.1-70B drops 29.9 points from 4K to 128K (96.5 → 66.6) and cannot be evaluated at 256K because it "is too large to fit context lengths greater than ~100K tokens on 8 80GB GPUs" (Figure 4 caption). Mistral-Large-2 drops 72.5 points from 4K to 128K (96.2 → 23.7) — a catastrophic failure that the paper attributes to the model's context window being insufficient. Both models are effectively unusable at 256K, either for memory or quality reasons.

3. Gemini-1.5-pro collapses at 256K: The paper's own evaluation shows Gemini dropping to 65.1% at 256K, with the note that they "examined Gemini-pro generations and noticed the model often fails to answer or generates a refusal." This is a rare example of a paper directly contradicting a proprietary model's claimed capabilities through empirical testing.

4. Jamba-1.5-Mini also leads its class: At 256K, Mini achieves 86.1% — lower than Large's 93.9% but far ahead of any other sub-20B model. The next-best model at 128K (since most smaller models can't reach 256K) is LLaMA-3.1-8B at 77.0% (128K), with Command-R at 76.0% (128K).

5. The effective length metric is binary but continuous performance is more informative: While the paper emphasizes that Jamba models are the "only" ones at 256K, the more practically significant finding is the shape of the scaling curves. Jamba's accuracy is essentially flat; competitors show steep degradation well before hitting memory limits. This means the Jamba advantage is not just about reaching 256K — it's about maintaining performance at any long context beyond ~32K.

∞BENCH (Table 5):

Jamba models also excel on this naturalistic long-context benchmark with ~100K average context length, which tests novel-length comprehension through question answering (EN.QA) and multiple-choice (EN.MC):

• EN.MC (multiple choice): Jamba-1.5-Large: 80.4; LLaMA-3.1-70B: 78.2; Mistral-Large-2: 36.9. Jamba leads LLaMA by 2.2 points and more than doubles Mistral's score, which appears to fail on this task despite its large parameter count.
• EN.QA (generative QA): Jamba-1.5-Large: 34.9; LLaMA-3.1-70B: 36.7; Mistral-Large-2 not reported. Jamba trails LLaMA by 1.8 points — essentially tied within evaluation noise.
• Jamba-1.5-Mini: EN.MC 76.9 (vs. LLaMA-3.1-8B 65.1); EN.QA 40.6 (vs. LLaMA-3.1-8B 27.1). The Mini model substantially outperforms LLaMA-3.1-8B on both tasks, leading by 11.8 and 13.5 points respectively.

These results complement the RULER findings: Jamba's advantage is clear on both synthetic retrieval tasks (RULER needle-in-haystack) and naturalistic comprehension tasks (∞BENCH novel QA), suggesting the hybrid architecture's benefits extend beyond simple pattern matching to genuine long-range understanding.

---

Multilingual Evaluation (Table 6)

Headline finding: Jamba-1.5 models exhibit strong multilingual performance despite minimal multilingual post-training data, with the Mini model leading its comparables and the Large model trailing but remaining competitive.

Jamba-1.5-Mini averages 64.30 across 7 languages on multilingual MMLU, compared to LLaMA-3.1-8B's 56.83 and Gemma-2-9B's 63.34. Jamba leads on 6 of 7 individual languages (all except French, where it's 65.9 vs. Gemma's 66.7), with the largest advantages in German (63.8 vs. 57.2 for LLaMA, 64.3 for Gemma) and Arabic (57.3 vs. 46.9 for LLaMA, 55.9 for Gemma).

Jamba-1.5-Large averages 73.94, trailing LLaMA-3.1-70B (77.76) by 3.82 points and Mistral-Large-2 (76.19) by 2.25 points. The pattern is consistent across all 7 languages — Jamba is behind both baselines in every language but the gap is modest (3–5 points typically). The paper attributes the multilingual capability to pre-training data inclusion ("emphasis on the following languages...") with cross-lingual transfer to the English-dominated post-training phase (Section 5.4).

---

Throughput and Latency Analysis (Figures 3, 4)

Headline finding: Jamba models provide substantially better latency than comparably-sized Transformer models, with the advantage growing dramatically at long context lengths. Throughput differences are more modest but still favorable to Jamba at long contexts.

Jamba-1.5-Mini vs. comparables (Figure 3, 2×A100 80GB, batch size 1, 512 output tokens):

• Latency (Figure 3a): At 4,096 tokens, all four models (Jamba, LLaMA-3.1-8B, Mixtral-8x7B, Mistral Nemo 12B) have similar latency (~5–10 s/t). As context grows, the gap widens dramatically: at 131,072 tokens, Jamba requires roughly 25 s/t while Mixtral-8x7B and Mistral Nemo 12B require ~50 s/t, and LLaMA-3.1-8B requires ~60 s/t — a 2–2.4× latency advantage for Jamba. At 262,144 tokens, Jamba requires roughly 40 s/t vs. ~70 s/t for Mixtral and Mistral Nemo (~1.75× advantage); LLaMA-3.1-8B's line appears to end or be off-scale.

• Throughput (Figure 3b): At 4,096 tokens, Jamba has slightly lower throughput than the comparables (~40 t/s vs. ~50 t/s). As context grows, the throughput curves converge and then cross: beyond ~65,536 tokens, Jamba's throughput exceeds or matches the comparables, with Jamba achieving ~25 t/s at 262,144 tokens vs. ~15–20 t/s for the others. The paper acknowledges this as "only a slight reduction in output tokens throughput" at short contexts, with the advantage emerging at long contexts.

Jamba-1.5-Large vs. comparables (Figure 4, 8×A100 80GB, batch size 1, 512 output tokens):

• Latency (Figure 4a): At 4,096 tokens, Jamba-1.5-Large and LLaMA-3.1-70B have similar latency (~7 s/t), with Mistral-Large-2 slightly higher (~10 s/t). The gap widens progressively: at 131,072 tokens, Jamba is at ~40 s/t, Mistral-Large-2 at ~75 s/t, and LLaMA-3.1-70B at ~140 s/t — a 3.5× advantage over LLaMA and 1.9× over Mistral. At 262,144 tokens, Jamba reaches ~70 s/t, Mistral-Large-2 ~150 s/t (2.1× advantage), and LLaMA-3.1-405B (included as the largest available model) cannot be measured beyond 65,536 tokens because the model "is too large to fit context lengths greater than ≈100K tokens on 8 80GB GPUs" — and even at 65K, it requires ~45 s/t vs. Jamba's ~15 s/t, a 3× latency advantage.

• Throughput (Figure 4b): At short contexts, Jamba has slightly lower throughput than LLaMA-3.1-70B and Mistral-Large-2 (~35 t/s vs. ~40 t/s). The curves cross around 32,768 tokens, after which Jamba maintains higher throughput. At 262,144 tokens, Jamba achieves ~12 t/s vs. ~5 t/s for Mistral-Large-2, a 2.4× advantage. LLaMA-3.1-405B traces a line that ends at ~65K with throughput declining to ~20 t/s, already below Jamba's ~30 t/s at that length.

The key practical finding: The latency advantage is largest precisely where it matters most for long-context applications. A user waiting for a response to a 256K-token query on Jamba-1.5-Large experiences roughly 70 seconds of latency; on Mistral-Large-2 (the next-best option that can physically run), they'd wait ~150 seconds — more than double. On LLaMA-3.1-70B, they simply cannot run the query at all on the same hardware. This is not a marginal efficiency improvement — it's the difference between feasible and infeasible deployment.

---

Quantization Benchmarks (Figure 2)

Headline finding: ExpertsInt8 matches FP8 latency on H100 GPUs and substantially outperforms GPTQ on A100 GPUs, with negligible overhead compared to unquantized BF16 inference.

Jamba-1.5-Mini, 2×H100 (Figure 2a): ExpertsInt8, FP8, and GPTQ cluster together at batch sizes 1–15, with GPTQ showing slightly higher latency at larger batch sizes (e.g., batch size 10: ~10 s/t for all three; batch size 15: ExpertsInt8 and FP8 at ~12 s/t vs. GPTQ at ~15 s/t). The unquantized ("None") line is consistently lower latency (since it avoids the dequantization step) but is only feasible when models fit in memory without quantization.

Jamba-1.5-Mini, 2×A100 (Figure 2b): FP8 is unavailable on A100s, so only ExpertsInt8, GPTQ, and unquantized are compared. ExpertsInt8 substantially outperforms GPTQ across all batch sizes. At batch size 30, ExpertsInt8 achieves roughly 20 s/t vs. GPTQ's ~35 s/t — a 1.75× latency advantage. At batch size 50, the gap widens to roughly 35 s/t vs. 70 s/t — a 2× advantage.

Jamba-1.5-Large, 8×H100 (Figure 2c): ExpertsInt8 and FP8 produce indistinguishable latency curves across batch sizes 1–20, with both tracking slightly above the unquantized (BF16) baseline. At batch size 10, all three configurations deliver roughly 25–30 s/t. This validates the paper's claim that ExpertsInt8 "matches FP8 in latency" and adds "negligible overhead" compared to the unquantized baseline. GPTQ is not tested on the Large model (likely because calibration at this scale would be impractically expensive).

Mixtral cross-validation (Figures 2d, 2e): The technique is also tested on Mixtral-8x7B (2×H100) and Mixtral-8x22B (8×H100), showing that ExpertsInt8 works on standard MoE architectures beyond the Jamba family. On Mixtral-8x7B, ExpertsInt8, FP8, and unquantized cluster together with similar latency; on Mixtral-8x22B, the same pattern holds.

Practical significance: The quantization results are essential for the paper's deployment story. Jamba-1.5-Large's 398B total parameters in BF16 would require ~796 GB just for weights, exceeding the 640 GB total on 8×80GB GPUs. ExpertsInt8 cuts the MoE and MLP weights (~90% of parameters) roughly in half, saving approximately $0.9 \times 398\text{B} \times 0.5 \times 2\text{ bytes} \approx 358\text{GB}$, bringing total weight memory to roughly 440GB and leaving ~200GB for the KV cache, activations, and overhead — sufficient for 256K-token contexts.

---

Ablation Studies and Robustness Checks

Mamba-1 vs. Mamba-2 in hybrid architecture (Figure 1): At both 350M and 1.3B parameter scales trained for 100B tokens, Mamba-1-Attention outperforms Mamba-2-Attention, despite Mamba-2 outperforming Mamba-1 in isolation. This is the paper's only architectural ablation and provides the empirical justification for using Mamba-1 in Jamba-1.5-Large. The hybrid architecture (both variants) also outperforms pure Mamba-2, confirming that the addition of attention layers provides benefits beyond what a stronger standalone SSM can achieve. The ablation is limited to small scales (350M, 1.3B) rather than the 94B-active-parameter scale of the final model, so the conclusion that Mamba-1 is preferable at scale is an extrapolation.

Quantization technique comparison (Figure 2, discussed above): ExpertsInt8 vs. FP8 vs. GPTQ vs. unquantized is evaluated across multiple models and hardware configurations, establishing that the technique matches the best available alternative (FP8 on H100) and is the best available option where FP8 is unavailable (A100). The paper does not report quality (perplexity or benchmark accuracy) comparisons between quantization methods — only latency. This is a notable omission: without quality measurements, the claim that ExpertsInt8 causes "no loss of quality" (Section 3.1, Figure 2 caption) is unsubstantiated by the reported experiments.

Activation Loss effectiveness (Section 3.2): The α = 10⁻⁵ configuration reduced activation magnitudes from peaks of 4×10⁶ to an "acceptable range (2K-3K max)" — a reduction of roughly three orders of magnitude. The paper reports that "we ran our full evaluation suite on the model using FP16 activations and obtained the same results as the BF16 evaluations without any nans/overflows." This is a validation that the technique solved the numerical stability problem, but the "same results" are not quantified (no side-by-side BF16 vs. FP16 evaluation table). The claim that the auxiliary loss has "no affect on the training even with α values up to at least 10⁻³" is based on experimentation described qualitatively rather than through systematic learning curve or loss comparisons.

Post-training data mix impact (implicit, Sections 5.3–5.4): The paper does not present formal ablation studies varying the proportion of conversational, skill-specific, and long-context data in the post-training mix. However, the RULER and ∞BENCH results serve as an implicit validation that the mixing strategy worked — models retained long-context capabilities despite post-training on predominantly short examples. The observation about multilingual transfer (Section 5.4) similarly serves as an implicit ablation: even with "only a very small fraction of non-english data" in post-training, multilingual performance was competitive, suggesting cross-lingual transfer from pre-training survived the English-dominated SFT phase.

Negative result: ReST-like on-policy revision training (Appendix K, not in the main paper but cited in the prior summary): An attempt to optimize the model with on-policy RL-style training caused performance to degrade substantially, which the prior analysis discusses under the revision model results. This negative result is relevant here as it demonstrates that the "careful SFT" approach isn't trivially improvable through standard preference optimization, reinforcing the paper's methodological claim.

Missing ablations that would strengthen the paper:

1. Attention-to-Mamba ratio scaling: The paper uses the 1:7 ratio from the original Jamba [24] without ablating it at the 94B scale. Does the optimal ratio change with model size? Would 1:3 or 1:15 perform differently at this scale? Without this ablation, the claim that the ratio "was found optimal in our work on Jamba" applies only to the smaller-scale experiments in the predecessor paper.

2. MoE frequency and expert count: The paper uses e=2 (MoE every 2 layers) with n=16 experts and K=2, citing the original Jamba paper. There's no ablation varying these parameters at the Large scale to confirm they remain optimal.

3. Number of blocks vs. layers per block: The paper uses 9 blocks of 8 layers each (72 total). The effect of this specific decomposition (vs. 12 blocks of 6 layers, or 6 blocks of 12) on quality and efficiency is unexplored.

4. Mid-training ablation: The paper introduces mid-training as a distinct stage to emphasize long-range capabilities, but doesn't compare a model trained with vs. without this stage on long-context benchmarks. The contribution of mid-training to the final RULER performance is therefore unknown — it's possible the pre-training and post-training alone would have sufficed.

5. Quantization quality evaluation: ExpertsInt8 is claimed to have "no loss of quality" but this is never demonstrated through perplexity measurements, benchmark accuracy comparisons, or any other quality metric. The latency comparisons are thorough; the quality comparisons are absent.

---

Critical Assessment

Claim 1: "Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks."

This claim is strongly supported for long-context benchmarks, with qualifications for "excellent results" on standard benchmarks.

The long-context performance is genuinely excellent and well-demonstrated. Table 4 shows Jamba-1.5-Large as the only model maintaining >93% accuracy at 256K on RULER, with competitors either failing to reach that length (LLaMA-3.1-70B, Mistral-Large-2) or collapsing in accuracy (Gemini-1.5-pro at 65.1%). The ∞BENCH results (Table 5) corroborate with strong performance on naturalistic novel-length tasks. The throughput/latency analysis (Figures 3, 4) provides the mechanistic explanation: lower KV cache memory (Table 1) and the Mamba layers' linear scaling translate directly into better latency at long contexts. The evidence chain — architectural efficiency → memory savings → latency improvement → feasibility at 256K — is complete and internally consistent.

However, "excellent results" on standard benchmarks is an overstatement. Table 2 shows Jamba-1.5-Large trailing the best competitor on 7 of 10 clean academic benchmarks, with an average deficit of roughly 4–6 percentage points. This is "competitive" or "comparable" performance, not "excellent" relative to the state-of-the-art. The chatbot benchmarks (Table 3) are stronger — Jamba-1.5-Large leads LLaMA-3.1-70B on Arena-Hard by 9.7 points — but still trails Mistral-Large-2. The paper would be more accurate stating that Jamba matches or approaches the quality of similarly-sized models while providing substantial efficiency advantages, rather than claiming excellent standalone results. The efficiency-for-quality trade is clearly documented; the quality itself is good but not market-leading.

Claim 2: "Jamba-1.5-Large can fit on a single machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality."

This claim is supported for feasibility, but "without loss of quality" is not experimentally validated.

The feasibility argument is well-supported by three pieces of evidence: (a) Table 1 shows the KV cache is 9GB at 256K vs. 80–88GB for comparables, freeing sufficient memory for weights; (b) ExpertsInt8 (Section 3.1) compresses the dominant weight tensors by ~50%, with Figure 2 showing the quantized model runs efficiently; (c) the Activation Loss (Section 3.2) ensures activations stay within FP16 range, preventing numerical failures at inference time. Together, these make a convincing case that the model physically fits and runs.

The "without loss of quality" claim for quantization is asserted but not demonstrated. No perplexity, benchmark accuracy, or other quality comparison between quantized and unquantized inference is reported. The paper states this as a property of ExpertsInt8 ("without loss of quality" in the abstract, "matches FP8 in latency, while surpassing other quantization techniques, without a loss in quality" in Section 3.1), but provides zero evidence. Given that quantization-induced quality degradation is a well-documented phenomenon (GPTQ papers show measurable perplexity increases at 8-bit), this claim requires experimental support that is absent from the paper. The statement should be treated as aspirational until validated.

Claim 3: "Jamba-1.5 models are the only models with an effective length of 256K on the RULER benchmark."

This claim is strongly supported with evidence that includes direct testing of competing models.

Table 4 is the evidence: Jamba-1.5-Large at 93.9% (256K), Jamba-1.5-Mini at 86.1% (256K), and all other models either cannot be evaluated at 256K or fail (Gemini-1.5-pro at 65.1%). The paper even re-tested Gemini-1.5-pro after the originally reported RULER results (from an earlier model version) showed better performance, finding that the current version fails — this is unusually rigorous for a model release paper and strengthens the claim's credibility.

The one qualification: RULER is a synthetic benchmark. While it's the standard for long-context evaluation and includes diverse task types (needle-in-haystack variants, variable tracking, aggregation, QA), it doesn't fully capture naturalistic long-context understanding. The ∞BENCH results (Table 5) partially address this by showing strong performance on novel-length comprehension, but the 256K claim specifically references RULER. Readers should understand that "effective length of 256K on RULER" means the model can perform retrieval and reasoning tasks when information is distributed throughout a 256K-token context; it does not necessarily mean the model maintains coherent understanding of a 256K-token narrative or argument.

Claim 4: "ExpertsInt8 matches FP8 in latency, while surpassing other quantization techniques."

This claim is supported for latency on H100 GPUs; the "surpassing other techniques" comparison is limited to GPTQ, and quality comparisons are absent.

Figure 2 shows ExpertsInt8 and FP8 producing indistinguishable latency curves for Jamba-1.5-Large on 8×H100 (Figure 2c) and for Jamba-1.5-Mini on 2×H100 (Figure 2a). On A100 GPUs (where FP8 is unavailable), ExpertsInt8 substantially outperforms GPTQ (Figure 2b). The paper also notes ExpertsInt8's practical advantages: no calibration (seconds vs. hours/days for GPTQ), no instability during calibration, and BF16 activations (avoiding the numerical issues that prompted the Activation Loss). These combine to make a credible case that ExpertsInt8 is the preferred quantization technique for MoE models in vLLM on both H100 and A100 hardware.

The limitations: (1) The "surpassing other techniques" claim is based on comparison to GPTQ only — other quantization methods like AWQ, SmoothQuant, or NF4 are not evaluated. (2) The "no loss of quality" assertion is repeatedly made but never quantified. (3) The technique's applicability is tied to the fused_moe kernel in vLLM, meaning it only helps models served through that specific inference framework.

Genuine weaknesses in the experimental design:

• No statistical rigor: All benchmark results are single-run evaluations without confidence intervals, error bars, or multiple random seeds. On test sets of 500 questions (RULER, Arena-Hard, ∞BENCH), differences of 1–3 percentage points are within the range of sampling noise. The paper's conclusions rely heavily on the consistency of patterns across multiple benchmarks rather than statistical significance of individual results, which partially mitigates this concern but doesn't eliminate it.

• Small-scale architectural ablation extrapolated to large scale: The Mamba-1 vs. Mamba-2 comparison (Figure 1) is performed at 350M and 1.3B parameters. The conclusion that Mamba-1 is preferable is then applied to Jamba-1.5-Large at 94B active parameters — more than 70× larger than the largest ablation. The paper provides no evidence that the Mamba-1 > Mamba-2 result holds at scale, and the hypothesized mechanism (attention layers reduce the need for larger state sizes) could plausibly change at different scales.

• No quality evaluation of quantization: This is the paper's most significant experimental gap. The claim that ExpertsInt8 introduces "no loss of quality" appears in the abstract, the quantitation section header, and the running text multiple times, but is never supported by any measurement. Even a single perplexity comparison or a subset of benchmark evaluations comparing quantized vs. unquantized inference would have addressed this. Without such evidence, readers should treat "no loss of quality" as an assertion, not a finding.

• Single model family, single architecture: All results are specific to the Jamba hybrid architecture. While comparisons to LLaMA, Mistral, and Mixtral models demonstrate the architecture's advantages, there's no way to disentangle the effects of the architectural design from the effects of the training data, training infrastructure, or post-training pipeline. A controlled comparison training a Transformer baseline on the same data with the same compute budget would be needed to isolate the architectural contribution, but such a comparison is acknowledged as prohibitively expensive.

• Missing mid-training ablation: The paper introduces mid-training as a novel stage in the training pipeline but provides no evidence that it matters. The final model's long-context performance could be primarily due to the architecture and the inclusion of long-context examples in post-training, with mid-training contributing little. Without an ablation, the three-stage pipeline is presented as the recipe but the necessity of each stage is unproven.

• Limited reproducibility of training details: The paper does not disclose the data mixture proportions, training hyperparameters (learning rate, batch size, optimizer settings, training duration), or the exact composition of the synthetic data pipelines. The model weights are released (enabling inference-time reproducibility), but the training process is not reproducible from the paper alone. This is standard for industry model releases but limits the scientific value of the training methodology observations.

• Narrow scope of long-context evaluation: RULER and ∞BENCH are well-regarded benchmarks, but they don't capture all aspects of long-context understanding. There's no evaluation of long-context summarization, multi-document synthesis, long-range coreference, or sequential decision-making over long horizons. The claim of "effective context length of 256K" is validated for retrieval-style tasks but may not generalize to other long-context capabilities.

What experiments would have strengthened the paper:

1. A quality comparison (perplexity or benchmark accuracy) between unquantized BF16 and ExpertsInt8 quantized inference on a representative subset of benchmarks, to validate the "no loss of quality" claim.

2. An ablation of the mid-training stage (pre-training → post-training vs. pre-training → mid-training → post-training) on RULER performance, to quantify the contribution of this training stage to long-context retention.

3. A scaling study of the attention-to-Mamba ratio at larger model sizes (at minimum, testing 1:3 and 1:15 at a mid-scale like 5–10B active parameters) to validate that the 1:7 ratio remains optimal or to characterize how the optimal ratio changes with scale.

4. Error bars or multiple evaluation runs for the key benchmarks (especially RULER at 256K, Arena-Hard, and ∞BENCH where test sets are 500 examples or fewer), to help readers assess whether the reported differences are statistically meaningful.

5. A comparison of Jamba-1.5-Large with unquantized vs. quantized attention/SSM layers (not just MoE/MLP), even if only at a smaller scale, to characterize the quality impact of quantizing the non-MoE components — this would help others decide whether to extend ExpertsInt8 beyond its current scope.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Cost Is Unaccounted for in the Practical Deployment Story

The assumption or constraint. The entire compute-optimal test-time scaling framework depends on estimating prompt difficulty before allocating the inference budget. The paper's method for doing so — generating 2048 samples per question and computing either ground-truth pass@1 (oracle) or PRM-predicted final-answer correctness (model-based) — is extraordinarily expensive. Section 3.2 acknowledges this explicitly:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The paper frames this as an "exploration-exploitation tradeoff" and suggests future work on "training models to directly predict difficulty of a question" but develops no such method.

The consequence. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. At 2048 samples per question, the difficulty estimation step alone consumes more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty_estimation + strategy_execution, and the former could dominate. This means the 4× figure represents an upper bound on achievable efficiency — a theoretical best-case that assumes free difficulty labels. A practitioner deploying this system in production would need to either pay the estimation cost on every query (making the overall efficiency much worse than reported) or accept the inaccuracy of a cheaper difficulty proxy (potentially degrading the strategy selection and with it the gains). The paper provides no guidance on how to navigate this tradeoff.

What evidence exists in the paper. The difficulty estimation method is described in Sections 3.2 and Appendix C. The 2048-sample cost is stated. Figures 4 and 8 show that predicted (non-oracle) difficulty bins perform nearly as well as oracle bins — the curves "largely overlap" — which demonstrates that the quality of the difficulty signal is robust even when ground-truth labels are absent. But the cost of obtaining that signal is never accounted for in the budget calculations. The paper reports per-question accuracy vs. generation budget without adding the 2048-sample estimation cost to the x-axis.

Mitigation status. The paper explicitly acknowledges the problem (Section 3.2) and flags it as future work. No solution is developed or evaluated. The cross-validation protocol partially addresses the circularity concern (using difficulty to select strategy then evaluating that strategy) but does not address the cost concern. A practitioner reading this paper is left with a clear demonstration that difficulty-conditioned allocation could work if difficulty were known cheaply, but no actionable method for obtaining that knowledge at acceptable cost.

---

6.2 Hard Problems Are Unsolved — Test-Time Compute Cannot Compensate for Fundamental Capability Gaps

The assumption or constraint. The paper's central framework assumes that the base model already possesses the necessary knowledge and reasoning capability to solve a problem, and that test-time compute helps by improving the search or refinement process. Section 5.3's difficulty-bin analysis reveals that this assumption fails for the hardest problems. On difficulty bin 5 (lowest base model pass@1), no method makes meaningful progress regardless of budget.

The consequence. This is a hard capability ceiling: test-time compute amplifies existing capability but does not create it from nothing. If the base model's pass@1 on a problem class is near zero, no amount of search, revision, or compute-optimal allocation will help — there are no correct solutions in the proposal distribution to find or refine. For any deployment where the problem distribution includes a non-trivial fraction of bin-5-difficulty questions, the compute-optimal framework offers zero benefit, and the only path to improvement is scaling pretraining (training a larger model, training on more data, or both). This limitation is fundamental to the approach, not an engineering issue that can be optimized away.

The FLOPs-matched comparison (Section 7) quantifies this concretely: on hard problems (bins 4–5), test-time compute with the smaller model is worse than the ~14× larger pretrained model across all values of the inference-to-pretraining ratio R. At R ≫ 1, the disadvantage is -37.2% for revisions and -52.9% for PRM search (Figure 1 bar charts, Figure 9). This means that for hard problems, not only does test-time compute fail to help — it actually performs worse than simply using a larger model with greedy decoding.

What evidence exists in the paper. The evidence is clear and consistent across all experimental sections:

• Figure 3 (right): Bin 5 accuracy hovers at 1–3% for all methods (beam search, best-of-N, majority voting) across all budgets from 4 to 256 generations.
• Figure 7 (right): Bin 5 revision model accuracy sits at roughly 2–3% regardless of the sequential-to-parallel ratio.
• Figure 9: The bin 5 scaling line in the FLOPs-matched comparison is essentially flat near 0–5%, well below the ~14× larger model's performance (stars), confirming that no feasible test-time budget closes the gap.
• Section 7 takeaway: The paper is explicit that "test-time compute provides minimal gains on problems that are fundamentally outside the base model's capability range."

Mitigation status. The paper is transparent about this limitation. Section 7 states clearly that test-time and pretraining compute are "not 1-to-1 exchangeable" and that pretraining remains necessary for hard problems. The difficulty estimation framework itself serves as a partial mitigation: by identifying bin-5 questions early, the system can avoid wasting test-time compute on them and instead escalate to a larger model or flag for human review. However, this only helps with allocation — it does nothing to improve bin-5 performance itself. The paper does not suggest any method for extending the approach to genuinely out-of-distribution or capability-exceeding problems.

---

6.3 The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate and Degrades Under RL-Based Optimization

The assumption or constraint. The revision model is trained on offline-constructed trajectories where all in-context answers are incorrect, followed by a correct target answer (Section 6.1). This training procedure assumes that the model will always benefit from another revision step — it never sees examples where the current answer is already correct and should be preserved. At inference time, this creates a structural failure mode: the model has no signal for when to stop revising.

The consequence. Section 6.1 reports that approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step. This is a direct consequence of the training data construction: the model learned that in-context answers are incorrect and need to be changed, so even when it produces a correct answer, it treats it as something to modify. The mitigation — using majority voting or verifier-based selection across the entire revision chain rather than taking the final output — is a patch, not a solution. It recovers the lost correct answers by looking backward through the chain, but it introduces the risk of selecting suboptimal answers: if the model improves incrementally and the final answer is genuinely better than earlier ones, chain-level selection might pick an earlier, weaker answer.

This reversion problem fundamentally limits the efficiency of sequential revisions. Even if the model reaches the correct answer at step 3 of a 10-step chain, the remaining 7 steps are not just wasted compute — they actively risk degrading the answer, and the selection mechanism must correctly identify step 3 as the best. The 38% reversion rate means that roughly 4 out of every 10 correct answers are lost to over-revision, making the effective yield of a revision chain substantially lower than the per-step pass@1 trajectory suggests.

A secondary consequence appears in Appendix K (Figure 16), where the team attempted to further optimize the revision model using ReST^EM (an RL-based self-improvement method). This degraded performance substantially: "fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio" at 256 generations. The paper hypothesizes that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data." This means the revision approach is sensitive to training methodology in ways that are not fully understood — the positive results depend on specific choices (offline data construction, edit-distance-based pairing, supervised fine-tuning only) that may not transfer to other training paradigms.

What evidence exists in the paper.

• The 38% reversion rate is stated in Section 6.1 as an observed problem that motivated the within-chain selection mechanism.
• Figure 6 (left) shows the revision model's per-step pass@1 trajectory: accuracy improves from ~18.2% at step 1 to ~24–25% at steps 15–20, but the improvement plateaus and remains in the 23–25% range rather than continuing to climb. This plateau is consistent with the reversion problem — the model is simultaneously improving some answers and degrading others.
• The ReST^EM failure is documented in Appendix K, Figure 16, with a clear degradation in sequential revision performance.
• The paper does not provide direct measurement of the reversion rate per step (the 38% figure is an aggregate), nor does it analyze which types of problems are most susceptible to reversion.

Mitigation status. The paper acknowledges the problem and implements a partial mitigation (within-chain selection via majority voting or verifier), which Section 6.1 describes. However, this is acknowledged as a workaround rather than a fix. A more principled solution — such as training the model with correctness-dependent revision targets (e.g., training it to output "no revision needed" when the current answer is correct) — is not explored. The ReST^EM failure is presented as a cautionary negative result with no proposed solution. The paper's finding that SFT-only training works well while RL-based optimization backfires suggests that the revision training methodology is brittle and that scaling it further may require fundamentally different approaches to data construction.

---

6.4 Single Benchmark, Single Model Family — Generality of Difficulty-Dependent Findings Is Unestablished

The assumption or constraint. All experiments in the paper use a single benchmark (MATH, 500 test questions) with a single base model family (PaLM 2-S*). The paper explicitly states that it "believe[s] this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is not verified through experiments with other model families, architectures, or training procedures.

The consequence. Several aspects of the paper's findings could be model-specific or benchmark-specific in ways that limit their generality:

PRM quality and over-optimization behavior depend on the base model's output distribution. The PRM is trained on PaLM 2-S* outputs using Monte Carlo rollouts (Section 5.1, Appendix D). A model with different calibration properties, different error patterns, or different solution-step structure might produce PRM training data with different characteristics, leading to a verifier with different over-optimization thresholds. The difficulty bins where beam search helps vs. hurts (easy bins 1–2 vs. medium bins 3–4) might shift or invert for different base models.

The revision model's ability to learn from incorrect in-context examples depends on the base model's in-context learning capabilities. PaLM 2-S* has specific in-context learning behavior that enables the edit-distance-based pairing approach to produce useful training trajectories. A model family with weaker in-context learning (e.g., smaller models, models with different architectural inductive biases) might not benefit from the same revision training procedure, or might require different pairing strategies.

MATH consists exclusively of competition-level math problems requiring symbolic reasoning. It is unclear whether the difficulty-dependent patterns generalize to other reasoning domains. Code generation tasks might have different structural properties (executable verification is possible, pass@k estimates are cheaper to obtain). Logical reasoning tasks might involve different error patterns. Factual knowledge tasks might be dominated by memorization rather than chain-of-thought reasoning, making test-time search less effective. The paper provides no evidence about whether "beam search helps on medium problems but over-optimizes on easy ones" holds for, say, HumanEval or MMLU.

The test set of 500 questions, split into five difficulty quintiles of ~100 each, then further split by two-fold cross-validation, means the compute-optimal policy is selected based on ~50 questions per fold per bin. This is a small sample. A single difficult question that happens to fall in the "easy" bin due to sampling noise could substantially affect the estimated optimal strategy for that bin. The paper does not report confidence intervals on the compute-optimal scaling curves, making it difficult to assess whether the observed strategy rankings (e.g., "beam search outperforms best-of-N on bin 3") are statistically reliable at this sample size.

What evidence exists in the paper. The evidence is entirely within the MATH + PaLM 2-S* domain:

• All main results (Figures 3–9, Tables in Section 5) use MATH and PaLM 2-S*.
• The paper acknowledges the single-model limitation in Section 4, stating the representativeness belief but providing no cross-validation across model families.
• The difficulty bin cross-validation (Section 3.2) addresses overfitting within the MATH benchmark but does not address generalization to other benchmarks.
• There are no experiments with alternative model families, alternative architectures, or alternative reasoning benchmarks.

Mitigation status. The paper acknowledges the limitation implicitly (by stating the representativeness belief) but does not attempt to mitigate it. No experiments with other models or benchmarks are reported, even at small scale. The paper's claims about the generality of difficulty-dependent scaling behavior — which is its central intellectual contribution — rest on a single model-benchmark combination. A practitioner deploying compute-optimal test-time scaling on a different base model (e.g., LLaMA, Mistral) for a different task (e.g., code generation, scientific reasoning) would need to re-derive the difficulty bins, re-train the PRM, and re-evaluate which strategies work at which difficulty levels — the paper provides a methodology for doing so but no evidence that the conclusions (beam search helps medium problems, revisions help easy ones) transfer.

---

6.5 Sequential Revision Strategies Introduce Latency That Parallel Strategies Avoid

The assumption or constraint. The paper measures test-time compute in "generations" (number of complete solutions sampled), which is a reasonable proxy for total FLOPs but ignores wall-clock latency — the actual time a user waits for a response. Sequential revisions are inherently serial: each revision depends on the previous one's output, so a chain of S sequential revisions takes roughly S times longer in wall-clock time than a single generation, even if the total FLOPs are equivalent to S parallel samples.

The consequence. The compute-optimal policies derived in Sections 5.3 and 6.3 favor sequential strategies in several important regimes: on easy problems, pure sequential revisions dominate (Figure 7, right); at low-to-moderate budgets, higher sequential-to-parallel ratios are optimal (Figure 7, left). But these strategies incur a latency penalty that the paper's cost model does not capture. A practitioner choosing between "256 parallel samples" (all generated simultaneously) and "16 sequential chains of 16 revisions each" (requiring 16 serial steps) faces very different latency profiles even though both consume ~256 generations of FLOPs.

Specifically, the revision model's sequential chains require chain_length serial forward passes. Each forward pass processes the full prompt plus all previous revisions, meaning later steps in the chain have increasingly long input contexts, further increasing per-step latency. A deployment with 64 parallel samples might return a result in 5–10 seconds (the latency of one generation plus selection overhead). A deployment with 64 sequential revisions might require 30–60 seconds (the latency of 64 serial generations, each with growing context). For latency-sensitive applications — interactive assistants, real-time coding tools, customer-facing chatbots — the sequentially-heavy strategies that the compute-optimal policy selects may be completely impractical regardless of their accuracy advantages.

The paper's throughput analysis (Section 4, Figures 3, 4) measures tokens-per-second for batch-1 inference but does not analyze the latency implications of sequential vs. parallel sampling strategies. The latency numbers reported (e.g., Figure 3a showing ~25 s/t at 128K for Jamba-1.5-Mini) are for single-generation latency, not for the multi-generation strategies studied in Sections 5–7.

What evidence exists in the paper. The paper does not directly measure or discuss the latency implications of sequential vs. parallel test-time compute allocation. The throughput/latency analysis (Figures 3, 4) covers single-generation inference with varying context lengths, which is relevant for understanding the base model's efficiency but does not address the multi-generation strategies that are the paper's core contribution. The FLOPs-matched comparison (Section 7) accounts for total computational cost but not for wall-clock time.

Mitigation status. The paper does not address this limitation. It is not mentioned in the limitations discussion (Section 8 focuses on other future work) and is not accounted for in the cost model. The paper uses "generations" as the universal unit of compute (Section 3.1), which treats sequential and parallel generations as equivalent — a reasonable approximation for FLOPs but not for latency. A latency-aware extension of the compute-optimal framework — one that penalizes sequential depth or incorporates a latency budget alongside the generation budget — is not discussed but would be necessary for practical deployment in latency-sensitive settings.

---

6.6 The ~14× Larger Model Baseline Is Not Compute-Optimally Trained, and the Comparison Gives the Larger Model No Test-Time Compute

The assumption or constraint. The FLOPs-matched comparison in Section 7 scales model parameters by ~14× while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining where both parameters and data are scaled proportionally (Hoffmann et al., 2022). The paper acknowledges this:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the ~14× larger model is evaluated using only greedy decoding — no best-of-N, no majority voting, no search, no revision chains — while the smaller model is given the full benefit of compute-optimal test-time strategies.

The consequence. Both choices make the pretraining baseline weaker than it could be, inflating the apparent advantage of test-time compute.

Fixed-data scaling: A compute-optimally trained larger model (scaling both parameters and data according to Chinchilla scaling laws) would likely achieve higher quality per FLOP than a model that scales parameters only, because it avoids the diminishing returns of training a very large model on insufficient data. The paper's comparison therefore understates what pretraining could achieve with the same total FLOPs. The reported advantages of test-time compute over pretraining — e.g., +27.8% on easy-medium questions at R ≪ 1 for revisions (Figure 1, top-right bar chart) — may shrink or reverse against a properly compute-optimal larger model.

No test-time compute for the larger model: The comparison gives the smaller model sophisticated inference-time strategies but denies the larger model any inference-time optimization at all. A fairer comparison would allocate some test-time compute budget to both models. For example, giving the ~14× larger model best-of-8 or best-of-16 sampling would be a much stronger baseline, since the paper has already demonstrated that these strategies provide substantial gains (Figures 3, 4). The current comparison conflates "test-time compute helps" with "test-time compute on a small model beats a large model with no test-time compute" — these are different claims, and the paper only demonstrates the latter.

The consequence is that the FLOPs-matched results (Figure 9, the bar charts in Figure 1) should be interpreted as an upper bound on the advantage of test-time compute over pretraining. They demonstrate that test-time compute can be more efficient than naive pretraining scaling (parameters only, greedy decoding only), but they do not establish that it is more efficient than best-practice pretraining scaling with even a modest test-time budget for the larger model.

What evidence exists in the paper. The paper is transparent about the fixed-data training setup (Section 7) and explicitly acknowledges it as a departure from compute-optimal pretraining. The greedy decoding limitation for the larger model is not explicitly discussed as a limitation, but it is stated in the experimental setup: the larger model uses greedy decoding with no extra test-time compute. Figure 9 shows the larger model's performance as stars at three x-axis positions (corresponding to the three R values), making the comparison visually clear.

Mitigation status. The paper acknowledges the fixed-data limitation and flags it as future work. The no-test-time-compute-for-large-model asymmetry is not acknowledged as a limitation. A more informative comparison — giving both models a proportional test-time compute budget, or scaling the test-time budget with the total compute budget — is not performed. The paper's conclusions about the training-inference tradeoff should be understood as conditional on these experimental choices.

7. Implications and Future Directions

How This Work Changes the Landscape

Jamba-1.5 shifts the conversation around efficient architectures from a quality-per-parameter framing to a deployment-feasibility framing. The dominant narrative in the open-weight model ecosystem has been: larger models trained on more data achieve better benchmark scores, and architectural innovations are judged by whether they improve the quality-per-FLOP curve. Jamba-1.5 reframes the question: given that 256K-context applications exist and are growing, which models can actually run those applications on hardware that practitioners can afford?

This is a reframing rather than a paradigm shift. The individual components — Mamba SSMs, mixture-of-experts, grouped-query attention — are known techniques. The paper's contribution is demonstrating that their combination scales to 94B active parameters without hitting a quality cliff, and that the resulting efficiency gap versus pure Transformers widens at long contexts rather than narrowing. The RULER results (Table 4) make this concrete: LLaMA-3.1-70B drops from 96.5% at 4K to 66.6% at 128K and cannot reach 256K on 8×80GB GPUs; Jamba-1.5-Large maintains 93.9% at 256K with a near-flat scaling curve. This is not incremental — it's the difference between a model that degrades gracefully at long contexts and one that stops working entirely.

The paper also provides a concrete resolution to a looming tension in the field: post-training on short examples erodes long-context capabilities. Section 5.3 frames this as two "partly conflicting" objectives and demonstrates a practical solution — a dedicated mid-training stage with long documents, plus mixing long-context examples into the SFT data. The fact that this works (93.9% RULER at 256K after instruction tuning) is a methodological contribution that transfers to any architecture. Prior to this, the field had ample anecdotal evidence that instruction-tuned models lose long-context performance (the Gemini-1.5-pro RULER degradation from previously-reported higher scores to 65.1% at 256K, which the paper documents through its own testing, may be an instance of this), but no clear recipe for preventing it. Jamba-1.5 provides that recipe and validates it at scale.

The Mamba-1 vs. Mamba-2 finding (Figure 1) should redirect research attention within the SSM community. The default assumption — that better standalone components produce better hybrid architectures — is empirically false at the tested scales. Mamba-2 outperforms Mamba-1 in isolation, but Mamba-1-Attention outperforms Mamba-2-Attention. The paper's hypothesis — that attention layers reduce the burden on the Mamba state, making Mamba-2's larger state size less necessary — suggests that component selection for hybrid architectures should be guided by how components interact, not by standalone benchmarks. This has implications beyond Mamba: as the field explores other hybrid combinations (attention + linear attention, attention + RWKV, attention + retentive networks), the methodology of testing components both in isolation and in combination becomes essential, and the results of isolation tests may be misleading.

ExpertsInt8 makes a narrower but practically significant contribution: it demonstrates that quantization strategies should be architecture-aware and deployment-aware. The observation that over 90% of parameters are in MoE/MLP layers is structural to all MoE models, not specific to Jamba. The technique — quantize the dominant weight matrices statically, dequantize inside the existing fused kernel, keep activations in BF16 — can be applied to Mixtral, DBRX, or any other MoE architecture served through vLLM. The fact that it requires no calibration and takes seconds at load time removes a significant practical barrier (GPTQ's hours-to-days calibration) that has limited quantization adoption in production. The paper has contributed the modified kernel to vLLM, meaning the technique is immediately available to practitioners.

The paper's "careful SFT over RLHF" stance (Section 5.4) is a methodological challenge to the dominant post-training paradigm. It doesn't prove that preference optimization is unnecessary — the comparison isn't controlled — but it provides a strong existence proof: you can achieve Arena-Hard 65.4 (exceeding LLaMA-3.1-70B's 55.7), IFEval 81.5, and strong safety metrics using only supervised fine-tuning with carefully constructed synthetic data. This makes the "SFT is insufficient for alignment" claim an empirical question that depends on data quality, not a settled conclusion, and suggests that investment in data engineering (synthetic generation pipelines, automatic validation, rejection sampling) may have higher marginal returns than investment in algorithmic sophistication (PPO, DPO, RLHF variants) for post-training.

Follow-Up Research This Work Enables

Scaling the attention-to-Mamba ratio at larger model sizes. The paper uses the 1:7 ratio from the original Jamba [24] without re-ablating it at the 94B-active-parameter scale. The original ablation was performed at much smaller scales. A natural follow-up would train 5B–10B active parameter hybrid models with ratios of 1:3, 1:7, 1:15, and 1:31, measuring both short-context benchmark quality and long-context RULER scaling curves, to determine whether the optimal ratio shifts with model scale. The hypothesis to test: larger models have more capacity per layer, which might reduce the need for frequent attention (favoring higher ratios like 1:15 or 1:31) — or conversely, larger models might need more attention to coordinate their increased representational capacity (favoring lower ratios). The Jamba-1.5-Large results at 1:7 provide a single data point; mapping the curve would tell us whether 1:7 is near-optimal or just the first ratio that works.

Quality impact of ExpertsInt8 quantization. The paper asserts "no loss of quality" from ExpertsInt8 at least five times (abstract, Section 3.1 introductory text, Figure 2 caption, and running text) but provides zero measurements. A direct follow-up would evaluate Jamba-1.5-Large in both unquantized BF16 and ExpertsInt8-quantized configurations on a representative benchmark subset — at minimum, MMLU, RULER at 128K, Arena-Hard, and HumanEval. If the quality difference is genuinely within 0.5 percentage points across all benchmarks, the claim is validated and ExpertsInt8 becomes the default recommendation for MoE model serving. If there is a measurable degradation (even 1–2 points on some benchmarks), practitioners need to know the tradeoff they're making. This experiment is straightforward to run given the released weights and the open-sourced vLLM kernel. A deeper follow-up would compare ExpertsInt8 not just to unquantized but to GPTQ, AWQ, and FP8 on the same quality benchmarks, producing a latency-quality Pareto frontier for MoE quantization methods.

Probing the mechanism of long-context retention through post-training. The paper introduces mid-training as a novel stage but provides no ablation demonstrating its necessity. A controlled experiment would train three variants of a Jamba-scale model (perhaps at the Mini 12B-active scale for tractability): (A) pre-training → SFT (no mid-training, no long-context examples in SFT), (B) pre-training → SFT with long-context examples mixed in (no mid-training), (C) pre-training → mid-training → SFT with long-context examples (the full Jamba-1.5 recipe). Evaluating all three on RULER scaling curves would disentangle the contributions of mid-training from the contributions of including long-context data in SFT. If variant B performs nearly as well as C, mid-training is unnecessary and the recipe simplifies. If variant A performs well, even the long-context SFT mixing might be unnecessary — the architecture alone might be sufficient. This would tell the community which parts of the Jamba-1.5 training recipe are essential and which are incidental.

Mamba-1 vs. Mamba-2 in hybrid architectures at larger scales. Figure 1 shows Mamba-1-Attention outperforming Mamba-2-Attention at 350M and 1.3B parameters trained for 100B tokens. The extrapolation to 94B active parameters is 70× beyond the largest tested scale. A follow-up would train hybrid models at 5B–10B active parameters with Mamba-1 and Mamba-2 backbones, matched for total FLOPs, and measure both short-context quality and long-context RULER performance. The paper's hypothesis — that attention layers reduce the burden on the Mamba state, making Mamba-2's larger state size unnecessary — predicts that the gap between Mamba-1-Hybrid and Mamba-2-Hybrid should widen with scale (as attention layers become more capable, further reducing the value of the SSM state size). If instead the gap narrows or reverses, the hypothesis is wrong and the mechanism needs rethinking — which would be an important negative result for the SSM community.

Difficulty-conditioned allocation policies applied to the efficiency-quality tradeoff. The Jamba-1.5 models exhibit a specific difficulty-dependent behavior on RULER: some tasks (needle-in-haystack with single needles) are likely easy and could use less compute-intensive inference configurations, while others (multi-needle, variable tracking, aggregation) are harder and benefit from full attention resolution. A follow-up could train a lightweight classifier that predicts, from the first few thousand tokens of input, which RULER subtask a query corresponds to (or more generally, whether it requires precise token-level retrieval vs. global aggregation), and dynamically adjust the attention-to-Mamba computation ratio — for instance, by processing attention layers at full precision for hard-retrieval queries but using a cheaper configuration (fewer attention heads, lower precision, or even skipping some attention layers) for aggregation queries. This would extend the Jamba architecture from static efficiency to adaptive efficiency, following the philosophy of the compute-optimal test-time scaling framework (Section 3 of the prior analysis) but applied at the architectural rather than the sampling level.

Extending the hybrid architecture to modalities beyond text. The paper's evidence is entirely text-based (MATH reasoning, long-document QA, chatbot interactions). A natural extension would test whether the hybrid Transformer-Mamba design transfers to modalities where long sequences are even more critical: DNA sequence modeling (genomes are millions of base pairs), audio processing (hours of waveform at high sample rates), or video understanding (thousands of frames). The key question is whether the "attention as context refresher" role — which works well for text where relevant information is sparsely distributed — generalizes to modalities with different temporal structure (e.g., audio has local periodicity that Mamba alone might capture well, making attention less necessary; video has both local temporal coherence and long-range scene changes that might benefit from more frequent attention). A concrete experiment: train Jamba-style hybrid models and pure Transformer baselines on a long-range DNA task (e.g., Genomic Benchmarks at 100K+ base pair contexts) and measure whether the RULER-like flat scaling curve transfers.

Practical Applications and Downstream Use Cases

Single-machine long-context document analysis for legal and scientific review. The paper demonstrates that Jamba-1.5-Large can process 256K-token contexts on a single 8×80GB GPU machine with throughput of ~12 tokens/second (Figure 4b) and end-to-end latency of ~70 seconds for a 512-token response at full context length (Figure 4a). This enables a concrete deployment scenario: a legal firm or research institution can deploy a 94B-active-parameter model on a single on-premises server (costing roughly $100K–150K for the hardware) that can ingest entire legal case files, regulatory documents, or scientific literature reviews — documents that routinely exceed 100K tokens — and answer questions about them without sending sensitive data to cloud APIs and without requiring multi-node GPU clusters. The alternative (LLaMA-3.1-70B or Mistral-Large-2) would require either truncation (losing information), multi-node deployment (increasing infrastructure cost and complexity), or cloud API usage (introducing data privacy concerns). The 9GB KV cache at 256K (Table 1) versus 80–88GB for comparables is the enabling number: it means the model leaves ~200GB of GPU memory for weights and activations on an 8×80GB machine, making single-machine deployment feasible where competitors require distributed serving.

Cost-efficient batch inference on long-document corpora. For organizations that need to process large collections of long documents — government archives, patent databases, medical record systems — the throughput advantage at long contexts translates directly to cost savings. At 256K-token contexts, Jamba-1.5-Large achieves ~12 output tokens/second on 8×A100 GPUs versus 5 tokens/second for Mistral-Large-2 (Figure 4b) — a 2.4× throughput advantage. For a batch job processing 10,000 long documents with an average of 512 output tokens each, Jamba-1.5-Large would complete in roughly 100 GPU-hours versus 240 GPU-hours for Mistral-Large-2. At typical cloud GPU pricing ($3–4 per A100-hour), this saves$400–600 for a single batch job. More importantly, Jamba-1.5-Large can run the job at 256K context lengths on 8 GPUs where LLaMA-3.1-70B cannot run at all (Figure 4 caption: "too large to fit context lengths greater than ≈100K tokens on 8 80GB GPUs"), meaning there are entire classes of long-document processing tasks where Jamba is the only feasible open-weight option at this quality tier.

On-device or edge-deployed conversational agents with long memory. The Mini model's efficiency profile — 4GB KV cache at 256K (Table 1), ~40 seconds latency at 256K on just 2×A100 GPUs (Figure 3a), and strong chatbot performance (Arena-Hard 46.1, exceeding LLaMA-3.1-8B's 21.3 by a wide margin) — enables a deployment scenario that is impractical with Transformer models: a locally-hosted conversational agent that maintains a full 256K-token conversation history (potentially weeks of daily interaction) and responds with reasonable latency on consumer-adjacent hardware. While 2×A100 GPUs are still datacenter hardware, the trajectory is clear: the KV cache memory advantage (4GB vs. 32GB for LLaMA-3.1-8B at 256K) means the Mini model could plausibly run on a single high-end consumer GPU (e.g., RTX 4090 with 24GB) at contexts where Transformer models would overflow memory. This opens up privacy-preserving personal AI assistants that remember entire conversation histories without requiring cloud connectivity — a use case where the combination of strong long-context performance (RULER 86.1% at 256K) and modest hardware requirements is uniquely enabled by the hybrid architecture.

Quantization-based deployment of MoE models on A100 clusters. The ExpertsInt8 technique is immediately applicable to any organization running MoE models on A100 hardware, which is still the dominant GPU in cloud and academic clusters (H100 availability remains limited). Figure 2b shows that on A100s, ExpertsInt8 provides roughly a 1.75–2× latency improvement over GPTQ for Jamba-1.5-Mini at moderate batch sizes. For a team serving Mixtral 8x7B or Mixtral 8x22B (Figures 2d, 2e show the technique works on these architectures), switching from GPTQ to ExpertsInt8 in vLLM would cut per-query latency nearly in half at batch size 30, with the additional benefit of eliminating the multi-hour GPTQ calibration step that must be re-run whenever the model or quantization parameters change. The paper's contribution of the modified fused_moe kernel to vLLM (linked in the footnote to Section 3.1) means this switch is a configuration change, not a research project — practitioners can adopt it immediately. For organizations with large A100 fleets that cannot justify upgrading to H100s, this is a significant cost-free performance improvement.