Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context

ArXiv: 2605.13831

🎯 Pitch

Long-context vision-language models don't need target-length-focused data at 128K—balanced sequence-length mixtures actually teach better retrieval across all positions and generalize to 2×–4× the training context without extra training. A pure diet of instruction-formatted long-document VQA with 80% retrieval tasks preserves short-context performance entirely, overturning the assumption that you must mix in short data. This recipe, distilled into the MMProLong model, required only a 5B-token budget and transferred zero-shot to webpage retrieval, video understanding, and compression.

---

1. Executive Summary

This paper presents a systematic empirical study of long-context continued pre-training (LongPT) for large vision-language models, extending Qwen2.5-VL-7B from 32K to 128K context using the MMLongBench document VQA benchmark suite. The work identifies long-document VQA as the primary effective supervision source—substantially outperforming OCR transcription baselines—and derives a practical recipe through ablations on three key design axes: sequence-length distribution (establishing that balanced, pool-native sampling beats target-length-concentrated data), task mixture ratio (finding that retrieval-heavy extraction-to-reasoning blends at 8:2 work best), and short-context data mixing (showing that pure long-document VQA largely preserves short-context capabilities without dedicated short-data mixing). The resulting model, MMProLong, improves long-document VQA scores by 7.1% over the base model at 64K and 128K contexts, generalizes beyond its training window to 256K and 512K without additional adaptation, and transfers to webpage-based needle retrieval, vision-text compression, and long-video understanding, establishing that instruction-formatted long-document VQA data teaches generalizable retrieval and reasoning across diverse multimodal long-context tasks while requiring only a modest 5B-token training budget.

2. Context and Motivation

The Core Problem: We Don't Know How to Train Long-Context Vision-Language Models

The central question this paper tackles is practical and pressing: when you want to extend a vision-language model's context window from 32K to 128K tokens, what data should you use, and how should you mix it? This matters because long-context capability has rapidly become table stakes for modern LVLMs—models without it cannot process long documents, hour-long videos, or sustained agent interactions—yet the published recipes for building this capability are surprisingly thin.

The paper frames this gap explicitly in Section 1:

"recent technical reports provide only limited details on the use of long-document data, leaving practical recipes for developing long-context vision-language models insufficiently explored. It remains unclear which types of long-context data to synthesize, how to mix different long-context tasks and incorporate short-context data, and how training choices such as length distributions affect the resulting model."

This is not a gap in the existence of long-context models—Gemini 3.1 Pro, Claude Sonnet 4.7, and Qwen3-VL all support 128K+ contexts. Rather, it is a gap in knowing how to replicate that capability. The technical reports for these models (cited in Section 2) provide only high-level descriptions without the granular ablations needed to guide practitioners who want to build their own long-context LVLMs or extend existing ones.

Why This Problem Matters: The Stakes of Underexplored Recipes

The paper identifies several concrete application domains that depend on long-context LVLMs (Section 1):

• Multi-hop reasoning over document collections — for example, answering a question that requires synthesizing evidence from a 50-page financial report spread across tables, text, and figures
• Spatiotemporal reasoning over hour-long videos — tracking events, objects, and causal chains across thousands of frames
• Context consistency in long-horizon agent tasks — maintaining state and memory across extended tool-use workflows where the model interacts with web pages, documents, and visual interfaces

Without reliable long-context capability, models fail on these tasks not because they lack reasoning ability, but because they literally cannot see enough of the input at once. The practical consequence is that organizations deploying LVLMs must either (a) use proprietary models with opaque training recipes, (b) attempt ad-hoc context extension with no guarantee of quality, or (c) invest in expensive trial-and-error data construction. This paper addresses the knowledge gap that makes option (b) risky and option (c) inefficient.

Where Prior Approaches Fall Short

The paper identifies several categories of prior work and explains their limitations relative to the specific problem of training long-context LVLMs:

1. LLM long-context training recipes don't transfer directly. The LLM community has established effective recipes for extending context windows—notably Gao et al. (2024) and Fu et al. (2024), both cited in Section 5.1 and the related work discussion. These works use text-only corpora like books (from SlimPajama) and code repositories (from the Stack) where sequence lengths naturally span 8K to 128K tokens. However, LVLMs introduce a fundamentally different challenge: their inputs interleave image and text tokens. A 128K-token LVLM sequence might contain dozens of rendered document pages, each producing hundreds or thousands of visual tokens through the vision encoder. The positional encoding structure is also different—Qwen2.5-VL uses mRoPE (multimodal Rotary Position Embedding), which decomposes position into temporal, height, and width components rather than a flat 1-D positional index. This means RoPE scaling heuristics developed for LLMs (like Dynamic-NTK, Section 3 and Appendix G.3) don't necessarily apply in the same way. The paper explicitly tests this in Appendix G.3, finding that the Dynamic-NTK scaling factor ($t^{d/(d-2)}$ where $t$ is the expansion factor) produces reasonable but not uniquely optimal results for mRoPE, with moderately scaled bases ($2 \times 10^6$ and $4 \times 10^6$) performing comparably.

2. Concurrent LVLM long-context work studies alignment, not extension. The paper distinguishes its contribution from Veselka (2026), a concurrent work cited in Section 2 that also studies long-document data construction for LVLMs. The critical difference is that Veselka (2026) builds on backbones that already support 128K or longer contexts (Qwen3-VL and Mistral 3.1). As a result, its findings—such as 1B-token LongPT outperforming 10B-token LongPT, and LongSFT outperforming LongPT—reflect context alignment (tuning a model to better use its existing long context) rather than context extension (genuinely expanding the window). The paper argues:

"its findings may reflect context alignment rather than true context extension... In contrast, we study LongPT on Qwen2.5-VL, whose native context window is only 32K, allowing us to directly examine how to extend LVLMs to longer context."

This is an important methodological distinction. A model with a native 128K context window already has positional encodings and attention patterns capable of handling that length; additional training mainly improves how it uses that capacity. Starting from 32K, the model must learn genuinely new positional regimes and long-range attention behaviors. The paper's choice of Qwen2.5-VL-7B as the base model therefore makes its findings more directly applicable to the harder problem of context extension.

3. Long-video methods are specialized, not general. Another line of work (Section 2) addresses long-context video understanding through temporal redundancy reduction and video token compression (LongVILA, Long-VITA, LongVU, BOLT, etc.). These methods are effective for video but are specialized—they exploit temporal smoothness and frame-to-frame redundancy that don't exist in document collections or webpage sequences. The paper positions its work as complementary: rather than developing a domain-specific compression or attention mechanism, it studies general-purpose training data and mixture design that should produce transferable long-context capability.

4. Existing multimodal long-context benchmarks exist but haven't been used to guide training. The paper notes (Section 2) that benchmarks like MMLongBench, MM-NIAH, VTCBench, and long-video suites (Video-MME, MLVU, LongVideoBench) provide thorough evaluation coverage, but they are used for assessment, not for training data design. The paper's contribution is to use these evaluation perspectives—retrieval, reasoning, counting, memory—to inform what training tasks to construct, then validate that the resulting model indeed transfers to these diverse evaluations.

How This Paper Positions Itself

The paper positions itself as a systematic empirical study that fills the recipe gap between knowing that long-context LVLMs are possible and knowing how to build one. It is not proposing a novel architecture or a new training objective. Instead, it treats training data design as the primary research object and conducts controlled ablations to answer specific, practical questions:

• What type of supervision is effective? (Section 4: long-document VQA vs. OCR transcription)
• What should the length distribution of training examples look like? (Section 5.1: balanced vs. long-biased)
• How should different long-context tasks be mixed? (Section 5.2: extraction vs. reasoning ratios)
• Do we need to mix in short-context data to prevent forgetting? (Section 5.3: pure long vs. blended)

Each ablation is conducted under a fixed 5B-token budget, making comparisons fair and practically grounded—the question is not "what improves performance if we throw more compute at it?" but rather "given a fixed, modest training budget, how should we allocate it across data design choices?"

The paper also positions its findings as testable and transferable. The final recipe is validated not just on long-document VQA (the training domain) but also on MM-NIAH (webpage haystacks), VTCBench (vision-text compression), and three long-video benchmarks (Video-MME, MLVU, LongVideoBench). The transfer to video is particularly notable because the training data contains no video examples—it is purely document-based. This suggests that long-document VQA teaches a general capability (evidence retrieval and reasoning over long multimodal contexts) rather than a domain-specific skill. Furthermore, Appendix G.7 validates the recipe on Qwen3-VL-8B, showing it is not specific to the Qwen2.5-VL backbone.

A subtle but important positioning choice: the paper explicitly limits its main experiments to the 7B/8B parameter scale. Section H (Limitations) acknowledges this:

"our systematic study is primarily conducted on 7B/8B-scale LVLMs. This choice enables controlled comparisons across data recipes, context lengths, and training budgets, but it also leaves open how the observed trends scale to substantially larger models."

This is a deliberate tradeoff: thoroughness at one scale over superficial coverage at many scales. The paper doesn't claim its exact mixture ratios (8:2 extraction-to-reasoning, for example) will be optimal at 70B parameters. It claims to have identified qualitative principles—balanced length distributions beat target-concentrated ones, retrieval is the primary bottleneck, instruction-formatted long data preserves short-context ability—that should inform training design at any scale, with the quantitative specifics subject to further scaling studies.

3. Technical Approach

3.1 Reader Orientation

This paper builds a training recipe—a prescriptive set of choices about what data to create and how to mix it—that takes an existing 7B-parameter vision-language model with a 32K-token context window and extends its effective context to 128K tokens through a modest 5B-token continued pre-training stage. The core insight is that long-document VQA (asking the model questions that require retrieving and reasoning over evidence buried in 32-128K token multimodal documents) provides substantially more effective supervision than alternative tasks like OCR transcription, and that careful choices about sequence-length distribution, task mixture ratios, and short-context data preservation can yield strong long-context performance that generalizes beyond both the training length and the training domain.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, connected in a data-centric pipeline:

1. Document Pool — a corpus of over 1.5 million PDF documents spanning diverse domains (engineering, medicine, social sciences, biology, etc.) that provides raw multimodal material for synthesizing training examples. Documents are rendered to images and parsed with an OCR expert to extract structural metadata (section boundaries, text blocks).

2. Long-Document VQA Synthesis Pipeline — a short-to-long generation procedure that (a) samples a coherent 8-15 page segment from a document, (b) sends it to a strong teacher LVLM (Seed 2.0) to generate a question-answer pair, and (c) re-embeds that QA pair into the original full document to create a long-context training instance where the answer is locally grounded but the model must process the full long context.

3. OCR Transcription Data (baseline comparison) — an alternative supervision source where the model must transcribe text from rendered document pages, used to establish that instruction-formatted VQA supervision is more effective than dense image-text alignment tasks.

4. LongPT Training Configuration — the continued pre-training stage itself, defined by hyperparameters including mRoPE base frequency scaling (from $1 \times 10^6$ to $4 \times 10^6$ via Dynamic-NTK), a 5B-token budget, 128K maximum sequence length, 4M-token global batch size, and FlashAttention with sequence parallelism for efficient training on 64 H20 GPUs.

5. Evaluation Suite — five downstream benchmarks used to assess the trained model: MMLongBench document VQA (at 64K, 128K, 256K, and 512K), MM-NIAH (webpage needle-in-a-haystack), VTCBench (long-context vision-text compression), and three long-video benchmarks (Video-MME, MLVU, LongVideoBench), all evaluated without any task-specific fine-tuning.

Information flows as follows: documents are sampled from the pool → the OCR expert parses structural metadata → the synthesis pipeline generates VQA pairs from short segments → full-document training instances are constructed → the base Qwen2.5-VL-7B model undergoes LongPT for 5B tokens → the resulting checkpoint (MMProLong) is evaluated zero-shot on all downstream benchmarks.

3.3 Roadmap for the Deep Dive

• First, the base model and training configuration (Section 3 from the paper, plus Appendices A.2 and G.3), covering what Qwen2.5-VL-7B is, how mRoPE works, why RoPE base frequency scaling matters for LVLMs, and the concrete hyperparameters used for all LongPT runs. This establishes the "canvas" on which all subsequent data-design experiments are painted.

• Second, the document pool and OCR expert (Section 4.1 and Appendix C), covering corpus construction, document-to-image rendering, layout parsing, and why structural metadata is necessary for the synthesis pipeline. Without this step, the synthesis pipeline cannot sample semantically coherent segments.

• Third, the long-document VQA synthesis pipeline (Section 4.2 and Appendix D), the paper's central data-generation contribution. We walk through segment sampling, QA generation with explicit anchoring constraints, and the three training tasks (extract-single, extract-multi, reasoning) with their specific prompt templates and evidence requirements.

• Fourth, the OCR transcription baseline (Section 4.3 and Appendix E), including full-document and needle-page variants, to understand what alternative supervision sources look like and why they underperform.

• Fifth, the three data-design ablations (Section 5), the empirical core of the paper: sequence-length distribution (pool-native vs. long-biased), task mixture ratios (extraction vs. reasoning grid search), and short-context data mixing (0%–80% ratios). We explain not just the results but the why behind each finding.

• Sixth, the final recipe and evaluation protocol (Appendices A.1 and B), covering the exact configuration of MMProLong, how evaluation is conducted at each context length, and how the LLM-judged scoring protocol handles different answer formats.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical data-design paper whose core idea is that the optimal way to train long-context vision-language models is to use instruction-formatted long-document VQA data with balanced sequence-length distributions, retrieval-heavy task mixtures, and minimal short-context data mixing. The key methodological contribution is the short-to-long synthesis pipeline that generates high-quality document-grounded QA training data without requiring the teacher model to process full 128K-token contexts.

---

Base Model and Training Configuration

What is Qwen2.5-VL-7B and why start from 32K? The paper selects Qwen2.5-VL-7B-Instruct as its base model (Section 3). This is a 7-billion-parameter vision-language model from the Qwen family whose native context window is 32,768 tokens (32K). The paper's choice is deliberate: starting from 32K means the model has never seen positional encodings beyond this range, so any improvement at 64K, 128K, 256K, or 512K reflects genuine context-window extension rather than better utilization of pre-existing long-context capacity. This distinguishes the paper from concurrent work (Veselka, 2026) that studies long-context training on backbones already supporting 128K+ contexts, where gains may reflect alignment rather than true extension.

mRoPE: why LVLM positional encoding is more complex than LLM RoPE. Qwen2.5-VL uses mRoPE (multimodal Rotary Position Embedding), an extension of the standard Rotary Position Embedding (RoPE) mechanism. In standard 1-D RoPE used by LLMs, each token position is represented by a single scalar index, and the rotary embedding applies a position-dependent rotation to the query and key vectors in attention. The key property that enables context-window extension is that RoPE encodes relative positions through the dot product of rotated vectors: the attention score between two tokens depends on their positional difference, not their absolute positions.

mRoPE decomposes this differently. Instead of a single 1-D position index, it splits the embedding dimensions into three groups and applies separate rotary embeddings to each, encoding:

• Temporal position — the sequential order of tokens in the flattened multimodal sequence
• Height position — the vertical coordinate of a visual token within its source image
• Width position — the horizontal coordinate of a visual token within its source image

This means that a visual token's position is represented in a 3-D space, not a 1-D space. This has an important consequence for context-window extension: visual position indices grow much more slowly than a flattened 1-D sequence index, because multiple visual tokens from a single image share similar height/width coordinates but different temporal positions. The paper notes this explicitly in Appendix G.3:

"visual position indices grow more slowly than a flattened 1-D sequence. It is therefore unclear whether the RoPE-scaling heuristic from Dynamic-NTK, originally developed for LLMs, directly applies to LVLMs with mRoPE."

Dynamic-NTK RoPE base frequency scaling. The standard approach for extending an LLM's context window is to increase the RoPE base frequency, a hyperparameter that controls the wavelength of the rotary embeddings. The Dynamic-NTK heuristic (Section 3 and Appendix G.3) prescribes scaling the base frequency by:

$\theta_{\text{new}} = \theta_{\text{old}} \times t^{d/(d-2)}$

where $\theta_{\text{old}}$ is the original RoPE base frequency (which for Qwen2.5-VL-7B's mRoPE is $1 \times 10^6$), $t$ is the context-window expansion factor (128K $\div$ 32K = 4), and $d$ is the attention head dimension (128 for Qwen2.5-VL-7B).

What it computes: the new base frequency is the original base multiplied by the expansion factor raised to a model-specific exponent. For Qwen2.5-VL-7B, plugging in $t = 4$ and $d = 128$ gives $\theta_{\text{new}} \approx 1 \times 10^6 \times 4^{128/126} \approx 1 \times 10^6 \times 4.063 \approx 4.06 \times 10^6$. The paper rounds this to $4 \times 10^6$.

Why this form: increasing the base frequency effectively "compresses" the rotary embeddings—positions that were previously far apart become, in terms of the dot-product attention score, closer together. This allows the model to extrapolate to longer sequences because newly encountered long-range positional relationships are mapped into a range the model has already learned to handle. The exponent $d/(d-2)$ comes from the theoretical analysis that the wavelength of the highest-frequency RoPE component should be scaled to match the new maximum sequence length. Without this scaling, positions beyond the training window would produce rotation angles the model has never seen, causing attention patterns to degenerate.

Ablation of mRoPE base frequency (Appendix G.3). The paper tests three values for the new base frequency: $2 \times 10^6$, $4 \times 10^6$ (the Dynamic-NTK default), and $8 \times 10^6$. Table 16 shows that $2 \times 10^6$ and $4 \times 10^6$ achieve comparable overall performance (e.g., 55.63 vs. 56.90 for extract-multi), while $8 \times 10^6$ does not yield consistent gains and sometimes degrades performance. The paper concludes:

"moderate mRoPE-base scaling is sufficient for extending LVLMs to longer contexts, while overly aggressive scaling is unnecessary."

This is an important empirical finding specific to LVLMs with mRoPE: the Dynamic-NTK heuristic developed for LLMs works reasonably well but is not uniquely optimal, and the relationship between base frequency and performance is not monotonic.

Training hyperparameters (Section 3 and Appendix A.2). Every LongPT run uses a fixed configuration to ensure fair comparisons across data-design ablations:

• Token budget: 5B tokens total. This is fixed across all experiments—when comparing pool-native vs. long-biased distributions or different extraction-to-reasoning ratios, the token count stays constant so no experiment gains an unfair advantage from seeing more data.
• Maximum sequence length: 131,072 tokens, which is 128K tokens (using binary prefixes where K = $2^{10}$, M = $2^{20}$, B = $2^{30}$). This is the target context window after extension.
• Global batch size: 4M tokens, corresponding to approximately 32 sequences per update (since 4M $\div$ 131,072 $\approx$ 31).
• Optimizer: AdamW with weight decay 0.1, $\beta_1 = 0.9$, $\beta_2 = 0.95$
• Learning rate schedule: peak learning rate $1 \times 10^{-5}$, with 10% linear warmup (so the first 500M tokens use linearly increasing LR) and cosine decay to $1 \times 10^{-6}$
• Infrastructure: VeOmni framework with FlashAttention-2 for efficient attention computation; Ulysses sequence parallelism size 2 (splits the sequence dimension across GPUs) and FSDP size 4 (shards model parameters); trained on 8 H20 nodes with 8 GPUs each (64 GPUs total)

Why VeOmni and FlashAttention? Training on 128K-token multimodal sequences is computationally expensive because the attention computation scales quadratically with sequence length. FlashAttention-2 reduces this by fusing the attention computation into a single GPU kernel that minimizes memory reads/writes, making it possible to fit 128K sequences in GPU memory. Ulysses sequence parallelism further distributes the long sequence across multiple GPUs, while FSDP distributes the model parameters, enabling the 7B model to train on 128K sequences without running out of memory.

---

Document Pool Construction and OCR Expert

Why documents? The paper identifies documents as a natural source for synthesizing multimodal long-context training data because (Section 4.1):

"each page combines rich visual layout with dense textual content and can be rendered into long multimodal sequences."

Unlike synthetic long-context data (which might just concatenate unrelated images with filler text), documents provide realistic long-range dependencies: a table on page 3 may be referenced in a paragraph on page 27; a figure on page 12 may illustrate a concept defined in text on page 8; section headers provide hierarchical structure that connects content across dozens of pages. These dependencies are what make long-context capability non-trivial—the model can't just attend to the nearest context; it must learn to retrieve information from arbitrarily distant positions in the sequence.

Pool construction and scale (Appendix C.1). The document pool comprises over 1.5 million PDF documents spanning a broad range:

• Document types: academic papers, books, and technical manuals
• Domains: engineering (12.7%), medicine (12.0%), social science (9.4%), biology (7.2%), economics & business (7.2%), physics (7.0%), computer science (6.8%), earth science (5.7%), mathematics (4.6%), language & literature (3.5%), and others (24.0%)
• Language distribution: 96.22% English, 3.59% Chinese, 0.19% other languages
• Page count range: 20–200 pages per document
• Average pages per document: 23.80
• Total pages across pool: 36,592,809

Filtering for evaluation contamination. To avoid training on evaluation data, the paper filters out documents whose SHA-256 hash of the PDF content matches any evaluation benchmark document. This is a cryptographic hash comparison—if two PDFs have identical content, they will produce the same SHA-256 hash, even if filenames or metadata differ. This is standard practice in benchmark-driven ML but is worth noting because long-document benchmarks (MMLongBench-Doc, LongDocURL) draw from real document sources that might overlap with the training pool.

Document rendering. Since LVLMs operate on images, not PDF files, each page is rendered to an image using PyMuPDF at DPI = 144 (Section 4.1). The choice of 144 DPI represents a practical tradeoff: higher DPI (e.g., 300) would preserve more visual detail but produce larger images and therefore more visual tokens, eating into the 128K-token budget for shorter documents. Lower DPI would be more token-efficient but might lose fine-grained visual information (small text, detailed figures, table structure) that the model needs to answer questions.

OCR expert model. After rendering, each page is processed by an OCR expert model fine-tuned from Seed 2.0 (ByteDance's proprietary multimodal model). This model does two things:

1. Recognizes text content in the rendered page image—the actual words, numbers, and symbols
2. Assigns structural labels to each text block: title, section heading, paragraph, table, figure caption, header, footer

This structural metadata is crucial for both the VQA synthesis pipeline (Section 4.2) and the OCR transcription baseline (Section 4.3). For VQA, the title and section labels enable the system to identify semantically coherent page segments—instead of randomly sampling 8-15 consecutive pages that might cut mid-section, the system can sample complete sections with clear topical boundaries. For OCR transcription, the recognized text provides the ground-truth target that the model is trained to generate.

Why an OCR expert rather than the base LVLM? The paper does not explicitly address this design choice, but the rationale is straightforward: the base LVLM (Qwen2.5-VL-7B) is being trained--the OCR expert is a fixed, frozen component that provides clean structural annotations. Using the base model to parse its own training data would introduce a feedback loop where errors in parsing compound during training. A dedicated OCR expert, fine-tuned specifically for document structure parsing, provides higher-quality, more consistent annotations.

---

Long-Document VQA Synthesis Pipeline

This is the paper's central data-generation contribution (Section 4.2, Figure 1, Appendix D). The key challenge it solves is:

How do you generate high-quality question-answer pairs for 128K-token multimodal documents when the teacher model itself can only process short contexts?

The answer is the short-to-long synthesis pipeline: generate the QA pair from a short segment that the teacher model can handle, then embed it back into the full document to form the long-context training instance. The model being trained must process the entire 128K context, but the QA pair only needs to be correct and unambiguous within that context.

Step 1: Segment sampling. For each document in the training pool (filtered to documents with 32–50 pages, which produce sequences in the 32K–128K token range after rendering), the pipeline:

• Identifies the document's section structure using OCR block labels (title and section)
• Randomly samples one or more consecutive sections whose total length spans 8–15 pages
• This produces a semantically coherent segment—the content within the segment belongs to the same topical section(s) of the document

Why 8–15 pages? This range is short enough that the teacher LVLM (Seed 2.0) can process all the page images simultaneously in its context window, ensuring high-quality QA generation. The window needs to be long enough that the QA pair can involve meaningful document structure (cross-page references, section-level context) but short enough to fit within the teacher model's context. The upper bound of 15 pages is an empirical choice balancing these constraints.

Why use section boundaries for sampling? Randomly cutting a document at arbitrary page boundaries would produce segments that might start mid-sentence or mid-argument, making it harder for the QA generator to produce coherent questions. Sampling at section boundaries ensures the segment has clean topical coherence—the QA generator sees the complete introduction to a topic, not a fragment.

Step 2: QA generation with anchoring constraints. The sampled page images are sent to the teacher LVLM (Seed 2.0) along with a detailed prompt (Appendix D.3, Table 12) that instructs it to:

• Generate a single question-answer pair grounded in the provided segment
• Explicitly categorize evidence into four types: Text (pure paragraphs), Layout (titles, headers, captions), Figure (charts, images), Table (structured data)
• Specify the evidence pages and evidence sources
• Produce the answer in one of four formats: String, Integer, Float, or List
• Critically: add explicit segment anchors to the question (e.g., "in the Introduction section," "on Pages 20-25," "according to the table on Page 13")

Why are anchors necessary? This is a subtle but crucial design choice. The QA pair is generated from a short segment where context is unambiguous—there's only one "Introduction" or one "Table 3" visible to the generator. But when the QA pair is placed back into the full 128K document, ambiguities can arise. A question like "What is the reported revenue?" might be answerable within the sampled section but ambiguous in a full financial report where multiple sections report different revenue figures for different divisions or time periods. The anchors resolve this by explicitly scoping the question to the specific section or pages where the evidence resides, preventing global-context false positives where a locally valid question becomes ambiguous when the full document is visible.

Why four evidence types and four answer formats? The evidence type classification serves as metadata that helps organize the generated data and potentially enables analysis of which evidence types are most challenging. The four answer formats cover the range of typical document VQA answers: text strings for entity names and descriptions, integers for counts and years, floats for measurements and percentages, and lists for multi-item answers. This format diversity ensures the training data covers different generation patterns.

Step 3: Full-document instance construction. Once the QA pair is generated from the segment, the pipeline:

• Recovers the original full document (all pages, including those outside the sampled segment)
• Combines it with the generated QA pair
• The result is a single training instance where the model receives all document pages as visual input plus the question as text input, and must generate the answer

The key property of this instance is that the answer is locally grounded but globally embedded: the evidence needed to answer the question exists within a specific segment of the document, but the model must identify which part of the long input is relevant. This is exactly the capability that long-context models need: the ability to locate and use sparse evidence within large contexts.

Data quality and efficiency (Section 4.2). The paper makes two claims about this pipeline:

"we find that the generated QA pairs are of high quality"

This is supported by a manual verification of 100 randomly sampled QA pairs across all three task types (Appendix D.4): 97 are fully correct, 2 contain incorrect answers, and 1 has an inaccurate evidence annotation. This 97% correctness rate suggests the pipeline produces reliable supervision with only a small amount of noise.

"this pipeline is also efficient, substantially reducing the cost of generating large-scale data"

This efficiency comes from the teacher model only needing to process 8–15 pages at a time, not the full 32–50 page document. If the teacher model had to generate QA pairs from full documents, the generation cost would scale with document length and might require a teacher model with a larger context window than Seed 2.0 supports. The short-to-long approach decouples generation cost from training instance length.

The three training tasks. Using this pipeline with different task-specific prompts (Table 13), the paper constructs three types of long-document VQA data, ordered by increasing evidence complexity:

1. Single-page extraction (extract-single): Questions that can be answered from a single page. The extra restriction in Table 13 is explicit: "you must generate a question that is strictly self-contained within a single page. Select one specific page from the provided sequence and generate a question based solely on its content." Example: "According to the Homemade Bitters recipe on Page 39, how long should the herbs soak in vodka?" This task requires the model to locate a specific page among potentially dozens and extract a piece of factual information.

2. Multi-page extraction (extract-multi): Questions that require aggregating information from multiple pages. The prompt instructs: "prioritize generating questions that require synthesizing information across different pages (>= 2 pages)." Example: "Based on Pages 6, 13, and 19, list all risk factors mentioned in the report." This task requires the model to locate multiple relevant pages and combine their information, testing cross-page retrieval and aggregation.

3. Reasoning: Questions that require numerical or logical operations over extracted information. The prompt specifies three operation types: calculate (arithmetic on numerical data), compare (identifying trends, maxima, minima), and count (frequency of specific items). Example: "What is the difference between total consumption and total imports for rice production in 2020?" This task adds a computation layer on top of retrieval—the model must find the right data points and then perform an operation on them.

Data statistics for the pool-native distribution (default, Table 10a, Figure 7a). For each task:

The average training instance contains 38.4 pages and 85,300 tokens, with individual instances ranging from 32K to 127K tokens. Each task individually exceeds the 5B-token training budget (5.04–5.06B tokens), which is why the paper uses these as separate training sets for single-task experiments (Table 1) and then mixes them for the final recipe (Section 5.2). Only 23.5–23.6% of instances exceed 100K tokens (Table 11), meaning the distribution naturally covers the full 32K–128K range rather than concentrating at the upper end.

---

OCR Transcription Data Synthesis (Baseline Comparison)

To establish that long-document VQA is genuinely more effective than alternative supervision sources, the paper constructs a contrasting category: OCR transcription (Section 4.3, Appendix E). The core idea is that instead of asking the model to answer questions about the document, you ask it to transcribe the text content of the pages. This forces the model to maintain fine-grained image-text alignment across the full context.

Synthesis pipeline. For each document:

1. Every page is parsed by the OCR expert to extract text elements (section titles, paragraphs, tables, captions)
2. A training instance is constructed where the visual input is the rendered page images and the target output is the parsed text elements

Two variants are constructed:

1. Full-document OCR (OCR-full): The model must transcribe text elements from all pages of the document. This is a dense supervision task—the model must attend to every page and produce text for every page. The idea is that this encourages image-text dependency modeling across the full context length.

2. Needle-page OCR (OCR-needle): Only 1–3 pages (randomly selected) need to be transcribed; the remaining pages serve as distractors. The model must identify which pages contain the transcription targets and which are irrelevant, making this closer to a retrieval task within the OCR format.

Data statistics (Table 10c, Figure 7c).

The OCR data is substantially larger than the VQA data (21.39B vs. 15.16B total tokens across all tasks) because the transcription targets are longer than VQA answers—transcribing a page produces much more output text than generating a single answer.

Why compare against OCR transcription? The paper's rationale (implicit in the comparison in Section 4.4) is that OCR transcription is the most "obvious" alternative supervision source for long-context LVLM training. If you want a model to handle long multimodal contexts, requiring it to transcribe text across all pages seems like a natural way to force it to attend to the full input. However, as the results in Table 1 show, this intuition is wrong: OCR transcription substantially degrades performance compared to the base model, even with an additional 5B-token SFT stage to improve instruction-following ability.

Why does OCR transcription underperform? The paper doesn't fully explain this, but the likely mechanism is: OCR transcription teaches the model to decode visual text into linguistic text at every position in the input, but this skill is different from retrieving and reasoning over specific evidence to answer a question. The OCR-trained model might become very good at reading text from images across the full context but worse at determining which text is relevant to a given question—the dense transcription objective may interfere with the selective attention patterns needed for retrieval. Additionally, OCR transcription outputs are not formatted as instruction-following responses, so the model may lose some of its instruction-following ability during continued pre-training, requiring the additional SFT stage to partially recover.

---

Data-Design Ablation 1: Training Sequence-Length Distribution

The question (Section 5.1). Given that training instances are synthesized from documents with 32–50 pages, producing multimodal sequences that naturally span 32K to 128K tokens (the "pool-native" distribution), should we instead reweight the data to concentrate more training mass near the target length of 128K?

Motivation for the long-biased distribution. The intuition for concentrating at 128K is straightforward: if we want the model to perform well at 128K context during evaluation, maybe we should train it mostly on examples near that length. This follows a common intuition in ML that training should match the test distribution. If 128K is the target, shouldn't most training examples be near 128K?

Construction of the two distributions:

• Pool-native (default): Training instances are synthesized from documents naturally sampled within the 32–50 page range, without additional length-based reweighting. Only 23.5–23.6% of instances exceed 100K tokens (Table 11). The distribution spans the full 32K–128K range with a mode around 85K tokens (Figure 7a).

• Long-biased: Training instances are synthesized from longer documents (50–100 pages), producing sequences where 83.9% of instances exceed 100K tokens (Table 11). The distribution is heavily concentrated near the upper end of the range (Figure 7b).

Data statistics for long-biased (Table 10b, Figure 7b):

Note that the long-biased data totals 20.47B tokens across all tasks—substantially more than the pool-native's 15.16B tokens, because longer documents produce longer training instances. However, each experiment uses only one task type at a time, and the training is capped at 5B tokens regardless of the total data available, so the comparison is fair in terms of training budget.

Results (Figure 2, Table 14). The pool-native distribution consistently outperforms the long-biased distribution across all three training tasks, with average gains of +1.3, +0.1, and +1.7 points for extract-single, extract-multi, and reasoning respectively (Figure 2). The full per-dataset breakdown in Table 14 shows that pool-native is better or comparable on nearly every individual benchmark at both 64K and 128K.

The paper's interpretation:

"long-context ability is not a discrete capability acquired only at a specific target length, such as 128K context. Instead, it requires continuous calibration across different absolute positions and relative image-text distances."

In other words, the model needs to learn to retrieve key information from any position in the context, not just from positions near the maximum length. Training mostly on 128K examples might teach the model good long-range attention at exactly that length, but it might perform worse on shorter contexts (where the attention patterns would be distributed differently) and might not generalize well to the specific positional regimes within a 128K sequence (e.g., information at position 90K vs. position 120K).

Why does balanced data help generalization? The likely mechanism is that balanced length distributions force the model to learn position-invariant retrieval: the ability to find relevant evidence regardless of where it appears in the context. If the model is trained mostly on 128K examples, it might learn that relevant information tends to be somewhere in the middle-to-late part of the context (since 128K examples have more "middle" positions). With balanced training, the model sees evidence at every possible position offset and develops attention patterns that work across the full range.

---

Data-Design Ablation 2: Multi-Task Long-Context Data Mixture

The question (Section 5.2). Given the three long-document VQA tasks, what is the optimal mixture ratio between information extraction (combining extract-single and extract-multi) and reasoning?

Grid search design. The paper groups the three tasks into two categories:

• Information extraction: extract-single and extract-multi, combined evenly (50% each within the extraction category)
• Reasoning: the reasoning task alone

Then it grid-searches the extraction-to-reasoning ratio in 20% increments, from all-reasoning (0:10) to all-extraction (10:0), training a separate model for each ratio with the fixed 5B-token budget. This produces six checkpoints evaluated on long-document VQA at 64K and 128K (Table 2).

Results. Moderately extraction-heavy mixtures perform best:

The 8:2 ratio achieves both the best 64K average (59.56) and the best overall average (57.70). The 6:4 ratio is close behind (57.27 overall). Note an interesting pattern: the 128K averages are relatively flat across ratios (54.24–56.40), while the 64K averages show more variation (56.35–59.56). This suggests that at 128K, all mixtures perform similarly because 128K is uniformly challenging regardless of the task mixture, while at 64K (which is still beyond the base model's 32K training window but not as extreme), the task mixture matters more.

The paper's interpretation:

"retrieving key information from long-context inputs remains a major bottleneck when extending the context window, while retaining a small amount of reasoning data helps preserve task diversity."

This makes sense mechanistically: if the primary challenge at long contexts is finding the right evidence (not reasoning over it once found), then training data should emphasize retrieval tasks. But including some reasoning data (20%) ensures the model doesn't forget how to perform operations on retrieved evidence—it maintains the "reasoning over evidence" capability that the base model already has.

Why does all-reasoning underperform all-extraction? A reasoning question requires both retrieval (finding the relevant data points) and computation (performing the arithmetic/logical operation). If the retrieval step is the bottleneck, then spending all training budget on reasoning examples gives the model fewer opportunities to practice retrieval compared to an extraction-heavy mixture. The all-reasoning model might be good at computing sums but bad at finding which numbers to sum in a 128K context.

---

Data-Design Ablation 3: Short-Context Performance Preservation

The question (Section 5.3). Long-context continued pre-training often degrades short-context performance—the model "forgets" how to handle shorter inputs because all its training is on long sequences. Do we need to mix in short-context data during LongPT to prevent this forgetting?

Experimental design. The paper keeps the 8:2 extraction-to-reasoning long-context mixture fixed and varies the proportion of short-context data from 0% (pure long-context training) to 80% in 20% increments. The total token budget remains 5B tokens, so adding short-context data reduces the amount of long-context data the model sees. For example, at 40% short data, the model sees 3B tokens of long-context data and 2B tokens of short-context data.

Short-context data source. The short-context data comes from LLaVA-OneVision (Appendix F), a publicly available instruction-tuning dataset that covers single-image and multi-image VQA tasks. This is chosen because it is representative of standard short-context VLM training data and aligns with the instruction-following format of the long-document VQA data.

Results on long-context performance (Figure 3, Table 15). Adding short-context data consistently reduces long-document VQA performance:

The pure long-context training (0%) achieves the best overall average of 57.70, substantially outperforming all mixtures. The 40% ratio provides a reasonable compromise (57.01 overall, down only 0.69 from pure long-context) while the 20% ratio is notably worse at 64K (55.98, down 3.58).

Results on short-context performance (Table 3). The surprising finding is that pure long-context training does not severely degrade short-context capabilities:

The 0% model drops only from 66.47 to 65.48 on the short-context average—less than 1 point. The 20% model actually slightly exceeds the base model's short-context average (66.53 vs. 66.47). Individual benchmarks show mixed patterns: General VQA improves (76.34 vs. 74.72 at 0%), multimodal reasoning drops (51.56 vs. 53.77 at 0%), and text recognition drops (85.50 vs. 88.10 at 0%), but none of the degradations are catastrophic.

The paper's interpretation:

"high-quality long-document VQA data can preserve the model's general short-context ability, possibly because its QA format still follows an instruction-following style despite the substantially longer input context."

The key insight is that the format of the long-document VQA data matters for short-context preservation. The data is instruction-formatted—it follows the same question-answering paradigm as the base model's original training, just with much longer inputs. This means the model's instruction-following circuitry continues to be exercised during LongPT, preventing catastrophic forgetting. If the long-context data had a completely different format (e.g., the OCR transcription task, which requires generating long text sequences rather than answering questions), the risk of forgetting would be higher.

The tradeoff and the paper's recommendation. The paper makes two recommendations based on the application:

• For maximum long-context capability: use 0% short-context data. This gives the best long-document VQA performance (57.70 overall) with only a minor short-context drop (65.48 vs. 66.47 base).
• For balanced long and short-context performance: use 40% short-context data. This preserves most long-context gains (57.01 overall) while keeping short-context nearly at base-model level (66.14).

The final MMProLong recipe uses the 0% setting, prioritizing long-context capability.

---

Final Recipe and Evaluation Protocol

The final MMProLong recipe (Appendix A.1, Table 6). Based on all three ablations, the final training recipe is:

• Data synthesis type: Long-document VQA (extract-single, extract-multi, and reasoning) synthesized via the short-to-long pipeline
• Length distribution: Pool-native (natural sampling from the document pool over 32K–128K)
• Task mixture: 8:2 extraction-to-reasoning ratio, with the extraction category split evenly between extract-single (40% of total) and extract-multi (40% of total), and reasoning at 20%
• Short-context data: None (pure long-context data)
• Alternative for short-context preservation: 60% long-context data, 40% LLaVA-OneVision data
• Base model: Qwen2.5-VL-7B-Instruct (original mRoPE base frequency $1 \times 10^6$)
• mRoPE base for 128K: $4 \times 10^6$
• Maximum sequence length: 131,072 tokens (128K)
• Token budget: 5B tokens (approximately 2,900 H20 GPU hours)
• Optimizer: AdamW (weight decay 0.1, $\beta_1 = 0.9$, $\beta_2 = 0.95$)
• Learning rate schedule: $1 \times 10^{-5}$ with 10% linear warmup and cosine decay to $1 \times 10^{-6}$
• Global batch size: 4M tokens (32 sequences per update)
• Framework: VeOmni with FlashAttention-2
• Parallelism: Ulysses sequence parallelism size 2, FSDP size 4

Evaluation protocol for long-document VQA (Appendix B.1). The primary evaluation uses the document category of MMLongBench, which contains three datasets:

1. MMLongBench-Doc (MMLB-D): Long-document understanding with visualizations
2. LongDocURL (LD-URL): Comprehensive document understanding integrating understanding, reasoning, and locating
3. SlideVQA (SLIDE): Document VQA on multiple slide images

All examples are instantiated at five standardized context lengths: 8K, 16K, 32K, 64K, and 128K tokens. The paper primarily evaluates at 64K and 128K (beyond the base model's 32K training window) and reports the macro average over the three datasets at each length, plus an overall average that is the macro average over 64K and 128K.

LLM-judged scoring (Tables 7 and 8). Instead of simple string matching, the paper uses LLM-based judging (via an unspecified judge LLM, likely GPT-4 or similar) that handles different answer formats:

• Binary scoring (Table 7): For simple answer formats (string, integer, float, "not answerable"), the judge compares the predicted answer to the reference answer and assigns 1 point if they match (accounting for semantic equivalence, numerical equivalence, synonyms, and plural/singular forms) or 0 points if they don't. The judge is instructed to extract the final answer from the student's solution first, then compare.

• List scoring (Table 8): For list-style answers, the judge extracts the predicted list and computes an F1 score based on overlap with the reference list. Specifically, the judge counts how many items from the predicted list match items in the reference list (using the same semantic equivalence rules as binary scoring) and reports both student_answer_count and covered_count.

Why LLM judging rather than exact match? Long-document VQA answers can have many valid surface forms—"45 percent," "45%," and "0.45" are all correct answers to a question about a percentage. Exact string matching would incorrectly penalize these variants. LLM judging can recognize semantic equivalence while still distinguishing genuinely incorrect answers from format variations. The cost, as noted in Appendix H, is that LLM judging introduces API cost that "becomes substantial when evaluating many checkpoints, context lengths, and model variants."

Longer-context evaluation (Appendix B.3). To test extrapolation beyond the 128K training window, the paper further evaluates at 256K and 512K context lengths (Section 6.2, Table 5). These longer contexts are constructed following MMLongBench protocol: the original test examples are padded with randomly sampled negative documents (documents unrelated to the question) on alternating sides (left and right) until the target length is reached. This preserves the original evidence position while surrounding it with distractor material, testing whether the model can maintain retrieval accuracy in increasingly noisy contexts.

Additional evaluation benchmarks. Beyond long-document VQA, the paper evaluates MMProLong zero-shot on:

• MM-NIAH (Appendix B.4): A multimodal needle-in-a-haystack benchmark built on webpages from OBELICS, with three task families (retrieval, counting, reasoning) each having text-needle and image-needle variants, evaluated at 64K and 128K. The AVG. score is the macro average over the three task families, with each task family first averaging its text and image variants.

• VTCBench (Appendix B.5): Long-context vision-text compression, using the VTCBench-Wild setting with three tasks: Retrieval (800 examples, containsAll accuracy), Reasoning (800 examples, containsAll accuracy), and Memory (600 examples, LLM-judged accuracy). The AVG. is the sample-count weighted average: $(800 \times R_{\text{ret}} + 800 \times R_{\text{reas}} + 600 \times R_{\text{mem}}) \div 2200$.

• Long-video benchmarks (Appendix B.6): Video-MME (multiple-choice accuracy), MLVU (multiple-choice subset average accuracy), and LongVideoBench (overall multiple-choice accuracy). All evaluated at 1fps with a maximum of 768 sampled frames and 24,576 video tokens, matching the Qwen2.5-VL technical report's configuration.

• Short-context benchmarks (Appendix B.7): Six benchmarks across three capabilities: General VQA (MMBench-V1.1, RealWorldQA), Multimodal Reasoning (MMMU dev/val, MMMU-Pro 10-choice, MathVista testmini), and Text Recognition (OCRBench).

Cross-validation for final recipe selection. Throughout the ablation studies, the paper evaluates each model variant directly on the test sets, but this doesn't constitute a train-test leak because the LongPT data is synthesized from a filter-banked document pool (excluding evaluation documents via SHA-256 hashing), and the evaluation benchmarks are fixed test sets the model hasn't seen during training. The final recipe is selected based on the best-performing configuration in the ablations, not through test-set hyperparameter search.

4. Key Insights and Innovations

Innovation 1: The Instruction Format as a Forgetting Shield

The most counterintuitive finding in this paper is that pure long-context training does not catastrophically degrade short-context performance—and the mechanism behind this result reframes how we should think about catastrophic forgetting in continued pre-training.

The dominant assumption in long-context LLM training, established by Gao et al. (2024) and Fu et al. (2024), is that extending context windows requires careful mixing of short-context data to prevent the model from "forgetting" how to handle shorter inputs. This makes intuitive sense: if you train exclusively on 128K-token sequences, the model's attention patterns, positional encodings, and internal representations might adapt to long-range dependencies at the expense of the local patterns that govern short-context reasoning.

The paper's Section 5.3 directly tests this assumption and finds that it does not hold—or at least, holds much more weakly—when the long-context data follows an instruction-following format. Table 3 shows that pure long-document VQA training (0% short-context mixing) preserves the short-context average to within ~1 point of the base model (65.48 vs. 66.47), with some short-context capabilities actually improving (General VQA rises from 74.72 to 76.34). This is in stark contrast to the OCR transcription baselines in Table 1, which degrade by 17.4 points without an additional SFT recovery stage.

The conceptual move here is to recognize that the format of the data, not just its length, determines forgetting dynamics. Long-document VQA data is instruction-formatted—it follows the same "question → answer" paradigm that the base model was trained on, just with much longer visual-textual inputs. The model's instruction-following circuitry continues to be exercised during LongPT, even though the inputs are 4× longer than its original training window. The OCR transcription data, by contrast, requires the model to produce long text sequences in a completely different format (transcription, not question-answering), which disrupts the instruction-following patterns and requires additional SFT to partially recover.

This is not just an empirical curiosity—it has direct implications for how practitioners design long-context training curricula. The paper's default recommendation (0% short-context data) means that the entire 5B-token budget can be spent on long-context data, maximizing long-context gains without the usual tradeoff penalty. The practical consequence is a simpler, more data-efficient recipe: you don't need to carefully balance short and long data if your long data is instruction-formatted enough to preserve the base model's response patterns.

This insight is fundamental rather than incremental because it changes the causal model of forgetting in continued pre-training. The standard model is that forgetting is driven by distribution shift in input length. This paper's evidence suggests that forgetting is driven by distribution shift in output format, and that input length is a secondary concern that can be mitigated by format consistency. This opens the door to designing long-context data that preserves specific short-context capabilities by mimicking their output structure, rather than relying on explicit short-data mixing as a patch.

Innovation 2: Balanced Length Distributions as Positional Generalization Training

The paper's Section 5.1 result—that a balanced sequence-length distribution (pool-native) outperforms a target-length-concentrated distribution (long-biased) by 1.3–1.7 points across tasks (Figure 2)—initially seems like a modest empirical preference. But the interpretation the paper offers transforms it into a conceptual contribution about what long-context capability actually is.

The naive view, implicit in much of the context-extension literature, is that long-context ability is a discrete capability that needs to be "unlocked" at the target length. Under this view, training should concentrate examples near the target (128K) because that's where the model needs to perform. The long-biased distribution, with 83.9% of examples exceeding 100K tokens (Table 11), operationalizes this intuition.

The pool-native distribution's superiority refutes this. The paper's interpretation is that long-context ability is not a discrete skill acquired at a specific length—it is position-invariant retrieval: the ability to find relevant evidence regardless of where it appears in the context. A model trained mostly on 128K examples might learn effective attention patterns for the middle-to-late positions that dominate long sequences, but would have less practice with evidence near the beginning (which is relatively rare in 128K examples but common in 64K examples). The balanced distribution forces the model to handle evidence at every relative position and every absolute distance, producing retrieval behavior that generalizes across lengths.

This reframes the training objective: LongPT should not teach the model to "handle 128K context" but rather to retrieve information in a length-agnostic way. The balanced distribution achieves this by providing training signal at every position offset, teaching the model that evidence can be anywhere, not just somewhere in the vast middle of a long document.

The significance of this finding is amplified by its contrast with LLM long-context training. In the LLM setting, Gao et al. (2024) and Fu et al. (2024) use corpora like SlimPajama and the Stack where sequence lengths are naturally distributed across the target range—they don't need to make an active choice about length distribution because their data sources provide balanced coverage by default. In the LVLM setting, because training data must be synthesized from documents with specific page counts, practitioners face an explicit choice: what length distribution should they target? The paper's finding that the natural (pool-native) distribution is optimal is practically useful because it means practitioners can avoid the additional complexity of length-based data reweighting, but it's also conceptually important because it provides evidence that the LLM community's implicit reliance on naturally balanced distributions was well-founded for reasons that go beyond convenience.

This is an incremental but clarifying insight. The idea that diverse training is better than narrow training is not new, but the specific claim—that balanced lengths teach generalization across absolute positions and relative distances—provides a mechanistic rationale that was absent from prior work, and the controlled ablation gives it empirical weight.

Innovation 3: Retrieval as the Primary Bottleneck in Multimodal Long-Context Training

The task mixture ablation in Section 5.2 produces a clear empirical result: extraction-heavy mixtures outperform reasoning-heavy ones, with an 8:2 extraction-to-reasoning ratio achieving the best overall score (57.70, Table 2). The paper interprets this as evidence that retrieval, not reasoning, is the primary bottleneck when extending LVLM context windows.

This is a diagnostic finding with practical and conceptual implications. The base model (Qwen2.5-VL-7B) already possesses strong reasoning capabilities—it can perform arithmetic, compare values, and count items when given the right evidence. What it lacks after context extension is the ability to find that evidence in a 128K multimodal context. If the model can't locate the relevant table on page 37, it doesn't matter how good its arithmetic is. The extraction-heavy mixture addresses this bottleneck directly by providing more training examples that require the model to practice retrieval, while the 20% reasoning budget ensures the model doesn't forget the reasoning skills it already has.

This finding also explains why the OCR transcription data underperforms so dramatically in Section 4.4. Full-document OCR requires the model to attend to everything—every page, every text block—rather than teaching it to selectively attend to relevant evidence. This is the opposite of what the bottleneck requires. The model learns dense image-text alignment but not sparse evidence retrieval, which is precisely the skill that long-context VQA demands.

The paper's contribution here is not the specific 8:2 ratio (which might not transfer to other models or datasets) but the diagnostic framework it establishes: when extending context windows, measure whether retrieval or reasoning is the bottleneck, and allocate the training budget accordingly. This framework should generalize: if a base model has weak reasoning but strong retrieval, the optimal mixture might be reasoning-heavy; if both are weak, a balanced mixture might be needed. The paper doesn't explore these alternatives, but it provides the conceptual structure for doing so.

This is a fundamentally diagnostic rather than prescriptive innovation. It doesn't give a universal recipe—it gives a way of thinking about the problem that enables other practitioners to design their own recipes based on their specific starting conditions and bottlenecks.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The primary evaluation uses the document category of MMLongBench (Wang et al., 2025), which contains three datasets: MMLongBench-Doc (MMLB-D; Ma et al., 2024), LongDocURL (LD-URL; Deng et al., 2025), and SlideVQA (SLIDE; Tanaka et al., 2023). All examples are instantiated at five standardized context lengths (8K, 16K, 32K, 64K, 128K tokens), though the paper reports primarily at 64K and 128K. The test set consists of 500 questions, and the authors additionally construct 256K and 512K variants following MMLongBench protocol: padding with randomly sampled negative documents on alternating sides until the target length is reached (Appendix B.3).

• Base model(s). All LongPT experiments use Qwen2.5-VL-7B-Instruct (Bai et al., 2025), whose native context window is 32,768 tokens (32K). The paper also validates the final recipe on Qwen3-VL-8B (Bai et al., 2025) in Appendix G.7 to test generalizability across backbones. For the final comparison (Table 4), the paper evaluates a wide range of open-source and closed-source LVLMs including Qwen2.5-VL at 7B/32B/72B, InternVL3 at 8B/14B/38B, InternVL3.5 at 8B/14B/38B, Gemma3 at 4B/12B/27B, Gemma4 at E2B/E4B/26B-A4B/31B, GPT-5.4/5.5, and Gemini-2.5-Pro/3.1-Flash/3.1-Pro.

• Metrics. Long-document VQA uses LLM-judged document QA scores from MMLongBench v1.1 protocol (Appendix B.1). For simple answer formats (string, integer, float, "not answerable"), the judge assigns a binary score of 1 or 0 based on semantic equivalence (Table 7). For list-style answers, the judge extracts the predicted list and computes an F1 score based on overlap with the reference list (Table 8). The AVG. column at each context length is the macro average over the three datasets; the overall AVG. is the macro average over 64K and 128K results. Additional metrics include MM-NIAH retrieval/counting/reasoning scores with an overall macro average (Appendix B.4), VTCBench containsAll accuracy for retrieval and reasoning tasks and LLM-judged accuracy for memory tasks with sample-count weighted averaging (Appendix B.5), long-video multiple-choice accuracy for Video-MME, MLVU, and LongVideoBench (Appendix B.6), and standard per-benchmark scores for short-context evaluation (Appendix B.7).

• Baselines. The primary baselines throughout the paper are: (i) the untrained Qwen2.5-VL-7B base model as the reference point for all LongPT improvements; (ii) OCR transcription-trained models (OCR-full and OCR-needle, both with and without an additional 5B-token SFT stage using LLaVA-OneVision data) as alternative supervision sources (Table 1); (iii) single-task LongPT models trained on each of extract-single, extract-multi, and reasoning individually (Table 1); (iv) for the final comparison, a comprehensive set of open-source and closed-source LVLMs (Table 4) including Qwen2.5-VL (32B, 72B), InternVL3 (8B, 14B, 38B), InternVL3.5 (8B, 14B, 38B), Gemma3 (4B, 12B, 27B), Gemma4 (E2B, E4B, 26B-A4B, 31B), GPT-5.4/5.5, and Gemini-2.5-Pro/3.1-Flash/3.1-Pro.

• Generation budget / compute accounting. All LongPT ablations are conducted under a fixed 5B-token training budget to ensure fair comparisons across data-design choices (Section 3). The training budget is measured in total multimodal tokens processed during continued pre-training, not in FLOPs or GPU hours. For the OCR transcription experiments that include an additional SFT stage (Section 4.4), the SFT stage uses an additional 5B tokens, and this extra cost is explicitly noted but the results are reported as a separate condition (OCR-full/SFT and OCR-needle/SFT in Table 1). The final MMProLong uses 5B tokens of pure long-context data with no additional SFT (Table 6). The auxiliary cost of training the OCR expert model and the QA generator (Seed 2.0) is not included in the training budget accounting.

• Cross-validation / statistical protocol. There is no formal cross-validation or statistical significance testing reported. The paper selects the final recipe based on the best-performing configuration in the ablations (e.g., 8:2 extraction-to-reasoning ratio, pool-native distribution, 0% short-context data). Since each ablation varies one design axis while holding others fixed, the selection is deterministic rather than statistical. The paper does not report confidence intervals, standard deviations, or p-values for any result. The only "validation" check is the human verification of 100 randomly sampled QA pairs from the synthesis pipeline (Appendix D.4), which found 97/100 correct—this assesses data quality, not model evaluation reliability. The absence of statistical rigor is a genuine limitation, particularly given that the test set contains only 500 questions split across three datasets and evaluated at multiple context lengths, meaning some per-dataset comparisons may involve small effective sample sizes (e.g., SlideVQA with ~100 questions at each context length).

---

Main Quantitative Results

Long-Document VQA vs. OCR Transcription: Establishing the Primary Supervision Source

The paper's first major experimental finding is that long-document VQA provides substantially more effective supervision for LongPT than OCR transcription (Section 4.4, Table 1).

Headline comparison. When trained with a 5B-token budget on individual tasks, all three long-document VQA variants improve over the base model, with multi-page extraction achieving the best overall average of 56.90—an improvement of 6.3 points over the base model's 50.59. In contrast, OCR-full degrades performance dramatically to 33.17, a drop of 17.4 points. OCR-needle fares better but still underperforms at 43.80, dropping 6.8 points.

Method-level breakdown (Table 1):

• The base Qwen2.5-VL-7B achieves 52.24 at 64K and 48.94 at 128K, with an overall average of 50.59. The degradation from 64K to 128K (3.3 points) confirms that the 32K-trained model struggles beyond its training window.
• extract-single: 56.86 at 64K, 54.53 at 128K, overall 55.69 (+5.1 over base)
• extract-multi: 58.02 at 64K, 55.77 at 128K, overall 56.90 (+6.3 over base)
• reasoning: 57.33 at 64K, 55.62 at 128K, overall 56.47 (+5.9 over base)
• OCR-full: 31.24 at 64K, 35.11 at 128K, overall 33.17 (−17.4)
• OCR-needle: 45.61 at 64K, 42.00 at 128K, overall 43.80 (−6.8)

Effect of SFT recovery on OCR-trained models. Even with an additional 5B-token SFT stage using LLaVA-OneVision instruction data, the OCR-trained checkpoints only partially recover. OCR-full with SFT reaches 53.84, only 3.2 points above the base model, while OCR-needle with SFT reaches 52.44, only 1.9 points above base. Both remain below any of the VQA-trained variants, despite the VQA variants requiring no SFT stage and thus using half the total training budget.

Per-dataset patterns. The degradation is most severe on LongDocURL: OCR-full drops from the base model's 49.57/51.85 (64K/128K) to 11.06/20.36. This dataset requires reasoning over document structure and evidence retrieval—exactly the capabilities that OCR transcription fails to teach. On SlideVQA, the degradation is less severe (75.00→59.00 at 64K), possibly because SlideVQA involves simpler visual layouts where transcription ability provides some incidental benefit.

The paper's takeaway is that "instruction-formatted supervision and task diversity, ranging from information extraction to complex numerical reasoning, are important for LongPT." The ablation establishes long-document VQA as the primary data source for all subsequent experiments.

---

Sequence-Length Distribution: Pool-Native Outperforms Long-Biased

The second axis of investigation compares two strategies for allocating the 5B-token training budget across different sequence lengths (Section 5.1, Figure 2, Table 14).

Headline result. The pool-native distribution (where 23.5–23.6% of instances exceed 100K tokens) consistently outperforms the long-biased distribution (where 83.9% exceed 100K) across all three task types, with average gains of +1.3, +0.1, and +1.7 points for extract-single, extract-multi, and reasoning respectively, as measured by the overall average across 64K and 128K.

Per-task comparison (Table 14):

• extract-single: pool-native achieves 56.69 vs. 55.37 for long-biased (Figure 2 reports +1.3 advantage for pool-native; Table 14 shows 55.69 overall vs. 54.37, matching the +1.3 gap)
• extract-multi: pool-native achieves 56.90 vs. 56.76 for long-biased (the narrowest gap at +0.1)
• reasoning: pool-native achieves 56.47 vs. 54.79 for long-biased (+1.7)

Context-length disaggregation (Table 14). The advantage is not uniform across context lengths. For extract-multi, the long-biased distribution actually matches or slightly exceeds pool-native at 64K (57.68 vs. 58.02 for LongDocURL, but pool-native leads on SlideVQA) while falling behind at 128K (54.33 vs. 55.77 overall). For extract-single, pool-native leads at both lengths. This suggests that the benefit of balanced training is more pronounced at the longer evaluation context, consistent with the interpretation that balanced training teaches position-invariant retrieval that generalizes better to extreme lengths.

The number of near-128K training examples does not predict performance. This is the critical negative result: despite the long-biased distribution providing nearly 4× more examples exceeding 100K tokens (83.9% vs. 23.6%, Table 11), these additional long examples do not translate to better 128K evaluation performance. The paper's interpretation—that long-context ability requires "continuous calibration across different absolute positions and relative image-text distances" rather than specialization to a target length—is supported by the fact that adding more 100K+ examples actively harms rather than helps.

---

Task Mixture Ratios: Extraction-Heavy Blends Are Optimal

The third experiment grid-searches the mixture of extraction and reasoning tasks under the fixed 5B-token budget (Section 5.2, Table 2), using the pool-native distribution for all variants.

Headline result. The 8:2 extraction-to-reasoning ratio achieves the best overall average of 57.70, outperforming both all-reasoning (0:10, 56.47) and all-extraction (10:0, 56.94) by approximately 0.8–1.2 points.

Full grid search (Table 2):

Notable pattern: the 128K averages are relatively flat across ratios. The 128K scores range from 54.24 (2:8) to 56.40 (10:0), a span of only 2.16 points, while the 64K scores range from 56.35 (4:6) to 59.56 (8:2), a span of 3.21 points. This asymmetry suggests that at 128K, the difficulty of managing the extreme context length partially masks the effect of task mixture—the fundamental challenge of 128K retrieval is hard enough that the precise mixture matters less. At 64K, where the context is still extended but more manageable, the training data composition has more room to influence performance.

The 8:2 ratio as a practical default. The paper adopts this ratio for the final recipe. It represents a strongly retrieval-heavy blend (since the extraction category itself is a 50/50 mix of extract-single and extract-multi, meaning the final data is 40% extract-single, 40% extract-multi, 20% reasoning, as specified in Table 6). The interpolation between 6:4 (57.27) and 8:2 (57.70) suggests that further increasing extraction dominance yields diminishing returns, and that some reasoning data is beneficial—all-extraction (10:0) scores lower than 8:2 (56.94 vs. 57.70), confirming that keeping at least 20% reasoning in the mixture helps maintain task diversity.

---

Short-Context Data Mixing: Minimal Forgetting Without Short Data

The fourth axis of investigation examines the tradeoff between long-context performance and short-context preservation when varying the proportion of short-context instruction data mixed into the LongPT stage (Section 5.3, Figure 3, Table 3, Table 15).

Long-context performance (Table 15). Adding any short-context data reduces long-document VQA scores:

The 0% (pure long-context) setting is clearly the best for long-document VQA. The 20% setting performs anomalously poorly—worse than both 0% and 40%—particularly at 64K (55.98 vs. 59.56 for 0%, a gap of 3.58 points). This non-monotonicity is not explained in the paper but may reflect an interaction where a small amount of short-context data disrupts the model's adaptation to long contexts without providing enough short-context signal to stabilize performance.

Short-context performance (Table 3). The headline finding is that pure long-context training causes only mild degradation:

The 0% setting shows a mixed picture: General VQA actually improves (76.34 vs. 74.72), multimodal reasoning drops modestly (51.56 vs. 53.77), and text recognition drops more noticeably (85.50 vs. 88.10). The overall average drop of ~1 point (65.48 vs. 66.47) is surprisingly small. The 20% setting essentially matches or exceeds the base model across all categories, achieving the highest short-context average of 66.53, though at the cost of a 2.13-point drop in long-context performance (55.57 vs. 57.70 overall).

The paper's final recommendation. The 0% setting is chosen for MMProLong, prioritizing maximum long-context capability while accepting a ~1-point short-context degradation. The paper explicitly notes that the 40% setting provides a balanced alternative for applications where stronger short-context preservation is required (57.01 long-context, 66.14 short-context).

---

Final MMProLong Results: Comparison Against Baselines

The final MMProLong model is evaluated against a comprehensive set of open-source and closed-source LVLMs (Section 6.1, Table 4). All long-document VQA results are at 64K and 128K.

Headline result. MMProLong achieves an overall average of 57.70, improving over its base model Qwen2.5-VL-7B by 7.11 points (from 50.59). This represents a 14% relative improvement.

Per-dataset improvement (Table 4):

• MMLongBench-Doc: 32.17→36.00 at 64K, 26.96→34.19 at 128K
• LongDocURL: 49.57→62.69 at 64K, 51.85→56.33 at 128K
• SlideVQA: 75.00→80.00 at 64K, 68.00→77.00 at 128K

The largest absolute gain is on LongDocURL (+8.15 points on average across both lengths), which is particularly notable because LongDocURL is also the dataset where OCR transcription training caused the most severe degradation (Table 1: 49.57→11.06 at 64K). This suggests that LongDocURL is highly sensitive to the quality of the training recipe—it can be substantially improved or catastrophically degraded depending on data choices.

Comparison against larger open-source models. MMProLong (7B) outperforms several substantially larger models:

• InternVL3-38B: 57.70 vs. 48.88 (gap of 8.82)
• InternVL3.5-38B: 57.70 vs. 33.81 (gap of 23.89—note the InternVL3.5 series performs poorly on this benchmark, particularly at 128K where SlideVQA drops to 1.01 and 0.00 for 8B and 14B variants)
• Gemma3-27B: 57.70 vs. 52.63 (gap of 5.07)
• Gemma4-26B-A4B: 57.70 vs. 47.74 (gap of 9.96)

However, MMProLong is outperformed by large models that have undergone extensive long-context training:

• Qwen2.5-VL-72B: 60.83 (gap of 3.13)
• Gemma4-31B: 67.10 (gap of 9.40)

Comparison against closed-source models. MMProLong trails the state-of-the-art significantly:

• GPT-5.4: 69.41
• GPT-5.5: 90.77
• Gemini-2.5-Pro: 74.37
• Gemini-3.1-Flash: 80.13
• Gemini-3.1-Pro: 83.66

The gap is largest on MMLongBench-Doc (MMProLong: 36.00/34.19 at 64K/128K vs. GPT-5.4: 61.56/52.96) and smallest on SlideVQA (MMProLong: 80.00/77.00 vs. Gemini-3.1-Pro: 93.00/93.00). This pattern suggests that MMProLong's remaining weakness is primarily on datasets requiring complex multi-hop reasoning (MMLongBench-Doc) rather than visual understanding of simpler document layouts (SlideVQA).

---

Generalization Beyond 128K: 256K and 512K Context Lengths

A critical question for any context-extension method is whether performance degrades at lengths beyond the training window. The paper tests MMProLong at 256K and 512K without additional training or adaptation (Section 6.2, Table 5).

Headline result. MMProLong maintains strong performance at both extended lengths, achieving overall averages of 55.09 at 256K and 52.52 at 512K, compared to its 128K average of 55.84 (Table 4). The base model degrades sharply: 38.12 at 256K and 19.49 at 512K, for an overall average of 28.80 across both lengths vs. MMProLong's 53.80.

Comparison at 256K (Table 5):

• MMProLong: 55.09 (MMLB-D: 29.69, LD-URL: 58.58, SLIDE: 77.00)
• Qwen2.5-VL-7B: 38.12 (MMLB-D: 25.47, LD-URL: 35.88, SLIDE: 53.00)
• Gemma3-12B: 47.37 (MMLB-D: 31.63, LD-URL: 47.47, SLIDE: 63.00)

Comparison at 512K (Table 5):

• MMProLong: 52.52 (MMLB-D: 31.91, LD-URL: 55.65, SLIDE: 70.00)
• Qwen2.5-VL-7B: 19.49 (MMLB-D: 13.44, LD-URL: 24.61, SLIDE: 20.41)
• Gemma3-4B: 15.51 (MMLB-D: 20.39, LD-URL: 26.14, SLIDE: 0.00)
• Gemma3-12B: 23.51 (MMLB-D: 24.18, LD-URL: 46.37, SLIDE: 0.00)

An interesting anomaly: MMProLong's 512K overall average (52.52) is numerically very close to its 128K average (55.84 from Table 4) and 256K average (55.09), with the MMLongBench-Doc score actually increasing from 29.69 at 256K to 31.91 at 512K. This non-monotonicity suggests that the model's retrieval ability is genuinely robust to context length—at least within the 128K–512K range—and that small fluctuations reflect dataset variance rather than systematic degradation. The sharp drops seen in baseline models (particularly SlideVQA dropping from 68.00 at 128K to 20.41 at 512K for Qwen2.5-VL-7B) indicate that the base model's attention mechanisms break down entirely beyond a certain length threshold, while MMProLong's remain functional.

Important caveat. The 256K and 512K evaluations are constructed by padding with random negative documents, which means the effective context difficulty may not be as high as a naturally 512K document. The model only needs to locate the original evidence (which was present at 128K) among additional distractor pages, rather than handle genuinely longer-range dependencies. This is still a valid test of retrieval robustness—can the model find the needle when the haystack grows larger?—but it does not test whether the model can handle documents with genuinely longer-range dependencies (e.g., a 512K document with information scattered across all pages).

---

Generalization to Other Multimodal Long-Context Tasks

The paper tests MMProLong on three additional long-context evaluation suites without any task-specific training (Section 6.2, Figures 4 and 5, Appendices G.4–G.6).

MM-NIAH webpage needle-in-a-haystack (Figure 4, Table 17). Averaged across 64K and 128K contexts:

• Retrieval: MMProLong 66.33 (average of 74.83 at 64K and 57.83 at 128K) vs. Qwen2.5-VL-7B 30.67 (50.00+11.33 averaged)—gain of +35.7
• Counting: MMProLong 18.17 vs. Qwen2.5-VL-7B 11.17—gain of +7.0
• Reasoning: MMProLong 63.83 vs. Qwen2.5-VL-7B 18.17—gain of +45.7
• Overall average: MMProLong 49.43 vs. Qwen2.5-VL-7B 20.00—gain of +29.4

The enormous reasoning gain (+45.7) is particularly noteworthy because MMProLong's training data contains only 20% reasoning examples, yet the model's reasoning over webpage haystacks improves dramatically. This supports the paper's claim that the training teaches a general retrieval capability that transfers across domains—once the model can find the evidence, its base reasoning ability (inherited from Qwen2.5-VL-7B) can operate on it effectively.

VTCBench vision-text compression (Table 18). MMProLong improves the overall weighted score from 48.23 to 52.73 (+4.50):

• Retrieval: 91.75 vs. 91.63 (essentially unchanged—the base model already performs well)
• Reasoning: 22.88 vs. 15.63 (+7.25)
• Memory: 40.50 vs. 33.83 (+6.67)

The retrieval score is near-ceiling for both models, but reasoning and memory show substantial gains. VTCBench tests whether models can preserve information when visual context is compressed—a task that requires robust multimodal representations. The improvement suggests that long-document VQA training strengthens the model's ability to encode and retain visual-textual information even when the presentation format differs from training.

Long-video understanding (Figure 5, Table 19). Despite containing no video training data, MMProLong improves over the base model:

• Video-MME: 67.78 vs. 65.10 (+2.68)
• MLVU: 73.55 vs. 70.20 (+3.35)
• LongVideoBench: 62.08 vs. 60.43 (+1.65)

These are modest but consistent gains. The improvement is smaller than for document-based tasks (+2–3 points vs. +7 points on long-document VQA), which is expected given that video has substantial domain-specific characteristics (temporal redundancy, motion, frame-to-frame dependencies) not present in the document training data. The fact that any improvement is observed supports the paper's claim that the learned capability is general: better evidence retrieval and reasoning over long multimodal contexts transfers partially to the video domain.

Recipe transfer to Qwen3-VL-8B (Appendix G.7, Tables 20 and 21). The paper applies the final MMProLong recipe to Qwen3-VL-8B, a model that already has native 256K-context capability from its own large-scale long-context training (100B tokens of LongPT, plus SFT and RL optimization for long-document tasks). This is characterized as a "diagnostic" experiment rather than a strict extension study:

• Long-document VQA: 67.51 vs. 65.62 for the Qwen3-VL-8B base (+1.89)
• MM-NIAH: 61.75 vs. 50.03 (+11.72)

The MM-NIAH gain on this already-strong backbone is substantial (+11.72), suggesting that even models with extensive long-context training can benefit from the specific recipe—particularly the extraction-heavy, instruction-formatted document VQA data. The long-document VQA gain is smaller (+1.89), likely because Qwen3-VL-8B already performs well on this task (65.62 base vs. MMProLong's 57.70), leaving less room for improvement.

---

Ablation Studies and Robustness Checks

mRoPE base frequency (Appendix G.3, Table 16). The paper tests three values for the mRoPE base frequency when scaling from 32K to 128K: $2\times10^6$, $4\times10^6$ (the Dynamic-NTK default), and $8\times10^6$. The finding is that $2\times10^6$ and $4\times10^6$ perform comparably overall (e.g., 55.63 vs. 56.90 for extract-multi), while $8\times10^6$ does not yield consistent gains and sometimes degrades performance (reasoning drops from 56.47 with $4\times10^6$ to 54.34 with $8\times10^6$). The paper correctly concludes that "moderate mRoPE-base scaling is sufficient" and that the Dynamic-NTK heuristic, while reasonable, is not uniquely optimal for LVLMs with mRoPE.

Human verification of QA pair quality (Appendix D.4). The paper manually inspects 100 randomly sampled generated QA pairs across all three task types. Of these, 97 are fully correct, 2 contain incorrect answers, and 1 has an inaccurate evidence annotation. This 97% correctness rate supports the claim that the synthesis pipeline produces high-quality supervision, though the paper acknowledges that the generated data "may still contain a small amount of noise."

Document pool domain diversity (Appendix C.1, Figure 6). The pool spans 10 labeled domains (engineering: 12.7%, medicine: 12.0%, social science: 9.4%, biology: 7.2%, economics & business: 7.2%, physics: 7.0%, computer science: 6.8%, earth science: 5.7%, mathematics: 4.6%, language & literature: 3.5%) plus an "others" category (24.0%). This domain diversity is not explicitly ablated—the paper does not test whether training on a narrower domain distribution would hurt performance—but it serves as a robustness check for the synthesis pipeline's ability to handle varied document types.

The synthesis pipeline's anchoring mechanism (Section 4.2). The requirement that QA-generator models add explicit segment anchors to questions (e.g., "in the Introduction section," "on Pages 20-25") is introduced to prevent "global-context false positives." This is not ablated—we don't see what performance looks like without anchors—but the rationale is clearly motivated and the paper provides a concrete example of the failure mode (ambiguous revenue questions in financial reports) that anchors are designed to prevent.

Negative result: ReST$^{EM}$ degrades revision performance (Appendix K). Not applicable to this paper. The paper does not study revision models—this would be a negative result from a different paper (in the reference example).

---

Critical Assessment

The experimental design of this paper is straightforward and internally consistent: fix a 5B-token training budget, vary one design axis at a time, and measure downstream long-document VQA performance. This produces clear, interpretable results that support the paper's major claims. However, several limitations in the evaluation design constrain how broadly these findings should be interpreted.

Claim 1: "Long-document VQA is substantially more effective than OCR transcription." This claim is well-supported by Table 1. The VQA-trained variants consistently outperform the OCR-trained variants by large margins (17–24 points for OCR-full, 7–13 points for OCR-needle), and even with an additional SFT stage, OCR variants underperform VQA variants trained with half the budget. However, the comparison has an important asymmetry: the VQA tasks produce instruction-formatted data (question → answer), while the OCR transcription tasks produce a different format (page images → transcribed text). The paper attributes the VQA advantage to "instruction-formatted supervision and task diversity," but these two factors are confounded—VQA is both instruction-formatted and more diverse (extraction + reasoning vs. transcription). A cleaner comparison would include an OCR transcription variant reformatted as instruction-following (e.g., "Please transcribe the text from pages 3-5") to isolate the format effect from the task-type effect.

Claim 2: "Balanced sequence-length data outperforms target-length-focused data." This is supported by Figure 2 and Table 14, but the effect size is modest (+1.3, +0.1, +1.7 points across tasks). With the small test set (500 questions across three datasets, ~100 per dataset for SlideVQA), these differences may not be statistically significant. The paper does not report confidence intervals or standard errors, which makes it difficult to determine whether the observed differences are reliable or whether the honest conclusion should be that both distributions perform similarly. The long-biased distribution also uses longer documents (50–100 pages vs. 32–50) and produces longer average training instances (114.3K vs. 85.3K tokens), meaning the comparison confounds length distribution with document characteristics—the longer documents might be qualitatively different (more technical, more dense) in ways that affect learnability independent of length.

Claim 3: "Retrieval is the primary bottleneck, favoring retrieval-heavy mixtures." The grid search in Table 2 provides clear evidence that extraction-heavy ratios perform best, but the range of variation is narrow: the difference between the best (8:2, 57.70) and worst (4:6, 55.73) overall averages is only 1.97 points. This is a small effect relative to the uncertainty in the evaluation. The paper's conclusion that retrieval is the primary bottleneck is reasonable given the monotonic improvement as extraction ratio increases (with the exception of the 2:8 dip), but a stronger test would involve an explicit diagnostic: measure the base model's retrieval accuracy and reasoning accuracy separately on the long-context benchmarks, and show that retrieval accuracy is indeed lower and more improved by LongPT. Without this decomposition, the bottleneck claim is an interpretation of the mixture results rather than a directly measured phenomenon.

Claim 4: "Pure long-document VQA largely preserves short-context capabilities." This is the paper's most striking and potentially impactful claim, but it rests on a single set of measurements (Table 3) with the short-context average dropping from 66.47 to 65.48. The claim is technically true—the drop is small (~1 point)—but the per-benchmark results show meaningful variation: OCRBench drops 2.6 points (88.10→85.50), while General VQA improves by 1.6 points. This heterogeneous pattern suggests that "preserving short-context capabilities" is not uniform—some capabilities are preserved or improved, while others degrade. A more precise claim would be "long-document VQA preserves or improves some short-context capabilities while causing modest degradation in others, with the overall average largely maintained." The paper also does not compare against a control condition where the model is fine-tuned on short-context data only, which would establish a ceiling for short-context performance after 5B tokens of additional training.

Claim 5: "MMProLong generalizes to 256K and 512K contexts without additional training." Table 5 shows that MMProLong maintains 55.09 at 256K and 52.52 at 512K, compared to 55.84 at 128K—essentially flat. This is impressive, but the evaluation methodology at these longer lengths (padding with random distractor documents) means the test measures retrieval robustness against distractors rather than genuine longer-range understanding. A document that is natively 512K tokens long would have genuinely long-range dependencies—a reference on page 1 to a concept explained on page 400—that the padded evaluation does not replicate. The paper's claim should be qualified: MMProLong generalizes its retrieval ability to longer contexts with distractor documents, but whether it can handle genuinely long-range dependencies at 512K is untested.

Claim 6: "The recipe transfers to other backbones." Appendix G.7 shows that applying the recipe to Qwen3-VL-8B yields gains (+1.89 on long-document VQA, +11.72 on MM-NIAH). This is encouraging but limited: (a) it's a single additional backbone from the same model family (Qwen), so it doesn't demonstrate transfer across architectures; (b) Qwen3-VL already has extensive long-context training, so these gains are incremental improvements on top of a strong baseline, not a demonstration that the recipe works from scratch on a different architecture; (c) the MM-NIAH gain (+11.72) is much larger than the long-document VQA gain (+1.89), which is unexplained and might reflect Qwen3-VL having specific weaknesses on MM-NIAH-style tasks that happen to be addressed by the recipe, rather than a general benefit.

Missing experiments that would strengthen the paper:

1. Difficulty-stratified analysis. The paper does not analyze performance by question difficulty, document type, or evidence position within the context. We don't know whether MMProLong's improvements come primarily from better handling of easy questions, better retrieval of evidence at specific position ranges (beginning vs. middle vs. end of context), or better performance on specific document types. This limits our understanding of how the recipe improves performance.

2. Scaling with training budget. All experiments use a fixed 5B-token budget. The paper doesn't show learning curves—does performance continue to improve with more data, or does it saturate? The concurrent work by Veselka (2026) found that 1B-token LongPT outperformed 10B-token LongPT, which (though the paper argues this reflects alignment rather than extension) raises the question of whether MMProLong's 5B-token budget is near-optimal or whether different conclusions would emerge at different scales.

3. Comparison against simple baselines. The paper compares against OCR transcription but not against simpler alternatives: (a) training on the VQA data without the long-context embedding (just the 8-15 page segments), which would isolate the benefit of the full-context training; (b) training on synthetic long-context data (concatenated unrelated documents with a question about one of them), which would test whether genuine document coherence matters; (c) a RAG-like baseline that chunks documents and retrieves relevant segments, which would establish whether the long-context modeling provides benefits beyond what retrieval-augmented short-context models can achieve.

4. Ablation of the teacher model. All VQA data is synthesized using Seed 2.0. The paper doesn't test whether data synthesized by a weaker teacher (or by Qwen2.5-VL-7B itself) would produce different results. If the recipe depends on a strong proprietary teacher model, its reproducibility is limited.

5. Analysis of position-dependent retrieval accuracy. The claim that balanced length distributions teach "position-invariant retrieval" would be directly testable by measuring retrieval accuracy as a function of the evidence's position in the context (e.g., first 25%, second 25%, etc.). The paper doesn't conduct this analysis, so the mechanistic interpretation of the pool-native advantage remains speculative.

The single most important limitation is the absence of statistical rigor. With a 500-question test set split across three datasets and evaluated at 2–4 context lengths, many of the per-dataset and per-length comparisons involve small effective sample sizes. The differences between the best and second-best configurations in the mixture grid search (57.70 vs. 57.27, a gap of 0.43) and the length distribution comparison (+1.3, +0.1, +1.7) are small enough that statistical noise could plausibly explain them. Without confidence intervals or significance tests, the paper's conclusions about which specific configuration is "best" should be treated as provisional—the qualitative trends (extraction > reasoning, balanced > concentrated) are more reliable than the exact optimal ratios (8:2, 4×10^6 mRoPE base).

Despite these limitations, the paper's core empirical contribution—that long-document VQA with instruction-formatted, extraction-heavy, balanced-length data can substantially improve long-context LVLM performance under a modest training budget—is well-supported by the consistent pattern of results across multiple datasets and generalization benchmarks. The practical recipe is clear and actionable, and the qualitative design principles (instruction format prevents forgetting, balanced lengths teach generalization, retrieval is the bottleneck) are likely to transfer even if the exact numerical optimal configurations do not.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For

The assumption or constraint. The entire compute-optimal framework depends on estimating prompt difficulty before allocating the inference budget. The paper's method for doing so—generating 2048 samples per question and computing average PRM scores—is described as expensive, and the authors explicitly acknowledge this gap in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity."

The consequence. The headline efficiency gain—4× over best-of-N at equivalent accuracy—is computed after difficulty is known, without amortizing the cost of learning it. Generating 2048 samples costs substantially more than the largest test-time budgets studied (256–512 generations), meaning that in a realistic deployment, the total cost could be dominated by difficulty estimation. The practical deployment efficiency of compute-optimal scaling could be much lower than reported, potentially even worse than uniform best-of-N when the estimation overhead is included. For applications where each query is independent (no reuse of difficulty estimates across similar problems), this overhead makes the approach impractical as described.

What evidence exists in the paper. The paper does not measure or report the token cost of difficulty estimation for any experiment. The predicted difficulty bins (using PRM scores instead of ground-truth labels) remove the need for ground-truth access but still require the 2048-sample generation step. Figures 4 and 8 show that predicted bins track oracle bins closely, which is encouraging, but neither figure accounts for the cost of producing those bin assignments.

Mitigation status. The paper acknowledges this as "a key avenue for future work" in Section 3.2 and suggests in Section 8 that future work could "pretrain or finetune models to directly predict difficulty of a question." No such model is developed or evaluated in this paper. A more realistic near-term mitigation—adaptive difficulty estimation where the initial budget is partly spent to assess difficulty and the remainder is allocated accordingly—is mentioned as an exploration-exploitation tradeoff but not implemented. Until this gap is closed, the reported efficiency gains should be treated as an upper bound under idealized conditions.

---

The Revision Model Suffers a 38% Correct-to-Incorrect Reversion Rate

The assumption or constraint. The revision model is fine-tuned exclusively on sequences where all in-context answers are incorrect followed by a correct target (Section 6.1, Appendix H). This training data construction means the model never sees examples where the current answer is already correct and should be preserved. The consequence is that during inference, when the model generates a correct answer in a revision chain, it has no training signal for what to do next. The paper reports (Section 6.1):

"approximately 38% of correct answers get converted back to incorrect ones using a naive approach"

The consequence. Sequential revision chains are inherently fragile: even when the model produces a correct answer at some step, the next revision is more likely than not to corrupt it. Without mitigation, this means longer revision chains do not monotonically improve—the model can cycle between correct and incorrect answers, and the final answer in the chain is often worse than intermediate answers. This limits the effectiveness of purely sequential strategies and forces the system to rely on across-chain selection mechanisms (majority voting or verifier-based selection) that pick the best answer from any point in the chain, discarding the naive assumption that later revisions are always better.

What evidence exists in the paper. The 38% reversion rate is reported directly in Section 6.1. The per-step pass@1 trajectory in Figure 6 (left) corroborates this: accuracy increases from ~18.2% at step 1 to ~24–25% by steps 15–20, but it fluctuates and does not show a monotonic improvement—some steps degrade relative to the previous. The paper also reports (Section 6.2) that majority voting or verifier-based selection across the chain is necessary to mitigate the problem, and Figure 6 (right) shows that sequential with best-of-N weighted selection outperforms sequential with majority voting, consistent with the verifier helping to identify which revision step is correct.

Mitigation status. The paper partially mitigates this via chain-level selection: rather than always taking the final revision output, the system applies majority voting or verifier-based selection across all steps in the chain to pick the best answer (Sections 6.1 and 6.2). This is effective as a patch but is fundamentally compensating for a training data design flaw rather than solving the root cause. A more principled solution—training the model to recognize when no revision is needed, or including correct-to-correct and correct-to-incorrect trajectories in the training data—is not explored. The negative result with ReST$^{EM}$ in Appendix K (where additional sequential revisions "substantially hurt" performance) further underscores that the revision training procedure is fragile in ways that are not fully understood.

---

Coverage Is Limited to a Single Benchmark and Model Family

The assumption or constraint. All main experiments use the MATH benchmark (500 test questions) with PaLM 2-S* as the base model. The paper states in Section 4:

"We believe this model is representative of the capabilities of many contemporary LLMs"

but this claim is unverified. The FLOPs-matched comparison in Section 7 uses a second PaLM 2 model family variant, but no experiments are conducted on different model architectures, different reasoning benchmarks (code generation, logical reasoning, scientific QA), or different task modalities.

The consequence. Several aspects of the findings could be model- or benchmark-specific. The PRM's over-optimization behavior depends on PaLM 2-S*'s output distribution; a model with different calibration or error patterns might show different difficulty-dependent scaling curves. The revision model's ability to learn from paired incorrect-correct trajectories depends on the base model's in-context learning capabilities, which vary substantially across model families. The MATH benchmark consists entirely of competition-level math problems requiring symbolic reasoning—it is unclear whether the difficulty-dependent patterns (beam search degrading easy problems, revisions helping easy problems) generalize to tasks requiring factual recall, code generation, or open-ended reasoning. The optimal sequential-to-parallel ratios, the 4× efficiency gains, and the $14\times$ pretraining equivalence are all calibrated to this specific setting.

What evidence exists in the paper. The difficulty-dependent analysis is broken out across five MATH difficulty quintiles (Figures 3, 7), each containing ~100 test questions. The two-fold cross-validation within bins further halves the sample used for strategy selection to ~50 questions per fold per bin. The paper does not report confidence intervals on any result, making it impossible to assess whether the observed differences between strategies are statistically reliable at this sample size. No out-of-distribution evaluation (different benchmark, different task type) is conducted.

Mitigation status. The paper does not address this limitation beyond acknowledging it implicitly through the choice of a single benchmark and model family. The authors do not claim that the specific ratios, thresholds, or efficiency gains will transfer to other settings—the contribution is framed as a methodology and set of principles (adaptive allocation based on difficulty, the proposal-verifier decomposition, the existence of over-optimization regimes) rather than as universal constants. However, a practitioner deploying this approach on a different model or task would need to replicate substantial portions of the analysis pipeline (PRM training, difficulty estimation, strategy grid search) to determine whether the same patterns hold. The paper provides a blueprint but no out-of-the-box transfer guarantees.

---

Hard Problems Remain Unsolved—Test-Time Compute Cannot Create Capability

The assumption or constraint. The paper's framework assumes that the base model already produces correct solutions at some non-trivial rate—roughly, that pass@1 on a problem class is above zero by enough that sampling or search can surface correct answers. This is the implicit precondition for both the revision model (which needs correct answers to use as training targets) and the PRM (which needs correct Monte Carlo rollouts to compute soft labels). The paper acknowledges this boundary explicitly in the Section 7 takeaway box and results.

The consequence. On the hardest questions (difficulty bin 5 in the paper's taxonomy), no method—search, revisions, or their compute-optimal combinations—makes meaningful progress regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%, well below the $14\times$ larger model's performance. This establishes a hard capability frontier: test-time compute can amplify existing capability but cannot create it from nothing. For problems genuinely outside the base model's reach, pretraining remains the only viable path.

What evidence exists in the paper. The difficulty-quintile analysis in Figures 3 (right), 7 (right), and 9 provides consistent evidence across all three experimental axes (search, revisions, FLOPs comparison). In every case, bin 5 shows near-zero performance with minimal or no improvement from additional compute. The FLOPs-matched comparison (Figure 9 and Figure 1 bar charts) quantifies this: on hard problems at $R \gg 1$, test-time compute shows a −37.2% to −52.9% relative disadvantage versus the larger pretrained model, depending on the method (revisions or PRM search).

Mitigation status. The paper is transparent about this limitation—the Section 7 takeaway box explicitly states that test-time compute "cannot compensate for fundamental capability gaps that larger pretraining would address." Section 8 also flags this as an open challenge. However, the paper does not provide guidance on how to determine whether a given problem is within the base model's capability range without the expensive oracle difficulty estimation process (2048 samples). In a deployment setting, a user would need to know before allocating test-time compute whether the problem is in bin 5 (where the budget is wasted) or bins 1–4 (where it helps). The predicted difficulty bins partially address this but still require the same expensive estimation step.

---

The FLOPs-Matched Baseline Uses a Weak Pretraining Counterpart

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately $14\times$ more parameters. However, the larger model is only scaled in parameters—training data is held fixed—and it uses greedy decoding with no test-time compute augmentation of its own. The paper acknowledges this in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

The consequence. The $14\times$ larger model baseline is weaker than it could be in two ways. First, compute-optimal pretraining following Chinchilla scaling laws (Hoffmann et al., 2022) would scale both parameters and data, which would likely produce a stronger model at the same total FLOPs. Second, the larger model uses only greedy decoding—no majority voting, no best-of-N, and no search or revision mechanisms. Giving the larger model even a modest test-time compute budget (e.g., best-of-8 majority voting) would create a much stronger baseline, since the larger model would benefit from the same mechanisms the paper advocates for the smaller model. The reported advantages of test-time compute over pretraining (e.g., +27.8% on easy questions at $R \ll 1$) may shrink or reverse against a properly compute-optimal pretrained model that also receives some test-time augmentation.

What evidence exists in the paper. The FLOPs-matched results in Figure 9 and the bar charts in Figure 1 implicitly rely on this baseline choice. The paper does not include an ablation where the larger model receives any test-time compute budget. The three $R$ values (0.16, 0.79, 22) correspond to different inference-to-pretraining token ratios, but all of them assume the larger model uses greedy decoding. Appendix I of the paper (not shown in detail in the provided content) may contain sensitivity analysis.

Mitigation status. The paper is transparent about this design choice and frames it as "representative of a canonical approach to scaling pretraining compute," citing the LLaMA model series (Touvron et al., 2023) as precedent for parameter-only scaling. The authors explicitly flag the compute-optimal pretraining comparison as future work. However, the specific numerical claims in the abstract and introduction—that test-time compute can outperform a $14\times$ larger model—should be interpreted with the understanding that this is relative to a specific, sub-optimally trained baseline. A fairer comparison would give both models access to test-time compute and ask: for a fixed total FLOPs budget (pretraining + inference), what is the optimal split between pretraining scale and inference-time augmentation?

---

The Revision and Search Axes Are Never Combined, Capping the Demonstrated Gains

The assumption or constraint. The paper studies two independent mechanisms—PRM-guided search and iterative revisions—but never combines them into a unified system. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions."

The consequence. The paper's results represent a lower bound on what test-time compute can achieve. The two mechanisms have complementary strengths: revisions improve the proposal distribution (generating better-quality candidates), while PRM search improves candidate selection (finding the best among generated candidates). Applying beam search to revision model outputs—using the PRM to guide which revisions to pursue, or using revised answers as the proposal distribution for best-of-N and beam search—could yield gains beyond either method alone, particularly on medium-difficulty problems where both mechanisms individually show benefits. The compute-optimal policy presented in this paper is suboptimal in a broader sense because it never exploits the complementarity of the two axes—it selects between them per-difficulty-bin rather than combining them within a single query.

What evidence exists in the paper. The difficulty-dependent analysis in Figures 3 and 7 shows that revisions excel on easy problems (bin 1–2) while beam search excels on medium problems (bins 3–4). This complementarity is what would make combination powerful—a system could use revisions to refine candidate solutions and then use PRM search to select among the refined candidates. The paper presents these as independent scaling curves but never intersects them. There is no experiment where revision model outputs are scored by the PRM and fed into beam search, or where the PRM is used to decide which revision chain to continue versus restart.

Mitigation status. The paper acknowledges this gap explicitly in Section 8 and frames it as future work. The individual scaling curves in the paper can be interpreted as ablation points in a larger design space—revisions alone, search alone—but the interaction effect between them is unknown. The unimodal results suggest the combined effect could be significant (since the two methods help on complementary difficulty ranges), but this is speculative without empirical evidence.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around long-context vision-language model training from whether it's possible to how to do it efficiently. Before this work, practitioners extending LVLM context windows had two options: follow LLM recipes that don't account for interleaved image-text inputs and structured visual positional encodings, or guess. The paper's systematic ablation of training data design choices under a controlled 5B-token budget provides the first public, reproducible recipe grounded in controlled experiments rather than institutional knowledge.

The conceptual shift is from viewing long-context training as a capacity expansion problem (extending the context window so the model doesn't break at longer lengths) to viewing it as a retrieval generalization problem (teaching the model to find evidence regardless of where it sits in the input). This reframing emerges most clearly from the length distribution ablation in Section 5.1: concentrating training examples at 128K—the natural choice if you think of capacity expansion—actively hurts performance compared to a balanced distribution spanning 32K–128K. The paper's interpretation—that balanced lengths teach position-invariant retrieval rather than a specific length adaptation—changes what practitioners optimize for. Rather than asking "how many 128K examples do I need?", the question becomes "is my training distribution exposing the model to evidence at all possible positions and distances?"

The practical significance is amplified by what this reframing eliminates. The dominant LLM recipe (Gao et al., 2024) prescribes mixing short-context data to prevent forgetting. This paper shows that when your long-context data is instruction-formatted VQA, you can skip that step entirely (Section 5.3, Table 3), simplifying the training pipeline and freeing the full training budget for long-context data. This is not a minor efficiency gain—it changes the optimization landscape from a two-knob problem (long/short ratio, long-data composition) to a one-knob problem (long-data composition), reducing the number of ablations a practitioner needs to run by roughly half.

The work also provides a framework for diagnosing training data problems via negative results. The OCR transcription baseline in Section 4.4 serves as a cautionary example: an intuitively appealing supervision source (transcribe all the text on all the pages) that catastrophically degrades long-context VQA performance (Table 1: 50.59 → 33.17). The mechanism is instructive—dense image-text alignment teaches the wrong kind of attention, encouraging the model to attend to everything rather than to locate sparse evidence. This negative result provides a concrete diagnostic: if your long-context training data doesn't look like instruction-following retrieval, it's probably teaching the wrong skill.

The paper's generalization results (Sections 6.1 and 6.2) further shift expectations about what long-context training data needs to cover. MMProLong transfers to webpage-based needle retrieval, long-video understanding, and vision-text compression without any domain-specific training data. The implication is that high-quality document VQA data teaches a general long-context capability—not document-specific skills—reducing the pressure to curate domain-specific long-context datasets for every downstream application. A single well-designed 5B-token LongPT stage may suffice for broad coverage.

In terms of magnitude, this is an incremental but clarifying contribution to the scaling literature. It doesn't introduce new architectures, objectives, or theoretical frameworks. What it provides is a systematic empirical map of a previously underexplored design space, establishing both what works (extraction-heavy VQA with balanced lengths) and what doesn't (concentrated-length training, OCR transcription as sole supervision). This makes the work practically actionable in a way that broader "scaling is all you need" narratives are not.

The paper also implicitly resolves a tension in the concurrent long-context LVLM literature. Veselka (2026) found that 1B-token LongPT outperforms 10B-token LongPT, and that LongSFT outperforms LongPT—findings that seem to contradict the premise that continued pre-training matters. This paper's distinction between context alignment (tuning a model that already supports 128K) and context extension (genuinely expanding from 32K) reconciles these results: Veselka's findings may reflect saturation on an already-long-context backbone, while MMProLong's 5B-token recipe addresses the harder problem of teaching a model to handle positional regimes it has never seen.

Research directions that become more attractive after this work:

• Efficient difficulty estimation and adaptive allocation for LVLMs. The paper's balanced-length finding implies that position matters as much as length; extending the compute-optimal test-time scaling frameworks to LVLMs would need position-aware difficulty metrics.
• Cross-domain long-context transfer studies. The generalization to video without video training data is promising but unexplained—mechanistic studies of how document-trained attention patterns transfer to video could inform more efficient multi-domain training.
• Verifier-based training data filtering. The paper's finding that instruction-formatted data preserves short-context performance suggests that data format, not just content, determines forgetting dynamics—verifier models that assess the "instruction-following quality" of long-context data could automate recipe design.

Research directions that become less attractive:

• Dense image-text alignment as a primary long-context objective. The OCR transcription results in Table 1 are so negative (17.4-point drop for OCR-full) that they effectively rule out pure transcription as a viable LongPT strategy, absent major innovations.
• Target-length-concentrated training for context extension. The long-biased vs. pool-native comparison (Figure 2, Table 14) provides clear evidence against this approach, at least within the 32K–128K range, reducing the motivation for complex length-based data reweighting schemes.

Follow-Up Research This Work Enables

Position-dependent retrieval accuracy on long-document benchmarks. The paper's central mechanistic claim—that balanced length distributions teach position-invariant retrieval—is supported by inference from aggregate results (pool-native outperforms long-biased) but never directly measured. A follow-up study would instrument the MMLongBench evaluation to track evidence position as a covariate: for each test question, annotate where in the document sequence the answer evidence appears (e.g., first quartile, second quartile, etc.) and measure per-quartile accuracy. The strong prediction from the paper's interpretation is that pool-native-trained models should show flatter accuracy curves across position quartiles compared to long-biased-trained models, which would show a U-shaped or skewed pattern reflecting their training distribution. This experiment requires modest annotation effort (adding position metadata to existing test questions) and would convert the paper's mechanistic interpretation into a directly verified, rather than inferred, phenomenon. A negative result—no position-dependent difference between pool-native and long-biased—would force a revision of the "position-invariant retrieval" explanation and redirect attention to other confounds (document difficulty, domain diversity, noise characteristics at different lengths).

Scaling laws for multimodal long-context training token budgets. This paper uses a fixed 5B-token budget for all ablations, providing a single point estimate of recipe effectiveness. A systematic scaling study would train MMProLong variants at 1B, 2.5B, 5B, 10B, and 20B token budgets (all using the final recipe: pool-native distribution, 8:2 extraction-to-reasoning, 0% short-context data) and measure long-document VQA accuracy at each budget, fitting a power-law or logarithmic scaling curve. This would answer the practical question this paper leaves open: is 5B tokens near-optimal, or would doubling the budget yield substantial further gains? The concurrent Veselka (2026) finding that 1B-token LongPT outperforms 10B-token LongPT on an already-long-context backbone makes this particularly interesting—does that saturation pattern also appear for genuine context extension (32K→128K), or does the harder problem require more data? Conducting this on Qwen2.5-VL-7B (to match the current paper) and on a 32K variant of a different model family (e.g., a hypothetical InternVL3-8B truncated to 32K) would also test whether the scaling behavior is architecture-specific.

Domain-stratified analysis of long-document VQA improvements. The paper's document pool spans 10 labeled domains (Appendix C.1, Figure 6) plus a large "others" category, but all results report aggregate scores. A domain-stratified evaluation would reveal whether MMProLong's improvements are uniform or concentrated in specific document types. For example, do the gains come primarily from engineering and medicine documents (which dominate the training pool at 12.7% and 12.0% respectively) or are smaller domains like mathematics (4.6%) and computer science (6.8%) seeing proportional benefit? If gains are highly correlated with domain representation in training, this would suggest that domain diversity matters more than the paper implies, and practitioners curating document pools should prioritize domain breadth over raw volume. If gains are uniform across domains, this strengthens the claim that the recipe teaches general retrieval capability. This experiment requires no model retraining—just splitting the existing MMLongBench evaluation by document domain metadata (which the benchmarks presumably include) and re-analyzing the existing MMProLong results.

Ablation of the teacher model strength on downstream LongPT effectiveness. All VQA training data in this paper is synthesized by Seed 2.0, ByteDance's proprietary multimodal model. This creates a reproducibility and cost concern: can researchers without access to Seed 2.0 replicate the recipe using weaker open-source teacher models? A systematic comparison would synthesize equivalent 5B-token VQA training sets using teacher models of varying strength: (a) Seed 2.0 (the original, as a ceiling), (b) a strong open-source model like Qwen2.5-VL-72B, (c) the base model itself (Qwen2.5-VL-7B), and (d) a substantially weaker model (e.g., Qwen2-VL-7B). Each synthetic dataset would be used to train MMProLong with the final recipe, and downstream performance would reveal how much the teacher model's quality matters. The paper claims the pipeline produces high-quality data (97% correct in manual inspection of Seed 2.0 outputs), but the 3% error rate might increase with weaker teachers, and this noise could interact with training dynamics in non-obvious ways. The hypothesis from the current paper's results is that teacher model strength matters but the pipeline is robust to moderate degradation—but this is entirely untested.

Combined PRM-guided search and iterative revision for long-context VQA. This paper's prior sections established that search and revisions have complementary difficulty-dependent strengths, but this paper itself studies long-context training, not inference-time strategies. An ambitious integration experiment would apply the compute-optimal test-time scaling framework (from the prior paper) to MMProLong: use the trained model as the proposal distribution, train a PRM on its outputs (following the Monte Carlo rollout procedure), and test whether adaptive allocation between best-of-N and sequential revision strategies further improves long-document VQA performance within a fixed inference budget. Concretely: for each MMLongBench test question, estimate difficulty (using MMProLong's pass@1 rate, with the PRM score as a practical proxy), then apply the optimal strategy per difficulty bin. The prediction would be that easy questions benefit from revisions (local refinement of answers), medium questions benefit from beam search against the PRM, and hard questions benefit from best-of-N with a large sample budget—the same qualitative pattern from the prior paper, but applied to a model whose long-context retrieval was explicitly trained rather than just prompted. This experiment would test whether the two papers' contributions are additive or redundant.

Negative result: testing whether MMProLong genuinely handles long-range dependencies at 512K. The paper's 256K and 512K evaluations (Table 5) demonstrate retrieval robustness against distractor documents but do not test genuine long-range dependencies—because the evaluation is constructed by padding with unrelated negatives, not by creating documents with evidence scattered across the full 512K span. A stress-test would construct a "long-range MMLongBench" variant where the evidence for each question is deliberately split across pages spanning the full context length (e.g., for a 512K document, half the evidence on page 2 of 200 and half on page 198 of 200). If MMProLong maintains its 52.52 average on this variant, the generalization claim is genuinely strong. If performance collapses, the current results reflect only distractor-robust retrieval (finding a local needle in an expanded haystack) rather than true long-range attention (connecting evidence across distant positions). This negative result would importantly qualify the paper's generalization claims and redirect research toward explicit long-range dependency training rather than relying on balanced length distributions to teach this capability implicitly.

Practical Applications and Downstream Use Cases

Document understanding pipelines with resource-constrained inference. Organizations building document Q&A systems on proprietary document collections (legal contracts, financial reports, academic literature) can apply the MMProLong recipe to extend a 7B open-source LVLM from 32K to 128K context for approximately 2,900 H20 GPU hours (Table 6). The resulting model achieves 57.70 on MMLongBench document VQA—competitive with or exceeding several open-source models 4–5× larger (InternVL3-38B: 48.88; Gemma3-27B: 52.63; Table 4). This enables on-premise or limited-GPU deployments where running a 70B model is infeasible but long-document capability is required. The ~1-point short-context degradation (Table 3: 65.48 vs. 66.47 base) means the same model can serve both long-document and general VQA workloads without maintaining separate deployments. The specific recipes—8:2 extraction-to-reasoning mixture, pool-native length distribution, no short-context mixing—are directly implementable given a document pool and a strong teacher model for data synthesis.

Video understanding without video-specific training data. MMProLong improves long-video benchmarks (Video-MME: +2.7, MLVU: +3.4, LongVideoBench: +1.6; Figure 5, Table 19) despite containing zero video training examples. For organizations that need basic long-video understanding capability (meeting summarization, lecture Q&A, security footage review) but lack the resources to curate large-scale video-instruction datasets, training on document VQA data provides a surprisingly effective bootstrap. The gains are modest (+1–3 points) compared to the document domain improvements (+7 points), so this is a practical starting point rather than a complete solution—but it means a single 5B-token LongPT stage yields improvements across documents, webpages, and video, reducing the need for separate domain-specific training pipelines.

Cost-efficient long-context model development for new LVLM backbones. The recipe transfer experiment in Appendix G.7 (Tables 20, 21) suggests the MMProLong recipe improves models beyond Qwen2.5-VL-7B. The recipe's components—extraction-heavy VQA data, balanced length distribution, instruction-formatted synthesis—are not backbone-specific. Research groups developing new LVLM architectures or training new base models from scratch can apply this recipe as a standardized long-context training stage rather than designing their own from scratch. The 5B-token budget is modest enough to serve as a diagnostic: if applying the recipe to a new backbone yields no improvement or degradation, that signals a fundamental architectural limitation in the backbone's ability to handle long multimodal contexts, providing actionable feedback earlier in the development cycle than waiting for full-scale long-context training.

When to Prefer This Method

The paper explicitly articulates a tradeoff between long-context performance and short-context preservation that maps directly to two deployment scenarios (Section 5.3, final paragraph):

• Prefer pure long-context training (0% short data) when: maximizing long-document VQA performance is the primary goal and short-context degradation of ~1 point (Table 3: 66.47 → 65.48 aggregate) is acceptable. This is the default MMProLong recipe and is appropriate for long-document Q&A systems, long-video analysis, and retrieval-heavy applications where the model will primarily process extended multimodal inputs.

• Prefer the 40% short-data mixture when: strong short-context preservation is required and a smaller long-context tradeoff is acceptable. This variant achieves 57.01 on long-document VQA (vs. 57.70 for pure long) and 66.14 on short-context benchmarks (vs. 65.48 for pure long, and close to the base model's 66.47). This is appropriate for general-purpose deployments where the same model serves both short-context tasks (single-image VQA, standard benchmarks) and long-context tasks (document analysis, video understanding). The 40% setting preserves the ~7-point long-document improvement over the base model (57.01 vs. 50.59) while nearly eliminating short-context regression.

The paper does not articulate a tradeoff between retrieval-heavy and reasoning-heavy mixtures for different application profiles, nor does it recommend different length distributions for different deployment contexts—these axes are treated as universally optimal at the configurations found in ablations rather than as application-dependent knobs. The recipe transfer to Qwen3-VL-8B (Appendix G.7) is characterized as a diagnostic validation, not as an A/B comparison that would support a "prefer this backbone over that one" recommendation.