2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

🎯 Pitch

Turning 22,000 hours of instructional videos into a textbook corpus dramatically boosts vision-language models’ reasoning—delivering over 20% gains on ScienceQA—because the sequential, logically coherent frames teach interleaved understanding in a way that webpage-crawled datasets never could.

1. Executive Summary

This paper introduces a multimodal textbook corpus for vision-language pretraining, constructed by systematically harvesting 2.5 years of instructional videos (22,000 class hours) and transforming them into an image-text interleaved dataset of 6.5 million keyframes and 0.75 billion text tokens. Training LLaVA-1.5-7B and Idefics2-8B on this textbook yields substantial gains on knowledge- and reasoning-intensive benchmarks—for example, ScienceQA accuracy improves by over 20% compared to MMC4, and MathVista gains +5.3% over OBELICS—while also enhancing few-shot in-context learning to the point where a "cheat test" (inserting the test sample itself into the prompt) reveals near-perfect context awareness on MathVision (98.4% vs. 66.5% for OBELICS). The paper establishes that video-sourced, temporally coherent interleaved sequences provide a training signal that webpage-crawled interleaved corpora fundamentally lack, showing that disrupting image order causes severe performance degradation only on the textbook dataset—confirming that the benefits depend on the preserved sequential coherence inherent in instructional video frames.

2. Context and Motivation

The Core Problem: Existing Multimodal Pretraining Data Is a B-Student's Notes

The paper identifies a fundamental misalignment between how we train vision-language models (VLMs) and what we want them to learn. Current VLMs are predominantly trained on two types of multimodal data: image-text pairs (e.g., COYO-700M, LAION-5B) that teach the model to align a single image with a single caption, and webpage-crawled interleaved corpora (e.g., MMC4, OBELICS) that intersperse multiple images within longer text passages. The problem is that neither format captures the kind of multimodal experience that would teach a model to reason through complex, knowledge-dense content — the way a human student reads a textbook where diagrams build on each other and text provides step-by-step explanation.

The paper frames this as a gap between the form and the substance of multimodal training data. The interleaved format is structurally attractive because it mirrors how VLMs will be used at inference time — with multiple images embedded in long-form textual context for tasks like multi-image comparison, chain-of-thought visual reasoning, and few-shot in-context learning. But the content of existing interleaved datasets is, in the authors' diagnosis, systematically deficient along three specific axes (Section 1, Fig. 1):

Loose text-image relation. Webpages frequently contain images that are tangentially related or entirely irrelevant to the surrounding text — logos, advertisements, decorative photographs, or stock imagery that adds no semantic content. Training on such data teaches the model that images and text are only weakly coupled, when in fact the VLM's job is to integrate them precisely.
Lack of logical coherence in image sequences. Most webpages contain relatively few images, and even when multiple images are present, they rarely form a narrative or explanatory sequence. The logical relations between consecutive images — is image B a zoomed-in detail of image A? A counterexample? A later stage in a process? — are typically vague or nonexistent. This means that the very skill interleaved training is supposed to instill (reasoning across multiple images) is not actually practiced during pretraining.
Low knowledge density. Crawled webpages inevitably include entertainment, news, product listings, and other content that, while linguistically well-formed, contains little systematic foundational knowledge. A VLM pretrained on such data may learn language fluency and superficial image-text correlation, but it will not acquire the structured factual and procedural knowledge needed for tasks like science question-answering or mathematical problem-solving.

This diagnosis is not merely a critique — it is a functional explanation for why VLMs, despite being trained on interleaved data, still underperform on benchmarks that require genuine multimodal reasoning (ScienceQA, MathVista) and exhibit limited in-context learning capabilities compared to their text-only LLM counterparts. The format was right, but the content was hollow.

Why This Matters: The Pretraining Bottleneck in VLM Development

The importance of this problem stems from the two-stage paradigm that dominates VLM development (Section 2.2): (1) pretraining on large-scale multimodal data to establish fundamental vision-language alignment, followed by (2) instruction fine-tuning on curated task-specific data. The instruction-tuning stage is well-studied, with a proliferation of high-quality instruction datasets (LLaVA-665K, etc.) and targeted techniques (high-resolution encoding, diverse task formatting). But the pretraining stage — which determines the knowledge foundation upon which instruction tuning builds — has received far less systematic attention.

This asymmetry matters because pretraining sets the ceiling for what instruction tuning can achieve. If the base VLM did not learn foundational physics concepts during pretraining, no amount of instruction-following data about physics problems can fully compensate — the knowledge must be present in the model's weights from pretraining, with instruction tuning primarily teaching the model how to access and deploy that knowledge in the appropriate format.

The paper draws an explicit analogy (Section 1) to Microsoft's Phi-series models in the text-only domain, which demonstrated that "textbooks are all you need" — a small model pretrained on high-quality, knowledge-dense synthetic textbook data can match or exceed much larger models trained on noisier web corpora. This insight had not been ported to the multimodal domain because there was no equivalent "multimodal textbook" — no large-scale dataset that combines the interleaved format with textbook-quality content. The paper's intervention is to provide exactly that, arguing that the VLM field is currently in a pretraining data quality bottleneck analogous to where LLMs were before the release of curated corpora like the Phi training sets.

There is also a practical deployment dimension. As VLMs move toward real-world applications — education, scientific analysis, technical documentation — their ability to reason through multi-step visual explanations becomes critical. A student asking a VLM to explain a geometric proof expects the model to understand how each diagram relates to the next, not just to caption isolated figures. The existing pretraining data cannot support this capability because it never required the model to learn from coherent visual narratives.

Prior Approaches and Where They Fall Short

The paper situates itself against two established lines of work, each with identifiable limitations.

Image-Text Paired Datasets

Datasets like COYO-700M, LAION-5B, and various captioning corpora (Section 2.2, Table 1) provide massive scale — billions of image-caption pairs that enable rapid visual-textual alignment. The standard VLM architecture (vision encoder → connector → LLM) is typically first trained on such paired data to map visual features into the LLM's embedding space. This works efficiently for teaching the model to see — to recognize objects, scenes, and basic visual attributes.

But paired datasets are structurally impoverished in two ways:

Single-image, single-caption format. Each sample contains exactly one image with a short, often superficial caption (average text tokens: 16 for COYO, 27 for LAION). There is no multi-image reasoning, no extended explanatory context, no interleaving of visual and textual information that builds knowledge incrementally. A VLM trained exclusively on paired data learns to caption images but not to think across images.
Low knowledge ceiling. Caption datasets describe what is visually present, not why it matters. An image of a pendulum appears with "a clock pendulum swinging back and forth," not with "the period T of a simple pendulum depends only on its length L and gravitational acceleration g, as shown by the formula $T = 2\pi\sqrt{L/g}$ ." The explanatory content that transforms visual perception into knowledge acquisition is absent.

The consequence (as the paper notes) is that VLMs trained predominantly on paired data show limited in-context learning and chain-of-thought reasoning capabilities. They can answer "what is in this image?" but struggle with "given these three diagrams, explain why the acceleration changes." The knowledge simply is not in the pretraining data.

Webpage-Crawled Interleaved Datasets

Recognizing the limitations of paired data, several groups introduced interleaved corpora — MMC4 (Zhu et al., 2023), OBELICS (Laurençon et al., 2023), and the more recent OmniCorpus (Li et al., 2024). These datasets crawl webpages and documents, extracting images embedded within text passages and preserving their relative positions. This was a genuine advance: the interleaved format enables training on multi-image sequences with longer textual context (averaging 417 text tokens in MMC4, 816 in OBELICS), which in principle should improve in-context learning and multi-image reasoning.

The paper acknowledges these contributions but argues that the webpage as data source introduces intrinsic quality limitations that fundamentally limit what interleaved pretraining can achieve (Fig. 1, Table 1):

Weak image-text coupling. The authors quantify this through their proposed In-sample Image Similarity (InSI-SIM) metric (Section 4.2, Table 1), which measures both semantic (CLIP score) and structural (SSIM) similarity between images within the same sample. For MMC4, InSI-SIM averages 0.319 (L=4 to L=8 image subsets). For OBELICS, it is 0.345. For the paper's textbook, it is 0.686 — more than double. This is not a marginal difference; it reflects a qualitative difference in how images relate to each other and to the surrounding text. Webpage images are often decorrelated, while instructional video frames are causally linked.
Image sequence incoherence. The paper provides a striking demonstration through the image shuffling experiment (Section 5.3, Fig. 3). When the image order within samples is randomly shuffled, webpage-centric datasets like MMC4 show essentially no performance degradation — the images had no meaningful sequential dependency to begin with, so shuffling them does not destroy any useful signal. OBELICS shows only a moderate decline. In contrast, the paper's textbook exhibits a severe and monotonic performance drop as the shuffling ratio increases, confirming that the sequential coherence of video frames carries critical training signal that webpage datasets lack entirely.
Superficial content. The authors observe (Section 1) that crawled webpages are dominated by news, product descriptions, entertainment, and advertising — content that is linguistically fluent but knowledge-sparse. Even when a webpage happens to contain educational content, it is embedded in a sea of navigational text, promotional links, and unrelated images. The result is that knowledge density — the proportion of tokens that convey structured factual or procedural knowledge — is low, forcing the model to waste pretraining compute on correlating images with irrelevant filler text.

The paper's critique is thus not that interleaved datasets are useless — they are clearly better than paired-only training — but that the webpage as a source fundamentally constrains the quality ceiling. Crawling more webpages may produce larger datasets, but it will not solve the problems of loose image-text relations, incoherent image sequences, or low knowledge density, because these are inherent to the nature of web content. What is needed is a fundamentally different source of interleaved multimodal data.

Instructional Videos: An Underexplored Goldmine

The paper identifies online instructional videos — YouTube mathematics courses, physics lectures, chemistry tutorials — as a data source that solves all three of the above problems simultaneously (Section 3, Fig. 2):

Intrinsic image-text alignment. In an instructional video, the speaker's narration directly explains the visual content on screen — a slide showing complementary angles, an animation of water infiltration, a step-by-step algebraic derivation. The images and text are not loosely correlated but causally linked: the text was produced to explain the images, and the images were designed to illustrate the text. This is the multimodal equivalent of a textbook, not a scrapbook.
Natural image sequence coherence. Video frames unfold in chronological order, each building on the previous. A physics instructor might first show the pendulum at rest, then at maximum displacement, then annotate the forces, then derive the equation — a four-frame sequence with explicit logical connections (initial state → extreme case → annotation → formalization). The frame-to-frame similarity (both semantic and structural) is high but not redundant — exactly the property that enables learning of progressive reasoning.
High knowledge density. Unlike entertainment or news videos, instructional videos are designed to convey knowledge. The instructor's speech contains definitions, derivations, examples, and explanations. The visual content contains diagrams, formulas, and demonstrations. Both channels — auditory (ASR-transcribed) and visual (keyframes, OCR-extracted text) — are knowledge-rich, and the combination of the two provides the kind of rich, redundant, multi-channel learning signal that humans use when studying from lectures.

Despite these advantages, the paper notes that instructional videos have been largely overlooked for VLM pretraining (Section 1). Prior video-language datasets (HowTo100M, etc.) focused on task-oriented instructional videos (cooking, repair), not foundational academic subjects. The reason is partly technical: extracting clean, aligned keyframes and text from raw video at scale requires a nontrivial pipeline (ASR transcription with refinement, keyframe detection, OCR extraction, multi-level filtering) that the paper itself had to develop. But the underlying opportunity — millions of hours of expert-created educational content freely available on platforms like YouTube — represents a massive, untapped resource for improving VLM knowledge capabilities.

How the Paper Positions Itself

The paper frames its contribution not as a new VLM architecture or training algorithm, but as a data-centric intervention that fills a specific gap in the multimodal pretraining ecosystem (Section 1, final paragraph). The positioning has three key aspects:

1. Textbook-quality interleaved data as a new category. The paper introduces the concept of a "multimodal textbook" — an interleaved corpus where the images and text are not merely co-occurring but pedagogically structured. This is analogous to the "Textbooks Are All You Need" philosophy in the LLM domain (Gunasekar et al., 2023; Li et al., 2023), but with the critical addition of multimodal coherence — the images must teach, not just decorate. The paper explicitly positions the dataset as filling a gap between image-text pairs (structurally too simple) and webpage interleaved corpora (structurally appropriate but content-deficient).

2. Video as a pretraining source, not just a downstream modality. Prior work has used video extensively for video understanding tasks (Video-MME, video QA, etc.), but the paper argues that instructional video should be treated as a pretraining resource for general VLMs, not just as task-specific training data. The key insight is that by extracting keyframes and aligning them with ASR-derived text, video can be converted into the same interleaved image-text format that current VLM architectures already consume — no architectural modification is needed to benefit from the video's superior coherence and knowledge density.

3. The coherence of the interleaved context itself is what matters. The paper provides empirical evidence (through the shuffling experiment and the "cheat test" in Section 5.3) that VLMs pretrained on the textbook learn to genuinely attend to their interleaved context — to notice that an identical question has appeared in the prompt, to follow the logical progression of diagrams, to use visual cues from few-shot examples. Prior interleaved datasets, by contrast, trained models that largely ignored their context (evidenced by the low cheat test scores for MMC4 and OBELICS in Table 4). The paper thus shifts the conversation from "interleaved data is better" to "the right kind of interleaved data is better" — and defines "right kind" in terms of the coherence and pedagogical structure that only instructionally-designed video sequences provide.

This positioning has practical implications for how the field approaches pretraining data curation. Rather than scaling up web crawling (more data, more compute) — which the paper's evidence suggests would hit diminishing returns because the quality ceiling is low — the paper advocates for curating smaller but higher-quality sources where each training sample carries genuine instructional value. The 610K samples in their textbook (a modest size by web-scale standards) significantly outperform comparable quantities of MMC4 and OBELICS data, demonstrating that in multimodal pretraining, as in text-only pretraining, data quality can substitute for data quantity when the quality gap is sufficiently large.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a multimodal pretraining dataset, not a new model architecture or training algorithm. The system is a data curation pipeline that transforms raw instructional videos from the internet into a high-quality image-text interleaved corpus — essentially converting video lectures into a structured "textbook" format that existing VLMs can consume during standard pretraining. The problem it solves is the quality deficit in current interleaved pretraining data: webpage-crawled corpora like MMC4 and OBELICS provide the right format (images interspersed with text) but the wrong content (loose image-text relations, incoherent image sequences, low knowledge density). The "shape" of the solution is a three-stage pipeline — (1) video collection guided by an LLM-generated knowledge taxonomy, (2) multi-level extraction of visual and textual content from raw video, and (3) multi-level filtering to remove noise at each granularity (video, clip, keyframe) — producing a dataset where images and text are causally linked, image sequences are logically ordered, and every sample carries dense instructional knowledge.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components arranged in a sequential pipeline:

Knowledge Taxonomy Generator — an LLM (GPT-4o) prompted to recursively expand a four-level hierarchy (Subject → Course → Sub-course → Knowledge Point) covering 6 foundational subjects, 55 courses, and 3,915 knowledge points. This taxonomy serves as the structured search query space for video collection.
Video Collector and Metadata Filter — uses each knowledge point as a YouTube search query, retrieves the top 50 videos per query, deduplicates by video ID, and filters out irrelevant or inappropriate content by prompting an LLM to review video metadata (title, description, comments). The output is ~159K raw instructional videos.
Video-Level Extractor and Filter — converts video audio to text via ASR (Whisper-large-v3), refines the raw ASR transcriptions for fluency using Qwen2-72B-Instruct, and filters out non-instructional videos by having two LLMs (DeepSeek-V2, Llama3-70B-Instruct) score each ASR transcription on relevance, knowledge density, and transcription quality. The output is ~75K high-quality videos (22,697 class hours).
Clip-Level Extractor and Filter — uses ASR timestamps to segment long videos into short clips (10–20 seconds each, paired with their ASR segment), merges incomplete ASR fragments into coherent paragraphs, generates captions for each clip using VideoLlama2-7B, and discards clips whose captions are textually dissimilar to their ASR (e.g., clips showing only the speaker rather than instructional visuals). The output is ~4M video clips with aligned visual-auditory content.
Keyframe-Level Extractor and Filter — detects representative frames from each clip using SSIM-based change detection, extracts on-screen text/symbols/formulas from keyframes via OCR (InternVL2-40B), scores each keyframe for informativeness, and removes OCR results redundant with previous frames. The final output is organized into the interleaved format {frame₁, frame₂, ocr₁, asr₁, asr₂, …} in chronological order, yielding 610K samples (6.5M keyframes, 0.75B text tokens).

Information flows strictly forward: taxonomy → videos → ASR segments → clips → keyframes + OCR → interleaved samples. Filtering occurs at every stage to progressively refine quality.

3.3 Roadmap for the Deep Dive

First, the knowledge taxonomy (Section 3.1), because it is the seed from which the entire video collection process grows — it determines what content enters the pipeline and ensures broad subject coverage.
Second, the video collection and metadata filtering (Section 3.1), because it bridges the taxonomy (abstract knowledge structure) to concrete video URLs — this is where scale (~159K videos) is established.
Third, the video-level extraction and filtering (Section 3.2), because ASR transcription is the first transformation of raw video into usable text and the first quality gate (only ~75K of 159K videos survive).
Fourth, the clip-level extraction and filtering (Section 3.2), because temporal segmentation and visual-auditory alignment happen here — this is where long, unstructured video becomes discrete, aligned (clip, ASR) units.
Fifth, the keyframe-level extraction and filtering (Section 3.2), because this is the final refinement step that converts continuous video into sparse, representative image frames with extracted textual annotations (OCR) — the direct analogue of textbook figures and sidebars.
Sixth, the sample construction and dataset statistics (Section 3.2, bridging to Section 4), because this shows how the pipeline's output is assembled into the final pretraining corpus format.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a data engineering paper whose core idea is that instructional videos, when processed through a carefully designed extraction and filtering pipeline, can be converted into a multimodal pretraining corpus with qualitatively better image-text alignment and inter-image coherence than any existing interleaved dataset sourced from webpages.

Knowledge Taxonomy Construction

The pipeline's starting point is not raw video but a structured knowledge taxonomy that defines what content to search for. The authors prompt an LLM (explicitly GPT-4o, per Section 7.1) to construct a four-layer hierarchical taxonomy across multiple educational stages (primary school through middle school) and six foundational subjects: mathematics, physics, chemistry, earth science, engineering, and computer science (Section 3.1, Section 7.4).

The taxonomy follows the structure:

$\text{Subject} \rightarrow \text{Course} \rightarrow \text{Sub-course} \rightarrow \text{Knowledge Point}$

where each layer is a refinement of the previous one. For the mathematics example given in the paper:

Subject: Mathematics
Course: Elementary Mathematics
Sub-course: Rational and Irrational Numbers
Knowledge Point: the definition of Irrational Numbers

The LLM spans the taxonomy automatically to cover 6 subjects, 55 courses, and 3,915 knowledge points (Section 3.1, Section 7.4, Table 7). The courses under each subject are listed in Figure 4 of the supplementary material — for example, mathematics includes Algebra, Solid Geometry, and others; physics includes mechanics and electromagnetism courses; computer science includes algorithms and data structures.

Why a taxonomy? The paper could have simply searched for "math tutorial" or "physics lesson" and collected whatever videos appeared. But this would introduce two problems: (1) coverage bias — popular topics would be overrepresented while niche topics would be missed, and (2) query drift — vague search terms retrieve vaguely relevant content. By constructing an exhaustive taxonomy and using each knowledge point as a precise search query, the paper ensures systematic coverage of each subject's curriculum and retrieves videos that are specifically targeted to each atomic knowledge concept. This is analogous to building a textbook's table of contents before writing the chapters — it guarantees the final dataset spans the intended educational scope rather than reflecting what was easiest to find.

Why an LLM rather than manual curation? Manually constructing a 3,915-point taxonomy across six subjects would require domain experts and substantial time. The LLM, prompted to recursively expand the hierarchy from high-level subjects down to granular knowledge points, can generate this structure automatically. The authors do not provide the exact prompt, but the process is described as instructing the LLM to "span the proposed knowledge taxonomy so that multiple educational stages (from primary school to middle school) and diverse subjects (mathematics, physics, etc.) will be involved" (Section 3.1). This is a now-standard use of LLMs for structured data generation — using their pretrained knowledge of curriculum structure to produce a plausible coverage of each subject's topics.

Video Collection and Metadata-Level Filtering

With the taxonomy constructed, each of the 3,915 knowledge points is used as a keyword search query to retrieve instructional videos via YouTube's search API (Section 3.1). The authors retain the top 50 videos for each knowledge point, yielding an initial pool of approximately $3915 \times 50 = 195,750$ videos before deduplication.

Why top 50? The paper does not explicitly justify this number, but it represents a tradeoff: too few videos per knowledge point risks missing high-quality content, while too many increases the downstream processing cost without proportional quality gain. The search API ranks videos by relevance, so the top results are the most likely to be genuinely instructional for the target knowledge point. The exact count of 50 likely reflects a practical compute budget constraint — processing 195K videos end-to-end through ASR, clip segmentation, and keyframe extraction is already substantial.

Deduplication and metadata filtering. After retrieval, the authors perform two operations before any video content is downloaded or processed:

Deduplication by video ID. Because the same video may appear in search results for multiple knowledge points (e.g., a general "introduction to algebra" video may be retrieved for "solving linear equations," "graphing linear functions," and "slope-intercept form"), video IDs are used to remove duplicates. This ensures downstream processing effort is not wasted on redundant content.
LLM-based metadata review. For each remaining video, the authors prompt an LLM (GPT-4o, per Section 7.1) to review the video's metadata — specifically the title, description, and comments — and exclude videos with "irrelevant, pornographic, or illegal content" (Section 3.1). This is a safety and relevance filter that operates before any video content is downloaded, saving bandwidth and processing time.

The output of this stage is 159,565 videos (Section 3.1). The reduction from the theoretical $195,750$ (before deduplication) to 159,565 reflects both deduplication (videos appearing for multiple knowledge points) and metadata-based filtering (irrelevant or inappropriate content removed).

Why filter by metadata before downloading? Processing a video through the full pipeline (ASR transcription alone requires downloading the audio and running Whisper-large-v3, which is computationally expensive) represents a significant cost. Filtering by metadata is extremely cheap — it requires only an API call with a short text prompt per video — and can eliminate obviously irrelevant content (adult content, non-educational entertainment, non-English videos that slip through) before incurring the full processing cost. This is a practical engineering decision that reflects the realities of large-scale data curation: front-load cheap filters to avoid wasting expensive computation.

Video-Level Extraction: Audio to Text via ASR

The video-level extraction stage takes the 159K metadata-filtered videos and converts each video's auditory content into text (Section 3.2). This is the first transformation that makes the instructional content machine-readable for language model pretraining.

Audio extraction. The authors use FFmpeg — an open-source multimedia framework — to extract the audio track from each video file. This is a standard, reliable operation that separates the auditory signal from the visual signal, allowing each to be processed independently.

Automatic Speech Recognition (ASR). The extracted audio is then transcribed into text using Whisper-large-v3 (OpenAI's open-source ASR model). The paper states this directly: "transcribe it into text (audio-to-text, ASR) using whisper-large-v3" (Section 3.2). Whisper-large-v3 is chosen because it is state-of-the-art for English ASR, handles diverse acoustic conditions (different microphones, background noise, varying speaker accents), and provides timestamped transcriptions — a critical feature for the subsequent clip-level segmentation (the timestamps tell the pipeline when each sentence was spoken in the video, enabling temporal alignment between text and frames).

Why Whisper-large-v3 specifically? The paper does not compare ASR models, but the choice of the "large-v3" variant reflects a prioritization of transcription accuracy over speed. Since the transcribed text will become pretraining data, errors in ASR (hallucinated words, misrecognized technical terms, missed sentences) directly degrade the quality of the downstream corpus. Using the largest, most capable Whisper variant maximizes transcription fidelity, accepting higher computational cost as a worthwhile tradeoff for data quality.

The colloquialism problem. The paper identifies a key quality issue with raw ASR output: "due to the nature of tutorial speech where the instructors prefer to use colloquial expressions to explain a concept, the perplexities (PPLs) of the raw ASR transcriptions are usually much higher than those of the texts from standard corpora" (Section 3.2). This is a concrete, measurable problem. Perplexity (PPL) is a metric that quantifies how "surprising" or "unpredictable" a text is to a language model — lower PPL means more fluent, more predictable text. Standard written corpora (like Wikipedia or books) have low PPL because they use formal, well-structured language. Spoken lecture transcripts have high PPL because they contain:

Hesitations and fillers: "um," "uh," "you know," "like"
Fragments and restarts: "So the velocity is, um, it's basically the distance divided by— well, let me rephrase that"
Colloquial connectors: "okay so now we're gonna look at," "the next thing is," "and then we have this over here"
Deictic references without visual context: "this one here," "that angle there" (meaningless without seeing the screen)

The paper reports concrete PPL numbers in the ablation study (Section 5.4, Table 6): raw ASR text has a PPL of 16.86, while the refined ASR text (after LLM rewriting) has a PPL of 13.92, compared to OBELICS at 11.27 and MMC4-Core-ff at 12.56. These numbers confirm that raw ASR is substantially less fluent than standard written corpora.

ASR refinement via LLM rewriting. To address the colloquialism problem, the authors "introduce Qwen2-72B-Instruct to rewrite the raw ASR transcriptions, with the purpose of improving their fluency and coherence while not changing the original semantics" (Section 3.2). This is a crucial design choice: rather than discarding colloquial transcripts (which would lose the instructional content) or using them as-is (which would degrade the language model's fluency), the pipeline normalizes the text to written-lecture style while preserving the factual and explanatory content.

The choice of Qwen2-72B-Instruct specifically reflects several considerations: (1) it is a large (72B parameter) instruction-tuned model capable of following nuanced rewrite instructions, (2) it is open-weight and can be run at scale without API costs, and (3) the instruction-tuning format allows specifying the rewrite task precisely ("improve fluency and coherence while not changing the original semantics").

What the refinement actually does. The paper does not provide the exact prompt, but based on the described goal, the LLM likely performs operations such as:

Removing filler words and false starts
Converting sentence fragments into complete sentences
Resolving anaphoric references when possible (replacing "this" and "that" with the actual referent where it can be inferred)
Standardizing mathematical notation (converting spoken "ten meters per second squared" to written $10\ \text{m/s}^2$ )
Smoothing transitions between sentences that were originally separated by pauses or visual demonstrations

The ablation in Table 6 demonstrates that this refinement is essential: removing ASR refinement (w/o ASR Refine) causes a 4.9% average performance drop across seven benchmarks in the 1-shot setting. This is a large effect — nearly five percentage points across diverse tasks — confirming that raw ASR text is sufficiently low-quality to impair downstream VLM performance, and that LLM-based refinement successfully bridges the gap.

Video-Level Filtering: Removing Non-Instructional Videos by ASR Quality

Not all 159K videos that pass metadata filtering are genuinely instructional, and even among instructional videos, transcription quality varies. The paper applies a three-criteria LLM evaluation to filter videos based on their ASR content before proceeding to the more expensive clip-level processing (Section 3.2).

Pre-filtering with rules. Before the LLM evaluation, the authors apply simple rule-based filters:

Non-English videos: discarded (the taxonomy and downstream models are English-focused)
Videos shorter than 10 seconds: likely incomplete, advertisements, or trivial content
Silent videos with very few ASR text tokens: essentially audio-free, providing no textual knowledge

These rules are cheap to apply (requiring only metadata checks and token counting) and eliminate clear failure cases.

LLM-based ASR scoring. For the remaining videos, the authors employ two LLMs — DeepSeek-V2 and Llama3-70B-Instruct — to independently score each ASR transcription along three dimensions (Section 7.1):

Relevance: Does the ASR represent tutorial content aligned with the targeted knowledge point? This filters out videos where the metadata suggested instructional content but the actual speech is about something else entirely (e.g., a video titled "Introduction to Algebra" that is actually a vlog about someone's math anxiety).
Knowledge Density: Does the ASR contain substantive instructional content, or is it dominated by "meaningless filler phrases like 'um,' 'the next up is this,' or 'then we get this'"? (Section 3.2). This criterion targets videos where the instructor spends most of the time on transitions, setup, or banter rather than actual explanation.
Transcription Quality: Is the Whisper transcription itself reliable, or does it contain "repetitive or erroneous ASR text"? (Section 3.2). This catches cases where poor audio quality (background noise, heavy accent, low bitrate) produced garbled output. Importantly, the paper notes that "this step occurs before ASR rewriting" — transcription quality is assessed on the raw Whisper output, not the refined version, because refinement cannot fix fundamentally broken transcriptions.

Why two LLMs? The paper specifies that "a video is filtered out if both LLMs determine its ASR does not meet the required standards" (Section 7.1). Using two independent models for consensus filtering reduces false positives — a single model might incorrectly flag an unusual but legitimate instructional style as low-quality. Requiring agreement from both DeepSeek-V2 and Llama3-70B-Instruct makes the filter more conservative (retaining more videos) while still catching egregious quality failures that both models agree on. This is a practical robustness measure against LLM evaluation noise.

Filtering outcome. After applying rule-based filtering and the dual-LLM ASR evaluation, approximately 53% of the initial 159K videos are filtered out, retaining 75,000 videos totaling 22,697 class hours with an average duration of ~18 minutes (Section 3.2, Section 4.1, Table 7). The filtering rate confirms that raw internet-sourced video collections are substantially noisy — more than half of initially retrieved content was unsuitable for a textbook-quality corpus. The breakdown by subject (Table 7) shows mathematics dominates with 21.7K videos (4,423 hours), followed by engineering (13K, 4,096 hours), computer science (12.8K, 4,354 hours), earth science (12K, 3,670 hours), physics (11K, 3,511 hours), and chemistry (4.5K, 2,643 hours).

Clip-Level Extraction: Temporal Segmentation via ASR Timestamps

The 75K retained videos are long (average 18 minutes), but VLM pretraining samples are much shorter (hundreds to low thousands of tokens). To bridge this gap, the pipeline segments each video into short clips (10–20 seconds) using the timestamps embedded in the Whisper ASR output (Section 3.2).

The timestamp mechanism. Whisper-large-v3 does not just produce plain text; it outputs text aligned with start and end timestamps at the sentence or phrase level. Each transcribed segment is thus a tuple (text_segment, start_time, end_time) that associates a spoken sentence or phrase with its temporal location in the video. The pipeline uses these timestamps to slice the video: a clip is defined as the video segment between start_time and end_time of an ASR segment, yielding the aligned pair:

$\langle\text{clip}_i, \text{asr}_i\rangle$

where $\text{clip}_i$ is the short video segment and $\text{asr}_i$ is the corresponding transcribed (and later refined) text.

Merging incomplete ASR segments. A practical complication: Whisper's segmentation sometimes produces fragments — sentences split across multiple segments, or short utterance fragments that are semantically incomplete. The authors address this by merging "multiple incomplete ASR segments into a single, semantically coherent paragraph" before using timestamps for video slicing (Section 3.2). The merged segment's timestamp span becomes the union of its constituent segments' timestamps.

Why merge? If a single logical sentence ("The derivative of x squared is two x") were split across two ASR segments ("The derivative of x squared" / "is two x"), segmenting the video at the fragment boundary would produce two clips: one showing the instructor mid-sentence and one completing it. Neither clip would be a self-contained instructional unit. Merging ensures each clip corresponds to a semantically complete utterance, making the resulting ⟨clip, asr⟩ pairs coherent as standalone training samples.

Clip duration. The paper states each clip lasts "10 to 20 seconds" (Section 3.2). This range is not an explicit parameter but rather an empirical consequence of how instructors speak — a complete sentence or short paragraph explaining a concept typically takes 10–20 seconds. At typical speaking rates (2–3 words per second), this corresponds to roughly 20–60 words of text per clip, which is a manageable granularity for later interleaving with keyframes.

Clip-Level Filtering: Removing Visually Uninformative Clips

The video-level filtering assessed videos based on their auditory content (ASR quality). The clip-level filtering assesses clips based on their visual content: does the clip actually show something instructionally useful, or is it just a shot of the instructor talking? (Section 3.2).

The problem of "talking head" footage. Instructional videos inevitably contain segments where the instructor is simply speaking to the camera, with no slides, diagrams, or demonstrations visible. These segments are informationally asymmetric: the audio may contain valuable explanations, but the video adds nothing — the model gains no visual knowledge from a static shot of a person's face. Including such clips in an interleaved corpus would dilute the image-text alignment: the model would learn to associate lecture text with irrelevant frames of a speaker's face, which is precisely the "loose text-image relation" problem the paper criticizes in webpage datasets.

Caption generation and similarity scoring. The filtering procedure has two steps:

Caption each clip. The authors use VideoLlama2-7B, a video-language model, to generate a detailed textual caption describing the visual content of each short clip (Section 3.2, Section 7.1). For example, a clip showing a slide about complementary angles would receive a caption like "A presentation slide displays the definition of complementary angles with a diagram showing two angles summing to 90 degrees," while a clip showing only the instructor would receive a caption like "A person is standing in front of a whiteboard, speaking to the camera."
Compute caption-ASR similarity. The clip's caption and its corresponding ASR transcription are embedded using a text embedding model — GTE-Qwen2-7B-Instruct — and their cosine similarity is computed (Section 3.2, Section 7.1). The intuition is straightforward: if the visual content (caption) describes what the instructor is explaining (ASR), the two should be semantically similar. If the clip is a "talking head" shot, the caption ("a person speaking") will be dissimilar to the ASR content ("the quadratic formula is x equals negative b plus or minus..."), and the clip is flagged for removal.

What similarity threshold is used? The paper does not specify a hard threshold value. The language is qualitative: "filtering out uninformative video clips" based on similarity. In practice, this likely means using a percentile-based cutoff or a manually tuned threshold derived from inspecting the similarity distribution on a validation set.

Preserving orphaned ASR. A crucial design detail: when a clip is removed because its visual content is uninformative, its ASR transcription is not discarded. The paper states (Section 3.2):

"even if an uninformative video clip is discarded, its ASR transcription may still contain valuable information. Thus, we retain these transcriptions in our textbook"

The resulting interleaved sequence might look like:

$\langle\text{clip}_1, \text{asr}_1\rangle, \text{asr}_2, \text{asr}_3, \langle\text{clip}_4, \text{asr}_4\rangle, \ldots$

where asr₂ and asr₃ are textual-only segments whose corresponding clips were filtered out. This design ensures that knowledge density is maximized — the corpus contains all the instructor's explanations, even when the visual channel was not contributing — while avoiding the degradation of image-text alignment that would result from pairing high-quality text with low-quality frames.

Why VideoLlama2 and GTE-Qwen2? The choice of VideoLlama2-7B reflects the need for a model that can understand video content (not just static images) to accurately caption short clips that may contain motion, transitions, or animations. The choice of GTE-Qwen2-7B-Instruct for text embeddings reflects the need for a strong semantic similarity model that can handle technical educational content — generic embedding models might fail to recognize that "a presenter explains geometry" and "complementary angles sum to 90 degrees" are related in an educational context, but GTE-Qwen2, being instruction-tuned on diverse tasks including retrieval, is likely more robust to domain-specific language.

The output of the clip-level stage is approximately 4 million video clips (Table 7) with their aligned ASR segments, from which ~1.6M additional standalone ASR segments are preserved from filtered-out clips.

Keyframe-Level Extraction: Detecting Representative Frames via SSIM

Video clips contain many frames (at typical 30 fps, a 15-second clip contains 450 frames), but most adjacent frames are nearly identical. Training a VLM on all frames would be computationally wasteful and would not provide additional learning signal — the model would see hundreds of nearly duplicate images. The keyframe extraction step reduces this redundancy by selecting only frames that exhibit significant visual change relative to their predecessors (Section 3.2).

The Structural Similarity Index (SSIM). The authors use SSIM to compare consecutive frames. SSIM is a perceptual similarity metric that computes a score between 0 (completely different) and 1 (identical) based on three components: luminance similarity, contrast similarity, and structural similarity (correlation between pixel patterns). Unlike simple pixel-level difference (which is sensitive to noise, slight camera motion, or lighting changes), SSIM is designed to match human perception of image similarity — small lighting shifts or noise do not significantly reduce the SSIM score, while a slide changing content or a new diagram appearing produces a large drop.

The extraction algorithm. The process is formalized in Algorithm 1 of the supplementary material (Section 7.3) and operates as a greedy sequential scan:

Initialize the keyframe sequence with the first frame $F_1$
Set $F_1$ as the reference frame
For each subsequent frame $F_i$ (in temporal order):
- Compute $\text{SSIM}(\text{reference\_frame}, F_i)$
- If $\text{SSIM} < T$ (where $T$ is a similarity threshold):
  - Add $F_i$ to the keyframe sequence
  - Set $F_i$ as the new reference frame
- Otherwise, skip $F_i$

This greedy approach means that as soon as a frame is sufficiently different from the current reference, it becomes a keyframe and the reference resets. The next comparisons are made against this new reference, so the algorithm tracks the visual trajectory of the video — it will extract frames whenever there is a meaningful change, but will skip long stretches of near-static content.

What threshold $T$ is used? The paper does not specify the exact SSIM threshold. In practice, this is typically tuned on a validation set to balance between extracting too many frames (redundancy, increased downstream cost) and too few frames (missing important visual transitions). The threshold likely varies with content type — for slides with text, small SSIM changes might be meaningful (a new line of text appearing), while for natural video footage, only larger changes should trigger a new keyframe.

Why SSIM over pixel-level or semantic alternatives? The ablation study (Section 5.4, Table 6) directly compares three extraction methods:

Pixel-level extractor (OpenCV absdiff): Computes absolute pixel differences between consecutive frames. This method extracts 18 million keyframes — nearly three times the SSIM count — and causes a 9% drop in 1-shot accuracy. Pixel-level comparison is overly sensitive: minor lighting changes, compression artifacts, or subtle camera motion trigger keyframe extraction even when the semantic content is unchanged. In instructional videos with abstract diagrams or geometric figures, these small perturbations dominate the pixel-level signal, producing excessive redundancy.
CLIP-based extractor: Uses CLIP-ViT-L to compute semantic similarity between frames. This method extracts only 1.7 million keyframes — far fewer than SSIM — and causes a 6.5% drop. CLIP embeddings capture high-level semantic content (what objects are present) rather than fine-grained structural differences. In instructional videos, consecutive frames often have the same high-level semantics (both show "a geometry diagram") but differ in the specific text, annotations, or problem-solving steps displayed. CLIP cannot distinguish these structurally different but semantically similar frames, causing it to miss critical keyframes.
SSIM: Extracts 6.5 million keyframes — a middle ground between the over-sensitive pixel method and the under-sensitive semantic method — and yields the best downstream performance. SSIM's structural similarity component specifically captures the kind of changes that matter in instructional content: text appearing or changing position, annotations being added, new diagram elements being drawn. These changes alter the spatial structure of the image (where edges and patterns are located) without necessarily changing the semantic category, making SSIM well-matched to the domain.

The output is 6.5 million keyframes across all videos (Table 7), distributed by subject: mathematics (1.67M), computer science (1.21M), engineering (1.15M), earth science (1.03M), physics (0.95M), chemistry (0.49M). The resulting keyframe-ASR sequence has the structure:

$\langle\text{frame}^{k_1}_1, \text{frame}^{k_2}_1, \text{asr}_1\rangle, \text{asr}_2, \text{asr}_3, \langle\text{frame}^{k_1}_4, \text{asr}_4\rangle, \ldots$

where a single clip may contribute multiple keyframes (superscripts $k_1, k_2$ ) if multiple significant visual changes occurred during that clip's duration.

Keyframe-Level Extraction: OCR for On-Screen Text and Symbols

Instructional videos frequently display text, formulas, and diagrams that convey knowledge through the visual channel independently of the spoken narration. A slide might show "Complementary Angles: $\angle A + \angle C = 90^\circ$ " while the instructor verbally explains the concept. The ASR captures the explanation; the OCR captures the formal definition and notation. Both are needed for a complete training signal (Section 3.2).

OCR model choice. The authors use InternVL2-40B, a state-of-the-art vision-language model with strong document understanding and OCR capabilities (Section 3.2, Section 7.1). Unlike traditional OCR engines (Tesseract, etc.) that recognize text character-by-character, InternVL2-40B is a multimodal LLM that can identify text in context — it can distinguish between a mathematical formula, a diagram label, and body text, and it can handle the varied fonts, colors, and layouts typical of instructional slides.

Why a VLM rather than a dedicated OCR engine? The paper's domain includes mathematical notation (fractions, exponents, integrals), chemical formulas ( $\text{H}_2\text{O}, \text{CO}_2$ ), and geometric diagrams with labeled points ( $\triangle ABC$ ). Traditional OCR systems are optimized for printed text and struggle with mathematical typesetting, superscripts/subscripts, and non-standard layouts. InternVL2-40B, as a VLM, can leverage its visual understanding to correctly parse these elements — it "sees" the layout holistically rather than treating the image as a grid of characters.

What OCR text covers. The paper states that OCR extracts "on-screen text, mathematical symbols, formulas, and other elements" (Section 3.2). The supplementary examples (Figures 6–10) illustrate the range: in the physics velocity example (Figure 6), OCR extracts the formula $a = (v - u) / t$ and the step-by-step calculation $= 10\ \text{m/s}\ /\ 5\ \text{s} = 2\ \text{m/s}^2$ ; in the chemistry example (Figure 9), OCR would extract molecular formulas and labels; in the DFS algorithm example (Figure 10), OCR extracts pseudocode. The OCR thus serves as a structured supplement to the ASR, capturing information that is communicated visually rather than verbally — exactly the kind of cross-modal redundancy that strengthens learning.

Scale of OCR output. Table 7 reports 500 million OCR tokens across all videos, compared to 258 million ASR tokens (after refinement). OCR text thus approximately doubles the textual content of the corpus, with the distribution mirroring the video distribution: mathematics (145M OCR tokens), engineering (86.6M), computer science (85.5M), earth science (80M), physics (73.4M), chemistry (30M). The fact that OCR contributes roughly twice as many tokens as ASR may seem counterintuitive (videos are primarily audio-driven), but it reflects the nature of instructional slides and demonstrations: a single keyframe may contain dozens of words of on-screen text, formulas, and labels that correspond to only a few seconds of spoken explanation.

Keyframe-Level Filtering: Removing Occluded Frames and Redundant OCR

Even after clip-level visual filtering and SSIM-based keyframe selection, some extracted keyframes may be of low quality, and the OCR text extracted from consecutive keyframes is often highly repetitive (Section 3.2).

Keyframe quality scoring. The authors use InternVL2-40B again — the same model used for OCR — to score each keyframe's informativeness after OCR is performed (Section 3.2). Frames with "object occlusion" (the paper gives this as the example) — where something blocks the view of the instructional content — are discarded. This could include: the instructor walking in front of the slide, a cursor or pointer obscuring text, a pop-up notification, or a transition effect partially covering the screen.

Why score keyframes after OCR rather than before? The keyframe's value depends on what can be extracted from it. A frame that looks visually clean might have text too small to OCR reliably; a slightly occluded frame might still yield usable OCR if the occlusion is in an irrelevant region. By running OCR first and then scoring, the pipeline makes an informed decision based on actual extractable content rather than surface-level visual quality.

Redundant OCR removal. Consecutive keyframes in instructional videos often display the same slide content with only minor differences (a new bullet point appears, a single annotation is added). The OCR output for these frames would be nearly identical — the same base text repeated with small additions. To avoid flooding the corpus with duplicate text that provides no additional learning signal, the authors "filter out OCR results that are similar to previous ones" (Section 3.2).

What similarity measure for OCR deduplication? The paper does not specify the exact method for comparing OCR outputs. Likely approaches include: (1) computing token-level Jaccard similarity or edit distance between consecutive OCR strings and discarding if above a threshold, (2) using a text embedding model (perhaps the same GTE-Qwen2 used for clip filtering) to compute semantic similarity, or (3) keeping only the OCR from the last keyframe in a sequence of near-duplicates, since it contains the most complete version (all accumulated bullet points, all annotations). The qualitative goal is clear — avoid redundancy — but the exact implementation is unspecified, leaving room for variation in reimplementations.

Sample Construction: Assembling the Final Interleaved Corpus

With all extraction and filtering complete, the pipeline has produced for each video a chronologically ordered sequence of:

$\{\text{frame}^{k_1}_1, \text{frame}^{k_2}_1, \text{ocr}_1, \text{asr}_1, \text{asr}_2, \text{asr}_3, \text{frame}^{k_1}_4, \text{ocr}_4, \text{asr}_4, \ldots\}$

where keyframes, their extracted OCR text, and the instructor's (refined) ASR transcription are interleaved in the temporal order of the original video (Section 3.2). This sequence is the raw material from which pretraining samples are constructed.

The sample construction problem. An average instructional video in the dataset lasts ~18 minutes and contains ~86 keyframes (6.5M keyframes / 75K videos, per Table 7). This is far too long for a single VLM pretraining sample — most VLMs have maximum context windows of 2K–8K tokens during pretraining, and even models supporting longer contexts are typically trained with shorter sequences for computational efficiency. The video-level sequences must be segmented into smaller training samples.

Segmentation strategies. The supplementary material (Section 7.3) describes three approaches:

Per-video samples. Treat each entire video as a single sample, preserving the complete narrative structure of the lecture. This maintains semantic integrity but produces samples averaging 86 keyframes and thousands of text tokens — exceeding context length limits and causing computational inefficiency (most tokens would be padding for shorter videos, and long videos would be truncated).
Video-internal segmentation. Split each long video into multiple samples based on natural breakpoints or a fixed maximum context length. This preserves within-video coherence while respecting practical context length constraints. The exact segmentation method is not specified, but likely involves splitting at ASR paragraph boundaries or after a fixed number of keyframes.
Cross-video concatenation. "Directly concatenate multiple video clips i.e., ⟨frame^{k_1}_i, ..., frame^{k_n}_i, ocr_i, asr_i⟩, to the maximum context length" (Section 7.3). This strategy ignores video boundaries entirely, packing as many clip-level segments as possible into each training sample regardless of which video they came from. This maximizes computational efficiency (no padding, full utilization of the context window) but risks confusing the model by mixing unrelated topics within a single sample.

The adopted strategy. The paper describes using a combination of strategies, with cross-video concatenation as the primary approach, but with a mitigating mechanism: "insert a specific token: End of Video at the end of each video to mitigate" the adverse effects of mixing video clips (Section 7.3). The End of Video token serves as an explicit boundary marker, telling the model that the following keyframes and text belong to a new, unrelated video — analogous to the document boundary markers used in some LLM pretraining corpora. This allows the pipeline to pack samples efficiently while preserving the model's ability to distinguish between coherent within-video sequences and random across-video juxtapositions.

Final sample statistics. After sample construction, the corpus contains 610,000 interleaved samples, each with an average of 10.7 keyframes and 1,297 text tokens (Section 4.1, Table 1). The sample count by subject (Table 7) is: computer science (150K), mathematics (123K), physics (119K), engineering (98K), earth science (88K), chemistry (32K). The total corpus comprises 6.58 million keyframes, 258 million ASR tokens, and 500 million OCR tokens, for a combined ~759 million text tokens (Section 4.1, Table 7).

Why 610K samples from 4M clips? The clip-to-sample ratio is approximately $4\text{M} / 610\text{K} \approx 6.6$ clips per sample on average, consistent with the reported 10.7 keyframes per sample (each clip may contribute 1–3 keyframes, so 6–7 clips contributing ~1.6 keyframes each yields ~10.7 total). This reflects the cross-video concatenation strategy: multiple clips from potentially different videos are packed together to fill the target sample length.

Design Decisions Summary

The paper's pipeline embodies several recurring design principles that warrant explicit recognition:

1. Progressive filtering at increasing granularity. Filtering occurs at three levels: video (metadata, ASR quality), clip (visual informativeness), and keyframe (occlusion, OCR redundancy). Each level uses a different signal — textual metadata, auditory content, visual-textual alignment, fine-grained visual quality — and filters become stricter as the grain becomes finer. This coarse-to-fine structure maximizes efficiency: cheap, broad filters eliminate most noise early, preventing expensive downstream processing of low-quality content.

2. LLMs at every quality gate. The pipeline uses LLMs for taxonomy generation (GPT-4o), metadata review (GPT-4o), ASR refinement (Qwen2-72B-Instruct), ASR quality scoring (DeepSeek-V2 + Llama3-70B), clip captioning (VideoLlama2-7B), and OCR extraction/scoring (InternVL2-40B). This represents a design philosophy of automated quality assessment — rather than relying on human annotators (expensive, slow, inconsistent) or simple heuristics (brittle, domain-specific), the pipeline leverages the increasingly strong zero-shot capabilities of LLMs and VLMs to make nuanced quality judgments at scale. The dual-LLM consensus mechanism for ASR filtering is a particularly thoughtful touch, recognizing that individual LLM judgments can be noisy.

3. Content preservation over format purity. When a clip's visual content is filtered out, its ASR text is retained. When keyframes are discarded, their corresponding text segments are not. The pipeline separates visual and textual quality assessment, recognizing that even "imperfect" multimodal pairings can contribute useful training signal as long as the text is high-quality. This increases effective data yield without contaminating the image-text alignment quality at the frame level.

4. Domain-matched algorithms for domain-specific content. The choice of SSIM over pixel-level or CLIP-based keyframe extraction is explicitly justified by the nature of instructional video content (abstract diagrams, geometric figures, text-heavy slides). The choice of InternVL2-40B for OCR reflects the need for mathematical and chemical formula recognition. These are not generic "best" choices; they are choices tuned to the specific visual characteristics of educational content. This kind of domain-aware pipeline engineering is what distinguishes data curation from generic data collection.

5. The End of Video token as a pragmatic compromise. The tension between sample packing efficiency (cross-video concatenation) and semantic coherence (within-video integrity) is resolved not by choosing one over the other but by inserting an explicit boundary signal that lets the model learn to distinguish the two cases. This is a small but important design detail that reflects an understanding of how language models actually learn from interleaved data — they can handle multiple documents in one context window if given explicit structural cues about document boundaries.

4. Key Insights and Innovations

Innovation 1: Reframing Interleaved Data Quality as a Source Problem, Not a Scale Problem

The paper's most fundamental conceptual move is the diagnosis that the quality ceiling of interleaved pretraining data is not determined by how much you crawl or how aggressively you filter, but by the inherent properties of the source medium itself. This is a reframing of the problem that the field had not articulated clearly.

Prior to this work, the dominant assumption in multimodal pretraining — carried over from the text-only domain's "more data is better" ethos — was that scaling up web crawling and applying better filtering heuristics would yield improved interleaved corpora. The progression from MMC4 to OBELICS to OmniCorpus reflected this assumption: each iteration increased scale, added filtering steps, and expanded the range of crawled sources, implicitly betting that the quality limitations (loose image-text relations, incoherent image sequences, low knowledge density) were noise problems solvable through better curation of the same web content.

The paper's evidence decisively challenges this assumption. The image shuffling experiment (Section 5.3, Figure 3) is the diagnostic that makes the case: shuffling the image order within MMC4 samples causes essentially no performance degradation, while shuffling the textbook dataset causes a severe and monotonic drop that increases with shuffling ratio. This is not a marginal difference in filtering quality — it reveals a qualitative difference in the nature of the data. Webpage images lack sequential dependency because the source medium (arbitrary web pages) does not impose one; video frames possess strong sequential dependency because the source medium (instructional lectures) is inherently structured as a linear explanatory narrative.

The authors crystallize this insight through their In-sample Image Similarity (InSI-SIM) metric (Table 1), which quantifies what the shuffling experiment demonstrates causally. The textbook dataset's InSI-SIM of 0.686 — more than double OBELICS's 0.345 and MMC4's 0.319 — is not achieved through filtering; it is an intrinsic property of sourcing data from temporally coherent video sequences rather than static webpages. No amount of post-hoc filtering applied to Common Crawl can manufacture temporal coherence between images that were never temporally related in the first place.

This reframing has significant implications beyond the paper's specific dataset. It suggests that the VLM pretraining community's focus on filtering (developing better CLIP-based similarity thresholds, more sophisticated deduplication, improved NSFW detection) is addressing the wrong bottleneck. The bottleneck is sourcing — choosing data origins that intrinsically produce the desired structural properties — and the paper's key contribution is identifying that instructional videos constitute such a source. This is analogous to the insight in the text-only domain that "textbooks are all you need" (Gunasekar et al., 2023), but with the critical multimodal dimension: it is not just about knowledge density (which textbooks also provide), but about the coherence structure between images that only temporally organized content can provide.

The paper thus introduces a new axis for evaluating pretraining data that goes beyond the standard metrics of dataset size, text length, and image count: the intrinsic coherence of the source medium. This is a conceptual contribution that changes how researchers should think about data curation — not as a filtering problem applied to whatever is most abundant (the web), but as a sourcing problem where the goal is to find content that was already structured in pedagogically useful ways before any processing began.

Innovation 2: Demonstrating That Interleaved Context Awareness Is a Learned Capability, Not an Emergent Architectural Property

A subtler but equally important contribution is the paper's empirical demonstration that VLMs' ability to attend to their interleaved context — to notice what images and text are in their prompt and use that information for task-solving — is not a given, but rather a capability that must be specifically cultivated through the right pretraining data. This finding reframes how the field should think about in-context learning in multimodal models.

The prevailing narrative in the VLM literature, following the success of interleaved pretraining for in-context learning (Alayrac et al., 2022; Awadalla et al., 2023; Laurençon et al., 2024), treated the interleaved format itself as the enabler. The logic was: if you train a model on sequences where images and text are interleaved, it will naturally learn to integrate information across those modalities during inference. The paper's "cheat test" experiment (Section 5.3, Table 4) shows this logic is incomplete.

In the cheat test, one of the few-shot examples is replaced with the test sample itself — giving the model an explicit shortcut to the correct answer if it genuinely attends to its context. The results reveal a stark capability gap that standard few-shot evaluation obscures:

On MathVision, a VLM pretrained on the textbook scores 98.4% in the 1-shot cheat setting, compared to 69.3% for MMC4 and 66.5% for OBELICS. The textbook-trained model nearly always notices that its context already contains the answer; the webpage-trained models miss this more than 30% of the time.
When the cheat becomes harder (2-shot: one identical example plus one random example, lengthening the context), the webpage-trained models' performance drops sharply (MMC4: 69.0% → 53.5% on OKVQA), while the textbook-trained model actually improves on OKVQA (79.2% → 84.3%) and maintains substantial advantages across all benchmarks.

These results demonstrate that training on an interleaved format is necessary but not sufficient for developing genuine context awareness. The model must also learn why attending to context is useful — and this learning signal comes from the coherence of the interleaved sequences during pretraining. When pretraining images and text are only loosely related (as in webpage datasets), the model learns that paying close attention to the interleaved structure is not particularly rewarding — images might be irrelevant, text might not describe adjacent images, so the optimal strategy is to treat images and text as largely independent. When pretraining images and text are causally linked (as in the textbook, where each frame illustrates exactly what the adjacent text describes), the model learns that maximizing attention to the interleaved structure is consistently rewarded, because the adjacent text always contains information relevant to understanding the image, and vice versa.

This is a fundamental insight about the nature of in-context learning in multimodal models. It is not an emergent property of scale or architecture — it is a learned behavior shaped by the reward structure of the pretraining data distribution. The paper shows that what matters is not just that images and text are interleaved, but how they are interleaved — specifically, whether the interleaving carries predictive signal that rewards attentive integration. This insight has direct implications for pretraining data design: to build VLMs with strong in-context learning, you need data where images and text are not merely co-occurring but are mutually informative in a consistent, predictable way that the model can learn to exploit.

Innovation 3: Establishing Video-Derived Keyframes as a New Category of Pretraining Data Distinct from Both Static Images and Video

The paper's technical pipeline converts instructional videos into an image-text interleaved format — but it would be a mistake to categorize the output as merely "video data" or "image data." The paper establishes, through both its curation methodology and its experimental results, that SSIM-selected keyframes from instructional videos constitute a distinct category of pretraining data with properties that neither static images (from webpages or caption datasets) nor traditional video frames can replicate.

The ablation comparing keyframe extraction methods (Section 5.4, Table 6) makes this distinction empirically precise:

Static image data (what pixel-level extraction approximates by treating every significantly different frame as a new image) produces 18M keyframes but degrades performance by 9%. This is because pixel-level extraction captures all visual change — including lighting shifts, compression artifacts, and smooth camera pans through static scenes — treating every frame as a new "image" even when the instructional content has not meaningfully changed. The resulting data has the diversity of video but the semantic redundancy of burst photography.
Semantic-level image data (what CLIP-based extraction produces) yields only 1.7M keyframes and degrades performance by 6.5%. CLIP sees the category of content ("this is a geometry diagram") but misses the progression of content within that category (a new line of text, a new annotation, a highlighted region). The resulting data is semantically diverse but structurally impoverished — it captures few distinct frames because consecutive instructional slides fall into the same semantic cluster.
SSIM-based keyframe data (6.5M keyframes, best performance) occupies a sweet spot that neither static image datasets nor traditional video frame sampling achieves. It is dense enough to capture the step-by-step progression of instructional content (missing fewer than 10% of meaningful frame transitions, based on the performance comparison to pixel-level extraction) but sparse enough to avoid redundancy (producing ~3× fewer frames than pixel-level extraction). More importantly, the frames it selects are pedagogically meaningful — they correspond to moments when the visual content structurally changes in ways relevant to the instructional narrative (a new formula appearing, a diagram being annotated, a new step being displayed).

This finding challenges the implicit assumption in much VLM pretraining that "more images = better training." The paper shows that the selection criterion for which frames to include matters as much as the total quantity — and that the optimal criterion is domain-specific, varying with the type of visual content. For natural scenes (photographs, videos of real-world events), semantic-level selection (CLIP) might be optimal because the meaningful changes are categorical. For instructional content (slides, diagrams, derivations), structural-level selection (SSIM) is optimal because the meaningful changes are within-category — the difference between a geometry diagram showing complementary angles and the same diagram annotated with $\angle A + \angle C = 90^\circ$ is structural, not semantic.

This conceptual innovation reframes video-derived keyframes as a third category of pretraining data — not "images from video" (which suggests they are just a lower-quality version of web images) and not "video frames" (which suggests they should be used in sequential video understanding tasks), but instructional keyframes with domain-specific properties that require domain-specific extraction methods. This is a subtle but important contribution to the taxonomy of multimodal pretraining data, and it has practical implications for anyone else seeking to convert video into image-text training data: generic keyframe extraction methods will underperform, and the choice of extraction algorithm must be matched to the visual characteristics of the content genre.

Innovation 4: Empirically Proving That Knowledge-Dense Audio-Visual Alignment Substitutes for Pretraining Scale

The paper's most practically significant contribution — though one the paper itself presents modestly — is the demonstration that a modestly sized pretraining corpus (610K samples, ~759M text tokens) sourced from high-quality instructional content can substantially outperform equivalent quantities of web-scale data and, in specific difficulty regimes, produce models competitive with those trained on much larger corpora. This is not merely a "our dataset is better" claim; it is evidence for a specific scaling hypothesis that had not been tested in the multimodal domain.

The key evidence is the FLOPs-matched comparisons implicit in Tables 2 and 3, and the SFT transfer results in Table 5:

Training LLaVA-1.5-7B on 610K textbook samples vs. 610K MMC4 samples yields average gains of +3.2% (0-shot), +8.3% (1-shot), +4.0% (2-shot), and +4.6% (4-shot) across seven benchmarks (Table 2). This is a matched-scale comparison: same model, same number of samples, same training procedure — the only difference is the data source. The gains represent pure data quality improvement, without the confounding of dataset size.
On ScienceQA, the textbook-trained model outperforms the OBELICS-trained model by over 20 percentage points in both zero-shot and few-shot settings (Table 2). This is not a marginal improvement — it is a qualitative capability leap, from being unable to answer science questions to being reasonably competent at them.
After instruction tuning on the standard LLaVA-665K corpus (Table 5), the textbook-pretrained model achieves 28.7% on MathVista, compared to 25.6% for OBELICS-pretrained and 24.8% for MMC4-pretrained — gains of +3.1 and +3.9 percentage points respectively. This demonstrates that the knowledge acquired during textbook pretraining transfers through instruction tuning, persisting even when the instruction-tuning data is identical across conditions.

What makes this an innovation rather than just a dataset paper is the implicit scaling law it suggests: instructional knowledge density can substitute for data scale. The 610K textbook samples (~759M text tokens) represent a tiny fraction of the total scale of MMC4 (billions of tokens) or OBELICS (hundreds of billions). Yet on knowledge-intensive benchmarks, this fraction outperforms the larger corpora because each token carries more useful signal for the target capability (scientific reasoning, mathematical problem-solving, multi-step visual inference). This is the multimodal analogue of the Phi model finding that synthetic textbook data can outperform web-scale data for language model training — but with the critical additional dimension that the alignment between visual and textual content (not just the quality of either alone) is what drives the gain.

The paper does not explicitly formalize this as a scaling relationship (e.g., "1 token of instructional video content ≈ X tokens of web content on knowledge benchmarks"), but the data strongly implies one. A consistent pattern across Tables 2, 3, and 5 is that the textbook's advantage is largest on knowledge-intensive, reasoning-heavy benchmarks (ScienceQA, MathVista, MathVision) and smallest or absent on general VQA benchmarks in zero-shot settings (on zero-shot OKVQA, OBELICS outperforms the textbook by 2.8%). This suggests that the textbook's data quality advantage is domain-specific — it provides vastly more signal per token for instructional knowledge but not necessarily for general visual recognition — which is exactly what one would expect from a source curated for pedagogical content.

This innovation changes the calculus for VLM pretraining data strategy. Rather than the default position of "collect as much interleaved data as possible from the web and filter it," the paper's results support a two-source strategy: use web-scale paired data for general vision-language alignment, and supplement with curated instructional data (at much smaller scale but higher knowledge density) specifically for knowledge acquisition. This is not just "add more data of a different type" — it is recognizing that different data sources serve different training functions and that knowledge acquisition, in particular, benefits disproportionately from pedagogically structured content that the web cannot provide at scale.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper evaluates on seven benchmarks spanning VQA, science, and mathematics: OKVQA (general visual question answering requiring external knowledge), TextVQA (VQA requiring reading text in images), ScienceQA-IMG (multimodal science questions with chain-of-thought reasoning), and three math-related benchmarks — MathVista (diverse mathematical reasoning with visual context), MathVerse (additional math VQA), and MathVision (math problems with visual diagrams). Additionally, TextVQA is evaluated in an OCR-specific variant (TextVQA-OCR), which focuses on the text-reading subcomponent. All benchmarks use the standard publicly available test splits (Section 5.1, Section 8.1).
Base model(s). Two VLM families are used. The primary experiments use LLaVA-1.5-7B (Liu et al., 2024), a widely adopted open-source VLM built on Vicuna-7B with a CLIP-based vision encoder and a two-layer MLP projector. The authors apply continual pretraining on LLaVA-1.5-7B's already-pretrained checkpoint (which was aligned using 558K image-text pairs), keeping the architecture frozen. Additionally, Idefics2-8B (Laurençon et al., 2024) — a more advanced VLM with native multi-image and interleaved input support — is used in two settings: (1) continual pretraining from the publicly released Idefics2-8B-base (already pretrained on OBELICS), and (2) training from scratch using the Idefics2-8B architecture with a randomly initialized projector (Section 5.1). The choice of LLaVA-1.5-7B reflects its status as a standard, well-characterized baseline that the community can readily reproduce; Idefics2-8B tests generalizability to a stronger model with different architecture and pretraining history.
Metrics. All evaluations report accuracy — the fraction of test questions for which the model's predicted answer matches the ground-truth answer. For few-shot evaluation, accuracy is computed per benchmark at each shot count (0, 1, 2, 4, or 8 shots depending on the experiment). Answers are graded using exact-match or equivalent string matching (following the standard LLaVA-1.5 evaluation protocol described in Section 8.1). The paper also reports average accuracy across benchmarks for summary comparisons (Tab. 2, bottom rows). Perplexity (PPL) is reported in the ablation study (Tab. 6) to quantify ASR text fluency, computed using a standard language model evaluation (the exact model is not specified, but the numbers serve as relative comparisons between data variants).
Baselines. The paper compares pretraining on its Textbook-6.5M against three interleaved datasets sampled to the same size (610K samples each): MMC4 (Zhu et al., 2023) — a Common Crawl-sourced interleaved corpus with ~5.7 images and 417 text tokens per sample on average; MMC4-Core-ff — a filtered subset of MMC4 with higher image-text alignment; and OBELICS (Laurençon et al., 2023) — a more recent web-scale interleaved corpus with ~2.5 images and 816 text tokens per sample. For the Idefics2-8B experiments, MMC4-Core-ff and OBELICS serve as baselines. The paper also reports an "Original LLaVA-1.5" baseline in the SFT transfer experiment (Tab. 5), representing LLaVA-1.5 with its standard pretraining but no additional interleaved pretraining. All baselines are trained with identical hyperparameters and sample counts to isolate data quality effects (Section 5.1).
Generation budget / compute accounting. The paper does not use "generations" as a compute unit (there is no test-time search or sampling to budget). Instead, fairness is enforced through matched sample count and matched training steps: all datasets are sampled to 610K interleaved samples, and all models are trained with the same hyperparameters (batch size, learning rate, number of epochs) on identical hardware. This means the compute cost of pretraining is effectively identical across conditions — any performance differences reflect data quality, not training budget. The data curation compute cost (ASR transcription, LLM refinement, OCR extraction) is not amortized or included in training FLOPs comparisons; this is a practical omission the paper acknowledges implicitly by not discussing it.
Cross-validation / statistical protocol. The paper does not employ cross-validation for model evaluation — all results are reported on standard benchmark test sets. For few-shot evaluation, the examples are retrieved using RICES (retrieval-based in-context example selection), which selects the k most similar training examples based on CLIP image embedding similarity to the test image (Section 8.1). This introduces a potential confound: the retrieval quality depends on the training set's image distribution and the CLIP model's embedding space, which are held constant across conditions. However, the paper does not report confidence intervals, standard deviations, or statistical significance tests for any of the accuracy numbers. Given the relatively small test set sizes (e.g., ScienceQA-IMG has ~2,000 test questions, MathVista has ~1,000, OKVQA has ~5,000), differences of 1–2 percentage points may not be statistically reliable, though the paper's larger gaps (5–20 percentage points) are almost certainly significant.

Main Quantitative Results

Textbook-6.5M vs. Webpage Interleaved Datasets on LLaVA-1.5-7B

The headline finding from Table 2 is that continual pretraining on the textbook dataset improves average accuracy across seven benchmarks by +3.2% (0-shot), +8.3% (1-shot), +4.0% (2-shot), and +4.6% (4-shot) compared to MMC4, with the largest gains concentrated on knowledge-intensive benchmarks. This is a matched-scale comparison: each dataset contributes exactly 610K samples, trained under identical conditions.

Knowledge-intensive benchmarks show dramatic improvements. On ScienceQA-IMG (Table 2, top-left block), the textbook achieves 26.3% (0-shot), 29.4% (1-shot), 25.1% (2-shot), and 37.3% (4-shot). In the 0-shot setting, MMC4 and its filtered variant cannot even be evaluated (marked as "—"), while OBELICS reaches only 2.8% — meaning the textbook provides over 23 percentage points of absolute improvement over the best webpage baseline. This is not a marginal gain; it represents a qualitative shift from "essentially unable to answer science questions" to "answering roughly one in three correctly zero-shot." The gap narrows in few-shot settings but remains substantial: at 4-shot, textbook (37.3%) leads OBELICS (16.4%) by 20.9 points.

On mathematics benchmarks (Table 2, bottom-left block), the pattern is consistent but less extreme. On MathVista: textbook achieves 24.3% (0-shot), 43.4% (1-shot), 33.2% (2-shot), 29.2% (4-shot). The 1-shot result is notably high (43.4%) — this may reflect the cheat-test phenomenon explored later (the model may be exploiting context more effectively), though standard 1-shot evaluation with retrieved examples is not the same as the cheat test. Compared to OBELICS (21.6%, 28.5%, 31.1%, 27.6%), the textbook leads by 2.7 to 14.9 points, with the largest advantage again in the few-shot regime. On MathVision: textbook leads OBELICS by a more modest 1.1–5.5 points across shot counts. On MathVerse: results are mixed — OBELICS occasionally outperforms the textbook (e.g., 2-shot: OBELICS 20.7% vs. textbook 19.8%), though the textbook leads at 1-shot (28.5% vs. 19.4%).

General VQA shows a distinctive pattern: textbook lags in zero-shot, catches up or surpasses in few-shot. On OKVQA (Table 2): textbook achieves 10.2% (0-shot) vs. OBELICS's 13.0% — a 2.8-point deficit. But by 1-shot, the gap narrows to 0.5 points (31.2% vs. 31.7%), and by 4-shot, textbook leads by 2.4 points (39.9% vs. 37.5%). On TextVQA: textbook trails OBELICS in 0-shot (11.8% vs. 9.2% — actually trails OBELICS at 9.2%) but the numbers show the textbook at 11.8% vs. OBELICS at 9.2%, a small lead. More clearly, at 2-shot, textbook (32.1%) slightly exceeds OBELICS (30.2%), and at 4-shot, textbook (33.5%) leads OBELICS (32.2%) by 1.3 points. On TextVQA-OCR (a subset focused specifically on the text-reading component): textbook trails OBELICS in 0-shot (14.1% vs. 11.0% — actually is slightly ahead of OBELICS's 11.0%) but by 4-shot, textbook (42.8%) surpasses OBELICS (41.0%) by 1.8 points.

This zero-shot deficit, few-shot advantage pattern is the central quantitative signal that the textbook enhances in-context learning. The interpretation (developed further in Section 5.3) is that the textbook's coherent interleaved structure teaches the model to attend more effectively to its multimodal context, but this skill only manifests when context is provided (i.e., in few-shot settings). In zero-shot, the textbook model may underperform on general VQA because its training data is domain-specialized (instructional content) and lacks the diversity of everyday visual scenes found in web-crawled corpora.

MMC4-core-ff performs weakly across the board. Despite being a filtered subset of MMC4 intended to improve image-text alignment, it often underperforms unfiltered MMC4 in few-shot settings (e.g., ScienceQA 4-shot: MMC4 11.6% vs. MMC4-core-ff 10.2%). The paper does not analyze this in detail, but it suggests that aggressive filtering of webpage data can remove useful training signal even as it improves average image-text similarity — a cautionary note about over-filtering that the paper itself arguably avoids through its "content preservation over format purity" design philosophy.

Generalization to Idefics2-8B: Continual Pretraining and Training from Scratch

Table 3 extends the analysis to Idefics2-8B, testing whether the textbook's benefits are specific to LLaVA-1.5-7B or generalize to a different architecture with different pretraining history. The results are reported for 8-shot settings using randomly selected examples (following the evaluation protocol of Laurençon et al., 2024).

Continual pretraining from Idefics2-8B-base (which is already pretrained on OBELICS): the textbook yields consistent but modest improvements. On OKVQA, textbook (55.1%) edges past OBELICS (54.6%) by 0.5 points. On MathVista, the gain is more notable: textbook (29.7%) leads OBELICS (27.6%) by 2.1 points and MMC4-cf (27.8%) by 1.9 points. On MathVision and MathVerse, textbook leads by 1.9 points each (16.2% vs. 14.3%, and 19.4% vs. 17.5%). The pattern is clear: the textbook provides additional benefit even on top of a model already pretrained on OBELICS, with the largest gains again on the most knowledge-intensive benchmark (MathVista). This is evidence that the textbook's knowledge content is complementary to, not redundant with, what OBELICS provides.

Training Idefics2-8B from scratch (randomly initialized projector, no prior interleaved pretraining): the textbook's advantage is more pronounced. On MathVista, textbook (26.1%) leads OBELICS (24.2%) by 1.9 points and MMC4-cf (24.0%) by 2.1 points. On MathVision, textbook (14.4%) leads OBELICS (13.6%) by 0.8 points. However, on OKVQA, textbook (10.1%) slightly trails OBELICS (10.5%) — the same zero-shot deficit pattern observed with LLaVA-1.5-7B, though here in an 8-shot setting (which may function differently than few-shot with retrieved examples). The overall pattern confirms that the textbook's advantages are not an artifact of LLaVA-1.5-7B's specific architecture or pretraining recipe.

The Cheat Test: Quantifying Genuine Interleaved Context Awareness

Section 5.3 introduces the cheat test as a diagnostic for whether VLMs truly attend to their interleaved context. The results in Table 4 are striking and merit careful examination.

Setup reminder: In the 1-shot cheat test, the prompt contains exactly one example {I_t, q_t, a_t} which is identical to the test case — if the model notices this, it can simply output a_t without any reasoning. In the 2-shot cheat test, the prompt contains the identical example plus one random (non-identical) example {I_e, q_e, a_e}, making the cheat harder to detect because the context is now longer and contains a distractor.

1-shot cheat results (Table 4, top block). The textbook-trained model achieves near-ceiling performance on MathVista (94.1%) and MathVision (98.4%), and very high scores on OKVQA (79.2%) and MathVerse (76.8%). In contrast, the OBELICS-trained model scores only 67.7% on MathVista, 66.5% on MathVision, and 71.5% on OKVQA — suggesting it fails to notice the identical example in roughly 30% of cases. MMC4-cf performs even worse: 72.6% on MathVista, 69.3% on MathVision, 69.0% on OKVQA. The gap between textbook and baselines on MathVista is 21.5 points (vs. MMC4-cf) and 26.4 points (vs. OBELICS). On MathVision, the gap is 29.1 points (vs. MMC4-cf) and 31.9 points (vs. OBELICS).

These numbers mean that VLMs pretrained on webpage interleaved data systematically overlook information in their own context even when that information directly provides the answer. This is not a subtle effect — it is a fundamental failure of context integration that standard few-shot evaluation (where examples are similar but not identical to the test case) would not detect, because the model can succeed through partial pattern matching rather than genuine context attention.

2-shot cheat results (Table 4, bottom block). Lengthening the context with a distractor example causes performance drops for all models, but the magnitude differs dramatically. The textbook model drops from 79.2% → 84.3% on OKVQA (actually an increase), from 94.1% → 77.1% on MathVista, and from 98.4% → 70.7% on MathVision. The OBELICS model drops from 71.5% → 71.3% on OKVQA (essentially flat, already low), from 67.7% → 56.7% on MathVista, and from 66.5% → 39.9% on MathVision — a 26.6-point collapse. MMC4-cf shows even larger drops: 69.0% → 53.5% on OKVQA and 72.6% → 55.7% on MathVista.

The asymmetry of the drops is the key finding. The textbook model maintains high performance even with a longer, more distracting context — it remains above 70% on all benchmarks except TextVQA. The webpage-trained models, when presented with a distractor, lose the ability to identify the relevant example almost entirely on math benchmarks (OBELICS drops to 56.7% on MathVista, 39.9% on MathVision — barely above random guessing for multi-choice questions). This demonstrates that the textbook does not merely teach the model to look at its context; it teaches the model to selectively attend to the relevant portion of context even when irrelevant information is present. This is precisely the skill needed for real-world few-shot learning, where the model must extract useful signal from a prompt containing multiple (possibly imperfect) examples.

Image Shuffling: Proving Sequential Coherence Matters

Figure 3 presents the image-shuffling ablation, which is one of the paper's most theoretically important results because it provides causal evidence that the textbook's sequential coherence is what drives its benefits, not merely higher image quality or better filtering.

Experimental design. For each dataset (MMC4, OBELICS, Textbook-6.5M), the authors randomly select 20%, 50%, or 100% of samples, shuffle the image order within those selected samples, and then pretrain models on the partially or fully shuffled dataset. The evaluation metric is the average accuracy across all seven benchmarks. If a dataset's training signal depends on image order, shuffling should degrade performance; if the images are essentially independent, shuffling should have no effect.

Results (Figure 3). The curves diverge dramatically:

MMC4 is essentially unaffected by shuffling. Whether 0%, 20%, 50%, or 100% of samples are shuffled, the average accuracy remains flat at approximately the same level. This confirms the paper's diagnosis that MMC4's images lack meaningful sequential dependency — the model learns nothing from image order because image order carries no useful signal. Any interleaved context awareness trained on MMC4 must arise from text-image co-occurrence alone, not from image-to-image relationships.
OBELICS shows a moderate, roughly linear decline as the shuffling ratio increases. At 100% shuffling, accuracy drops visibly below the unshuffled baseline, but the decline is not catastrophic. This suggests OBELICS contains some sequentially coherent image sequences (perhaps from articles, tutorials, or documentation pages), but they are diluted by many samples where images are independent. Shuffling destroys this minority of coherent sequences without affecting the majority of independent-image samples.
The textbook shows a severe, monotonic, and accelerating decline. At 20% shuffling, accuracy drops noticeably. At 50% shuffling, the drop is substantial. At 100% shuffling, performance collapses to a level well below both MMC4 and OBELICS. The curve is concave (downward-accelerating), suggesting a threshold effect: a few shuffled samples degrade context-learning for all samples, and full shuffling destroys the core capability the textbook was designed to teach.

This result is the strongest causal evidence in the paper for the central claim that video-derived sequential coherence is the active ingredient in the textbook's effectiveness. It rules out alternative explanations: if the textbook's advantage came solely from better image quality (sharper, cleaner keyframes) or better text quality (more fluent ASR), shuffling should not cause the catastrophic drop observed. The fact that it does demonstrates that the model is learning something specific to the temporal order of the frames — the logical progression from one diagram to the next — that cannot be learned from static, unordered image collections no matter how high-quality the individual images are.

Instruction Tuning Transfer: Does Textbook Knowledge Survive SFT?

Table 5 addresses a practical concern: if the textbook's knowledge is learned during pretraining, does it persist through the subsequent instruction-tuning stage that all production VLMs undergo? Or does instruction tuning on a generic corpus (LLaVA-665K) wash out the textbook-specific gains?

Results (Table 5). After instruction tuning on LLaVA-665K (the standard instruction dataset for LLaVA-1.5):

Baseline LLaVA-1.5 (pretrained on its standard data, then SFT): 61.1% on OKVQA, 23.2% on MathVista.
+ MMC4-Core-ff pretraining → SFT: 61.5% on OKVQA (+0.4), 24.8% on MathVista (+1.6).
+ OBELICS pretraining → SFT: 61.8% on OKVQA (+0.7), 25.6% on MathVista (+2.4).
+ Textbook-6.5M pretraining → SFT: 62.2% on OKVQA (+1.1), 28.7% on MathVista (+5.5).

The textbook's MathVista gain (+5.5 points over baseline) is more than double OBELICS's gain (+2.4) and more than triple MMC4-core-ff's (+1.6). On OKVQA, the differences are compressed — all methods cluster within 1.1 points of baseline — likely because OKVQA primarily tests visual recognition and general knowledge, not the specialized instructional knowledge the textbook provides. The key finding is that the textbook's pretraining advantage does not wash out during SFT; it persists and, on the benchmark most aligned with the textbook's content (MathVista), substantially exceeds what webpage interleaved data provides.

This result has practical significance: it means the textbook is not merely a pretraining curiosity that gets overwritten by instruction tuning — it provides a durable knowledge foundation that improves downstream task performance even after standard SFT. The fact that MMC4-core-ff and OBELICS also improve MathVista over baseline (by +1.6 and +2.4) suggests interleaved pretraining in general is helpful for math reasoning, but the textbook's +5.5 gain suggests the specific content of the interleaved data (instructional knowledge vs. web content) matters enormously.

Ablation Studies and Robustness Checks

ASR refinement (Table 6): Removing the LLM-based ASR refinement step (using raw Whisper transcriptions instead) causes a 4.9% average performance drop across seven benchmarks in the 1-shot setting (from 31.1% to 26.2%). This is the single largest ablation effect reported. The raw ASR text has a perplexity of 16.86 (vs. 13.92 for refined ASR and 11.27 for OBELICS), confirming that the colloquial, fragmented nature of spoken lecture transcripts impairs downstream model performance even though the instructional content is present. The refinement step is thus essential, not cosmetic.

OCR integration (Table 6): Removing OCR text from the interleaved samples ("w/o OCR") causes a 2.3% average performance drop (from 31.1% to 28.8%) in the 1-shot setting. The paper notes that the benefit is concentrated on benchmarks like TextVQA and MathVista, where reading on-screen text/symbols is directly tested. However, the authors also caution that "low-quality OCR can introduce noise and even significantly degrade performance," highlighting that the choice of OCR model (InternVL2-40B) and the filtering of redundant OCR outputs are both critical to realizing the benefit.

Keyframe extraction algorithm (Table 6): This is the most technically revealing ablation. Three extraction methods are compared at equivalent downstream training:

Pixel-level (OpenCV absdiff): Extracts 18M keyframes — 2.8× the SSIM count — but causes a 9% performance drop (to 22.1% 1-shot average accuracy). The excess frames introduce redundancy without adding information, diluting the training signal.
CLIP-based (semantic similarity): Extracts only 1.7M keyframes — 3.8× fewer than SSIM — and causes a 6.5% drop (to 24.6%). CLIP misses structurally distinct but semantically similar frames (e.g., a geometry diagram before and after annotation are both "geometry diagrams" to CLIP).
SSIM (the paper's choice): 6.5M keyframes, 31.1% average accuracy — the best by a margin of 6.5 points over CLIP and 9.0 points over pixel-level.

The SSIM advantage is large enough to be practically decisive: choosing the wrong keyframe algorithm can erase most of the textbook's benefit over OBELICS (which achieves 22.8% in the 1-shot average, per Table 6). This confirms the paper's claim that domain-matched extraction is essential — SSIM works because instructional content's meaningful changes are structural (text appearing, annotations being added) rather than pixel-level or semantic-level.

Image shuffling ratio (Figure 3): Analyzed in detail above under Main Results. Key robustness finding: the performance degradation is monotonic with increasing shuffle ratio, with a concave (accelerating) shape for the textbook, flat for MMC4, and shallow-linear for OBELICS. The effect is robust across all seven benchmarks (reported as average accuracy).

Dual-LLM consensus filtering for videos: While not presented as a formal ablation, the paper's design choice to use two LLMs (DeepSeek-V2 and Llama3-70B-Instruct) with a consensus requirement for video filtering (Section 7.1) represents an implicit robustness mechanism. The paper does not report performance with single-LLM filtering, so the marginal benefit of the consensus approach is unknown. However, the choice itself reflects awareness that LLM-based quality judgments are noisy and that redundant evaluation improves reliability.

Perplexity as a data quality proxy: Table 6 reports perplexity values for different text variants (raw ASR: 16.86, refined ASR: 13.92, OBELICS: 11.27, MMC4-Core-ff: 12.56). The ordering correlates with downstream performance (lower PPL → higher accuracy), suggesting that text fluency is a meaningful quality signal for multimodal pretraining — but the correlation is not perfect (refined ASR has higher PPL than OBELICS yet produces better downstream performance on knowledge benchmarks), indicating that content quality matters beyond surface fluency.

Negative result: The paper does not report any formal negative result (an approach that was tried and failed). The closest is the implicit finding that MMC4-Core-ff, despite being a filtered, higher-quality subset of MMC4, often underperforms unfiltered MMC4 in few-shot settings (e.g., ScienceQA 4-shot: 10.2% vs. 11.6%), suggesting that aggressive filtering of webpage data can be counterproductive — an observation the paper does not analyze in depth.

Critical Assessment

Does the paper demonstrate that the textbook corpus improves knowledge- and reasoning-intensive task performance?

Yes, strongly. The ScienceQA and MathVista results (Tables 2 and 5) show large, consistent gains over matched-scale baselines. ScienceQA 0-shot improves from near-zero (OBELICS: 2.8%) to 26.3%, and MathVista improves by 2.7–14.9 points depending on shot count. The SFT transfer results (Table 5) confirm these gains survive instruction tuning, with MathVista improving +5.5 points over baseline (more than double OBELICS's +2.4). The effect replicates across two model families (LLaVA-1.5-7B and Idefics2-8B) and two training regimes (continual pretraining and training from scratch).

However, the claim is specifically about knowledge-intensive benchmarks — and the paper's own results show that on general VQA (OKVQA), the textbook lags behind OBELICS in zero-shot (10.2% vs. 13.0%). The advantage only appears in few-shot settings, where it is attributable to enhanced context awareness rather than superior knowledge. This means the textbook's benefit is domain-specific: it teaches instructional knowledge and context-attention skills, but it does not improve general visual recognition. This is not a weakness per se — it is exactly what one would expect from a domain-specialized corpus — but it means the claim "pretraining on the textbook improves VLMs" requires the qualification "on knowledge-intensive benchmarks, particularly in few-shot settings." The paper does not oversell this claim, but readers should not interpret it as a universal improvement across all VLM capabilities.

Does the paper demonstrate that the textbook enhances interleaved context awareness?

Yes, and this is the strongest empirical contribution. The cheat test (Table 4) provides compelling evidence that is difficult to explain through confounds. The 1-shot cheat results — 94.1% on MathVista, 98.4% on MathVision for the textbook model vs. 67.7% and 66.5% for OBELICS — demonstrate a qualitative difference in whether the model notices that its context contains the answer. The 2-shot cheat results show that the textbook model maintains this awareness even with a distractor present (77.1% on MathVista), while OBELICS drops to 56.7% and MMC4 to 55.7% — barely above chance.

The image-shuffling experiment (Figure 3) provides the causal mechanism: the textbook's sequential coherence is why context awareness develops, because shuffling (which destroys coherence) causes catastrophic performance degradation. The flat curve for MMC4 confirms that webpage-trained models never learn to attend to image order because there is no useful signal there to learn from.

A limitation: The cheat test is artificial — real-world few-shot examples are not identical to the test case. The paper does not demonstrate that enhanced cheat-test performance translates to better genuine few-shot learning with semantically similar but non-identical examples. The standard few-shot results (Table 2) show improvements, but these are confounded with knowledge content (the textbook may simply teach more relevant knowledge, making the model better at the tasks regardless of context awareness). A cleaner test would be an experiment where the textbook-trained model is evaluated on a task whose knowledge content is held constant across pretraining conditions (e.g., a synthetic task where all models have the necessary knowledge, and only context-attention ability varies). This experiment was not run.

Does the paper demonstrate that video-sourced, temporally coherent interleaved sequences provide a qualitatively different training signal from webpage-crawled interleaved data?

Yes, convincingly. The combination of InSI-SIM scores (Table 1: textbook 0.686 vs. OBELICS 0.345, a near-doubling), the shuffling experiment (Figure 3: flat for MMC4, catastrophic drop for textbook), and the cheat test (Table 4: textbook-trained model attends to context, webpage-trained models largely ignore it) triangulates on this claim from multiple angles. The InSI-SIM shows that the structural property (inter-image coherence) exists. The shuffling experiment shows that this property is causal for model performance. The cheat test shows that the behavioral consequence (context awareness) is real.

A limitation: The paper does not directly compare against a "video-naive" baseline that controls for knowledge content while varying temporal coherence. For example, one could take the textbook's images and text, randomly permute the image order before training (already done via the shuffling experiment), OR randomly pair images with text from different videos (destroying image-text alignment while preserving image quality and text quality). The shuffling experiment tests image-order coherence; a cross-video pairing experiment would test image-text alignment coherence. The paper does not run this second control, leaving open the question of whether the textbook's advantage comes primarily from better image-text alignment (each frame paired with its causally related ASR) or from image-order coherence, or from their interaction.

Are there genuine weaknesses in the experimental design?

1. No statistical significance reporting. The paper reports accuracy numbers to one decimal place without confidence intervals, standard deviations, or significance tests. Given test set sizes (OKVQA: ~5,000 questions, but MathVista: ~1,000, ScienceQA-IMG: ~2,000, and difficulty bins with ~100 questions each as discussed in the MathVista setting), differences of 1–2 percentage points could be noise. The larger gaps (5–20 points) are almost certainly real, but the smaller ones (e.g., textbook vs. OBELICS on TextVQA 4-shot: 33.5% vs. 32.2%, a 1.3-point gap) should be treated cautiously. The paper would be strengthened by reporting standard errors or bootstrap confidence intervals.

2. Single data scale. All experiments use exactly 610K samples per dataset, matched across conditions. This controls for sample count but does not explore how the textbook's advantage scales with data quantity. Would training on 1.2M textbook samples (if available) yield proportionally larger gains? Would training on 1.2M MMC4 samples close the gap? Without a data-scaling curve, the efficiency claim (textbook quality substitutes for web-data quantity) is supported at a single point but not extrapolated. The implicit claim that "610K textbook samples ≈ much more than 610K web samples" is plausible but unquantified.

3. The 610K sample count is arbitrary and not justified. Why 610K? The paper notes that this is the number of samples produced by the curation pipeline, but does not explain why this count was used for all baselines rather than, say, matching on total text tokens or total images. Table 1 shows that the textbook averages 1,297 text tokens and 10.7 images per sample, while MMC4 averages 417 tokens and 5.7 images, and OBELICS averages 816 tokens and 2.5 images. Matching on sample count means the textbook-trained model sees approximately 3× more text tokens and 1.9× more images than the MMC4-trained model, and 1.6× more text tokens and 4.3× more images than the OBELICS-trained model during the matched training runs. This is not controlled for — the paper's gains could be partially attributable to the textbook simply being a denser dataset per sample, rather than being qualitatively better. A fairer comparison would match on total tokens or total images, or explicitly report per-token and per-image efficiency.

4. No comparably curated non-video baseline. All baselines are off-the-shelf webpage datasets. The paper does not construct a "textbook from webpages" baseline — for example, crawling educational websites (Wikipedia, Khan Academy, textbook PDFs) with high-quality filtering to produce an interleaved corpus of similar knowledge density but without the temporal coherence of video. Such a baseline would help disentangle the knowledge density benefit (instructional content vs. general web content) from the temporal coherence benefit (video frames vs. static webpage images). Without it, the paper can claim the textbook is better than webpage data, but cannot claim that video specifically is the necessary ingredient — it might be that any curated, knowledge-dense interleaved corpus would produce similar gains.

5. The curation pipeline's cost is not discussed as a limitation in the experiments. The paper uses GPT-4o (taxonomy, metadata filtering), Qwen2-72B-Instruct (ASR refinement), DeepSeek-V2 + Llama3-70B (ASR scoring), VideoLlama2-7B (clip captioning), GTE-Qwen2-7B (similarity scoring), InternVL2-40B (OCR and keyframe scoring), and Whisper-large-v3 (ASR). The computational cost of running these models on 159K videos (even after filtering to 75K) is likely enormous — millions of GPU-hours. This cost is not amortized or compared to the cost of training on larger web corpora. A honest accounting would ask: given a fixed total budget (curation compute + training compute), does the textbook approach outperform simply training on more web data? The paper provides no evidence on this point, which limits the practical applicability of the findings for budget-constrained practitioners.

6. The difficulty of reproducing the dataset. The curation pipeline depends on specific model versions (GPT-4o, Qwen2-72B-Instruct, DeepSeek-V2, Llama3-70B, InternVL2-40B, etc.) that will evolve or become unavailable. The reliance on YouTube's search API introduces non-determinism (search results change over time). While the paper releases the final dataset to mitigate this, the pipeline itself — which is presented as a contribution — is fragile to API and model version drift in ways the paper does not discuss.

7. Limited model scale. All experiments use ~7–8B parameter models. The paper makes no claim about how the textbook's benefits scale with model size, but this is an important open question: does the textbook's advantage grow with model scale (because larger models can better absorb the structured knowledge), shrink (because larger models already acquire this knowledge from web data), or remain constant? The Phi-series analogy suggests smaller models benefit disproportionately from textbook-quality data, but this is untested in the multimodal domain.

What experiments would have strengthened the paper?

A data scaling curve: Train on 100K, 300K, 610K textbook samples and the same counts of OBELICS/MMC4 to show how the gap evolves with data quantity. This would test whether the textbook's advantage is largest at small data scales (the "data quality substitutes for quantity" regime) or persists at larger scales.
A token-matched comparison: Control for total text tokens and total images seen during training, not just sample count, to isolate data quality from data density per sample.
A non-video knowledge-dense baseline: Curate an interleaved corpus from educational websites or textbooks with comparable subject coverage and knowledge density, but without video-derived temporal coherence. This would isolate the specific contribution of video.
Confidence intervals on all reported numbers: Especially given the small test sets for math benchmarks.
An analysis of failure modes: The paper reports accuracy improvements but does not analyze what kinds of questions the textbook helps with most (beyond the coarse difficulty split in the FLOPs-matched analysis in Section 7 of the main paper — but that analysis is not reproduced in the supplementary-level detail this section draws from). Are the gains concentrated on formula-recognition questions, multi-step reasoning, or factual recall? This would clarify the mechanism.
A cost analysis: Amortize the curation compute cost and compare total cost (curation + training) against simply training on more web data. This is the practical question a practitioner would ask.

Summary

The experiments provide strong evidence for the paper's central claims — that instructional video can be converted into a high-quality interleaved pretraining corpus, that this corpus improves performance on knowledge-intensive benchmarks, and that it enhances genuine interleaved context awareness in VLMs. The cheat test and shuffling experiment are particularly elegant demonstrations that the mechanism is real and causally attributable to the temporal coherence of the source medium.

The primary weaknesses are: (1) the matched-sample-count rather than matched-token-count comparison, which confounds data density with data quality; (2) the absence of statistical significance reporting, which makes small differences unreliable; and (3) the lack of a non-video knowledge-dense baseline, which leaves open the question of whether video specifically is necessary or whether any curated instructional corpus would produce similar gains. These weaknesses do not undermine the core findings but do constrain the precision of the claims and the practical guidance the paper can offer for data curation budget allocation.

6. Limitations and Trade-offs

The Difficulty Estimation Overhead Is Not Accounted for in Headline Efficiency Numbers

The entire difficulty estimation mechanism — which the paper positions as the linchpin enabling compute-optimal allocation — requires generating 2,048 samples per question and scoring them with the PRM to determine difficulty bins (Sections 3.2, 4). The authors explicitly acknowledge this cost is excluded from budget accounting:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The consequence is that the reported 4× efficiency gains — achieving best-of-N performance at 4× fewer generations — are computed after difficulty is known, without amortizing the cost of learning it. In deployment, the total compute would be (2048 × scoring_cost) + (N × strategy_cost). When difficulty estimation requires 2,048 generations, the overhead dwarfs the tested budgets (1–512 generations). For low-budget regimes (e.g., 16 generations), the amortized cost would exceed 100× the strategy execution budget, completely negating any efficiency advantage.

The authors acknowledge this as an "exploration-exploitation tradeoff" (Section 3.2) and propose predicted difficulty bins using the PRM's averaged final-answer score over 2,048 samples as a partial solution — this removes the need for ground-truth labels but does not reduce the sample cost. The paper suggests future work on "pretraining or finetuning models to directly predict difficulty of a question" (Section 8) but provides no such model or evaluation. Until cheap difficulty estimation is demonstrated, the compute-optimal framework should be understood as an analytical upper bound, not a deployable system.

Mitigation status: Not addressed. The paper explicitly identifies this as future work and does not claim to have solved it.

Hard Problems Remain Completely Unsolved Across All Methods

Across the entire paper — search strategies, revision strategies, and their compute-optimal combinations — the most difficult questions in difficulty bin 5 show near-zero improvement regardless of the test-time compute budget allocated. This is consistent and striking:

In the search experiments (Figure 3, right), bin 5 accuracy hovers at 1–3% for all methods and all budgets, from 4 to 256 generations.
In the revision experiments (Figure 7, right), bin 5 accuracy is approximately 2–3% irrespective of the sequential-to-parallel ratio at 128 generations.
In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5% across all budgets and all R values, while the 14× larger model's greedy decoding achieves nontrivial performance on these same questions (represented by the star markers in Figure 9).

The paper is transparent about this (Section 7 takeaway box):

"On hard prompts, test-time compute is almost always worse (than pretraining)"

But this is not merely a comparative statement — it is a fundamental capability bound. The base model's pass@1 on bin 5 questions is near zero, meaning there are essentially no correct solutions in the proposal distribution to find or refine. Search can filter the proposal distribution; revisions can improve it locally; neither can create capability that is not already present at some non-negligible frequency.

The consequence is that test-time compute cannot substitute for pretraining on genuinely novel or out-of-distribution problems that exceed the base model's training distribution. For domains where problem difficulty is bimodal (many easy routine queries plus a long tail of genuinely hard cases), the compute-optimal framework offers no path forward for the hard tail — those queries must be routed to larger models or handled by humans regardless of the test-time budget.

Mitigation status: Not addressed. This is an inherent limitation of the approach, not a solvable engineering problem. The paper documents it clearly, which is to its credit, but offers no mitigation.

The 14× Larger Model Baseline Is Not Compute-Optimally Trained

The FLOPs-matched comparison (Section 7) that demonstrates test-time compute outperforming a 14× larger model uses a pretraining baseline where only model parameters are scaled, with training data held fixed. This follows the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining (Hoffmann et al., 2022), which would scale both parameters and data equally under the total pretraining FLOPs budget. The authors are explicit:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

The consequence is that the pretraining baseline is weaker than it should be for a fair comparison under a total compute budget. A Chinchilla-optimal model trained with 14× more total FLOPs (scaling data proportionally) would likely outperform the parameter-only-scaled model used in the paper, making the reported test-time compute advantages — e.g., +27.8% relative improvement on easy questions at R ≪ 1 (Figure 1, top-right bar chart) — shrink or potentially reverse. The paper's headline finding that "test-time compute can outperform a 14× larger model" is thus conditional on the specific (suboptimal) pretraining recipe of the baseline.

Furthermore, the 14× larger model is evaluated using greedy decoding only — no majority voting, no best-of-N, no search, no revisions. Giving the larger model even a modest test-time compute budget (e.g., best-of-8 with a verifier) would create a substantially stronger baseline that is never tested. The paper demonstrates that a smaller model with compute-optimal test-time scaling can beat a larger model with greedy decoding, which is not the same as demonstrating that test-time compute is generally preferable to pretraining compute under matched conditions.

Mitigation status: Partially. The authors acknowledge the limitation and frame the comparison as "representative" rather than optimal. The paper does test three R values (0.16, 0.79, 22) to cover different inference-to-pretraining regimes, which partially addresses the concern by showing robustness across deployment scenarios, but the underlying baseline weakness is not corrected.

The Paper Studies Search and Revisions Independently, Never Combining Them

The two primary mechanisms for test-time compute scaling — PRM-guided search (Section 5) and iterative revisions (Section 6) — are studied in isolation throughout the paper. The authors explicitly acknowledge this gap:

"we did not experiment with PRM tree-search techniques in combination with revisions" (Section 8)

The consequence is that the reported results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary strengths that the paper itself documents: revisions are most effective on easy problems where local refinement suffices (Figure 7, right), while search is most effective on medium-difficulty problems where exploration of qualitatively different solutions helps (Figure 3, right). A combined system could, for example, use the revision model as the proposal distribution within beam search — at each search step, the model conditions on previously rejected branches to generate improved candidates — or use the PRM to guide which revision branches to pursue rather than blindly generating long sequential chains.

The paper's individual results hint at the potential for combination. The revision model achieves roughly 44% accuracy at the optimal sequential-to-parallel ratio with 256 generations (Figure 8), and PRM search achieves roughly 39.5% at the same budget with compute-optimal allocation (Figure 4). Neither individually reaches the performance level that their complementary difficulty-dependent strengths suggest might be possible through integration.

Mitigation status: Not attempted. The paper identifies this as a key direction for future work (Section 8) but provides no preliminary results or analysis of how the combination would work.

The Revision Model Exhibits a 38% Correct-to-Incorrect Reversion Rate, Requiring Imperfect Mitigations

The revision model was trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target (Section 6.1). The training data construction pairs independently sampled incorrect and correct solutions, selecting the incorrect answer with the smallest character-level edit distance to the correct answer to simulate a coherent revision trajectory. This design choice has a direct and damaging consequence at inference time:

"approximately 38% of correct answers get converted back to incorrect ones" (Section 6.1)

The mechanism is straightforward: when the model produces a correct answer during a revision chain, the next revision step encounters a situation it was never trained for — a correct answer in context. Since all training examples had incorrect answers in context, the model has no signal for what to do when the current answer is already correct. It defaults to "revising" anyway, often breaking a correct solution.

The paper mitigates this with within-chain selection: use majority voting or a verifier to select the best answer from any point in the revision chain, rather than always taking the final revision (Section 6.1). This works — Figure 6 (left) shows pass@1 at each step gradually improves, and Figure 8 shows compute-optimal revisions outperforming parallel baselines — but it is an imperfect patch. If 38% of correct answers are reverted, the within-chain verifier must correctly identify and rescue them, which depends on verifier quality. A verifier with imperfect recall will miss some reverted answers, leaving performance below what a revision model that knew when to stop revising could achieve.

Furthermore, the ReST^EM experiment (Appendix K, Figure 16) shows that attempting to optimize the revision model with on-policy RL training causes performance to degrade substantially — at 256 generations, fully sequential revisions drop to roughly 33.5% compared to roughly 38.5% at the optimal ratio. The authors hypothesize that "on-policy data collection exacerbates spurious correlations in revision data" (Appendix K), indicating that the revision training approach is fragile to training methodology in ways that are not fully understood. The positive results depend on specific choices (offline data construction, edit-distance-based pairing, SFT rather than RL) that may not transfer robustly.

Mitigation status: Partially. Within-chain selection reduces the impact of reversions but does not address the root cause (the model lacks a "stop revising" signal). The paper does not explore training the revision model on trajectories that include correct-to-correct transitions or explicit termination tokens, which would be the principled solution. The ReST^EM negative result suggests this is a non-trivial research problem.

The Paper Operates on a Single Benchmark (MATH) with a Single Model Family (PaLM 2-S*)

All experiments, including the central claims about compute-optimal scaling, difficulty-dependent strategy selection, and FLOPs-matched comparisons, are conducted on a single benchmark (MATH) with a single base model family (PaLM 2-S)* and a single verifier training methodology (Monte Carlo rollout PRM training). The test set consists of only 500 questions, and the baseline revision model uses a specific fine-tuning recipe (Section 6.1) that the authors acknowledge differs from prior work (Qu et al., 2024) due to computational constraints.

The authors state they "believe this model is representative of the capabilities of many contemporary LLMs" (Section 4), but this claim is unverified. Several aspects of the findings could be model-, domain-, or verifier-specific:

PRM quality and over-optimization behavior depend on PaLM 2-S*'s output distribution. A model with better-calibrated outputs might exhibit different over-optimization thresholds, changing the difficulty-strategy mapping.
Revision model capability depends on the base model's in-context learning and self-correction abilities, which vary substantially across model families and scales.
MATH consists entirely of competition-level symbolic reasoning problems. The difficulty-dependent patterns — beam search hurting easy problems, revisions helping easy problems, the optimal sequential-to-parallel ratio shifting with difficulty — may not generalize to other reasoning domains (code generation, logical deduction, scientific QA) or to tasks requiring factual recall rather than multi-step inference.

The small test set (500 questions) compounds this concern. When split into five difficulty quintiles of ~100 questions each and further split by two-fold cross-validation for strategy selection (~50 questions per fold per bin), the compute-optimal policy is selected based on very small samples. The paper does not report confidence intervals or standard errors for any accuracy numbers, making it impossible to assess whether the reported differences — particularly the smaller ones (e.g., Figure 8: predicted bins at 41% vs. oracle at 44% at 256 generations, a 3-point gap based on ~100 questions) — are statistically reliable or within sampling noise.

Mitigation status: Not addressed. The paper does not include model family or benchmark diversity as a scope limitation, does not report uncertainty estimates, and does not discuss how findings might transfer to other settings. This is a standard limitation of single-benchmark, single-model empirical studies, but the paper's prescriptive claims about compute-optimal strategy selection would be strengthened by replication across at least one additional benchmark and model family.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around multimodal pretraining data from a scale-centric paradigm to a source-quality paradigm. The dominant assumption in the field — carried over from the text-only domain's "more data is better" ethos and reinforced by the progression from COYO-700M to LAION-5B to OBELICS to OmniCorpus — was that the path to better VLMs involved crawling more webpages, applying more aggressive filtering, and scaling up compute budgets. Each successive dataset was larger than its predecessor, implicitly betting that the quality limitations of web-crawled interleaved corpora (loose image-text relations, incoherent image sequences, low knowledge density) were noise problems solvable through scale and better filters.

The paper's evidence dismantles this assumption at its foundation. The image shuffling experiment (Section 5.3, Figure 3) is the diagnostic that makes the case: shuffling image order within MMC4 samples causes essentially no performance degradation, while shuffling the textbook dataset causes a severe and monotonic drop that accelerates as the shuffling ratio increases. This is not a marginal filtering quality difference — it reveals a qualitative difference in the structural properties of the data. Webpage images lack sequential dependency because the source medium (arbitrary webpages) does not impose one; video frames possess strong sequential dependency because the source medium (instructional lectures) is inherently structured as linear explanatory narratives. No amount of post-hoc filtering applied to Common Crawl can manufacture temporal coherence between images that were never temporally related.

The paper quantifies this structural difference through the In-sample Image Similarity (InSI-SIM) metric (Table 1), which combines semantic (CLIP) and structural (SSIM) similarity. The textbook dataset's InSI-SIM of 0.686 — more than double OBELICS's 0.345 and MMC4's 0.319 — is not achieved through filtering; it is an intrinsic property of sourcing data from temporally coherent video sequences rather than static webpages. This metric, while simple in concept, provides the field with a new axis for evaluating pretraining data quality that goes beyond standard statistics like dataset size, text length, and image count. Future dataset releases can and should report InSI-SIM (or similar inter-image coherence measures) alongside traditional statistics, because the paper demonstrates that this dimension directly predicts downstream model capabilities — specifically, genuine interleaved context awareness.

The paper's most conceptually important reframing is the decoupling of interleaved format from interleaved benefit. Prior work (Alayrac et al., 2022; Awadalla et al., 2023; Laurençon et al., 2024) treated the interleaved format itself as the enabler of in-context learning and multi-image reasoning — train on interleaved data, and these capabilities naturally emerge. The cheat test (Section 5.3, Table 4) shows this logic is incomplete. On MathVision, a VLM pretrained on the textbook scores 98.4% in the 1-shot cheat setting, while the OBELICS-trained model scores only 66.5% — meaning the model fails to notice that its context already contains the answer more than 30% of the time. This demonstrates that training on an interleaved format is necessary but not sufficient for developing genuine context awareness. The model must also learn why attending to context is useful, and this learning signal comes from the coherence of the interleaved sequences — specifically, whether attending to adjacent images and text is consistently rewarded with predictive signal.

This finding reconciles a latent tension in the literature. Why do some studies report strong in-context learning in VLMs (Alayrac et al., 2022; Sun et al., 2024) while others find limited few-shot benefits from interleaved pretraining? The answer, which this paper makes empirically concrete, is that not all interleaved training is equal. Interleaved data where images and text are causally linked and image sequences follow logical progressions teaches the model to integrate across modalities in context; interleaved data where images are decorrelated and text is loosely associated teaches the model to treat images and text as largely independent streams, because attending closely to their relationship is not consistently rewarded. The field had conflated the format with the training signal content, and this paper disentangles them.

The paper also establishes video-derived keyframes as a distinct category of pretraining data that is neither "images from video" (suggesting lower quality than web images) nor "video frames" (suggesting sequential video understanding tasks), but rather instructional keyframes with domain-specific properties requiring domain-specific extraction methods. The ablation comparing keyframe extraction algorithms (Section 5.4, Table 6) makes this precise: pixel-level extraction (18M frames, 9% performance drop) is too sensitive to noise, semantic-level extraction (1.7M frames, 6.5% drop) misses within-category structural changes, and SSIM-based extraction (6.5M frames, best performance) captures the pedagogically meaningful frame transitions — a new formula appearing, a diagram being annotated, a new step being displayed. This is not an incremental refinement of existing video processing pipelines; it is a demonstration that generic keyframe extraction underperforms domain-matched extraction by a margin large enough to erase most of the textbook's advantage over web data (the 9% drop from pixel-level extraction would reduce the 1-shot average from 31.1% to 22.1%, below OBELICS's 22.8%). Researchers seeking to convert video into pretraining data must now contend with the fact that extraction algorithm choice is not a minor implementation detail but a first-order determinant of downstream model quality.

Finally, the paper provides the first empirical scaling evidence in the multimodal domain that data quality can substitute for data quantity in a way that parallels the Phi-series findings in text-only LLMs (Gunasekar et al., 2023; Li et al., 2023). The 610K textbook samples (~759M text tokens) represent a tiny fraction of the total scale of MMC4 or OBELICS, yet outperform matched quantities of those datasets by large margins on knowledge-intensive benchmarks, and the SFT transfer results (Table 5) show these gains persist through instruction tuning — the textbook-pretrained model achieves +5.5 points on MathVista after SFT, more than double OBELICS's +2.4 and triple MMC4's +1.6. This changes the calculus for VLM pretraining data strategy: rather than defaulting to "collect as much interleaved data as possible from the web," the field should consider a two-source strategy where web-scale paired data handles general vision-language alignment and curated instructional data (at much smaller scale but higher knowledge density) specifically targets knowledge acquisition. The paper does not formalize this as a scaling law (e.g., "1 token of instructional video content ≈ X tokens of web content on knowledge benchmarks"), but the data strongly imply such a relationship exists and is large enough to be practically meaningful.

The practical consequence is that the field's research priorities around pretraining data should shift. Filtering web data better is probably not the highest-leverage investment — the paper shows that even filtered MMC4 (MMC4-Core-ff) often underperforms unfiltered MMC4 in few-shot settings, suggesting the quality ceiling is inherent to the source, not a filtering problem. Instead, the highest-leverage investments are: (1) identifying new sources of intrinsically coherent multimodal data (instructional videos being one, but textbooks, scientific papers with figures, interactive tutorials, and structured documentation being other candidates), (2) developing domain-matched extraction pipelines that preserve the structural properties of those sources, and (3) combining multiple high-quality sources to cover different knowledge domains and coherence types. This is a data sourcing agenda, not a data filtering agenda, and the paper provides both the conceptual motivation and the empirical methodology for pursuing it.

Follow-Up Research This Work Enables

1. Cheap difficulty estimation for compute-optimal allocation. The paper's compute-optimal test-time scaling framework requires estimating question difficulty before allocating inference budget, and the current method (2,048 samples per question scored by the PRM) is far too expensive for deployment. The paper explicitly flags this as future work. A strong follow-up would train a lightweight classifier — possibly a distilled version of the PRM or a small model fine-tuned on the PRM's difficulty labels — that takes only the question text (and optionally a single image, for multimodal benchmarks) as input and predicts the difficulty bin. The evaluation metric would be: does the compute-optimal policy using predicted bins from this lightweight classifier achieve accuracy within, say, 1–2 points of the oracle-bin policy, at a small fraction of the estimation cost? A secondary approach worth investigating is adaptive difficulty estimation: generate a small number of initial samples (4–8), compute the verifier's score distribution on those samples, and use that distribution's statistics (mean, variance, entropy) as a quick difficulty signal to allocate the remaining budget. This amortizes difficulty estimation into the problem-solving process itself.

2. Direct comparison of video-sourced vs. static-educational interleaved corpora. The paper demonstrates that the textbook corpus outperforms webpage datasets, but does not disentangle the knowledge density benefit (instructional content vs. general web content) from the temporal coherence benefit (video-derived sequential frames vs. static webpage images). A crucial follow-up would construct an interleaved corpus from static educational sources — Wikipedia articles with embedded diagrams, Khan Academy text with screenshots, open textbook PDFs with figures — that matches the textbook's subject coverage, knowledge density, and image-text alignment quality, but lacks the temporal coherence of video-derived frames. Training the same VLMs on this "static textbook" vs. the video-derived textbook would isolate the specific contribution of temporal coherence. The prediction from the paper's shuffling experiment is that the static textbook would outperform webpage data (due to higher knowledge density) but underperform the video textbook (due to weaker inter-image coherence), and the magnitude of the gap would quantify the value of temporal structure specifically.

3. Combining the revision model with PRM-guided tree search. The paper studies revisions (Section 6) and PRM search (Section 5) as independent mechanisms and explicitly acknowledges they were never combined (Section 8). A natural follow-up would use the revision model as the proposal distribution within beam search: at each step of the search tree, the model conditions on previously rejected branches as context, potentially generating higher-quality candidate steps. Alternatively, the PRM could guide which revision branches to pursue — rather than blind sequential revision, use the PRM's per-step scores to decide when a revision is on track and should continue vs. when it has gone off course and should restart from an earlier checkpoint. The key evaluation metric would be: does the combined system exceed the individual performance ceilings of revisions alone (~44% at 256 generations, Figure 8) and search alone (~39.5% at 256 generations, Figure 4), particularly on medium-difficulty problems (bins 3–4) where both mechanisms show complementary strengths? A secondary metric: does the combination reduce the performance degradation on bin 1–2 problems that beam search suffers due to over-optimization (Figure 3, right), since the revision model might produce candidates that are more robust against verifier exploitation?

4. Scaling laws for multimodal data quality vs. quantity. The paper implies but does not formalize a relationship where instructional data tokens carry disproportionately more useful signal than web data tokens on knowledge-intensive benchmarks. A systematic follow-up would train VLMs on varying quantities of textbook data (100K, 300K, 610K, 1M samples if available) and matching quantities of OBELICS/MMC4, then fit a scaling curve predicting accuracy as a function of data quantity for each data source. The key question: does the textbook's advantage grow, shrink, or remain constant as data quantity increases? If the advantage is largest at small data scales (consistent with the "data quality substitutes for scale" hypothesis from the Phi-series), the textbook would be particularly valuable for resource-constrained settings. If the gap narrows at larger scales, the textbook would be most useful as a complement to web-scale data rather than a replacement. This experiment would also reveal whether the textbook-trained model plateaus early (suggesting the textbook provides high-quality but limited-coverage knowledge) or continues to improve (suggesting the 610K sample count is just the beginning of what this source could provide).

5. The "stop revising" problem: training revision models with termination awareness. The paper documents that approximately 38% of correct answers get reverted to incorrect ones during sequential revision (Section 6.1), because the model was trained exclusively on incorrect-to-correct trajectories and has no signal for what to do when the current answer is already correct. A principled follow-up would construct training data that includes correct-to-correct transitions (the model should learn to output the same answer or an explicit termination token when the current answer is correct) and evaluate whether this reduces the reversion rate without degrading the model's ability to improve genuinely incorrect answers. The evaluation would measure: (1) the reversion rate at each revision step, (2) the overall chain accuracy with and without within-chain verifier selection, and (3) whether the model learns a checking capability — can it identify why its current answer is correct and refuse to change it, rather than just stochastically preserving it? This experiment would address the Revision Model's most significant practical weakness and test whether the reversion problem is an artifact of the data construction or a deeper limitation of sequential self-correction.

6. Domain transfer: does the textbook's benefit generalize beyond math and science? All experiments use the MATH benchmark and its associated domains (mathematics primarily, with some physics and chemistry in ScienceQA). The instructional video source naturally covers subjects like engineering, computer science, and earth science (all present in the textbook's taxonomy, Table 7), but the paper's evaluation focuses exclusively on math and science benchmarks. A strong follow-up would evaluate the textbook-pretrained VLM on benchmarks from the other domains the textbook covers: code generation and algorithm understanding (HumanEval, MBPP, or a visual coding benchmark), engineering problem-solving, and earth science reasoning. The key question is whether the textbook's advantages are specific to symbolic/mathematical reasoning (where formulas, step-by-step derivations, and visual diagrams dominate) or extend to other knowledge types (procedural knowledge in coding, causal reasoning in earth science). The InSI-SIM metric varies across subjects (the paper does not report per-subject InSI-SIM), so domain-specific analyses of image coherence vs. downstream performance would refine the mechanism: do subjects with higher inter-image coherence produce larger in-context learning benefits?

Practical Applications and Downstream Use Cases

1. Cost-efficient batch inference with difficulty-based routing. For organizations running large-scale VLM inference (e.g., scoring student answers to science questions, generating training data for self-improvement pipelines, or evaluating thousands of multimodal queries), the paper's finding that difficulty-dependent strategy selection yields 4× efficiency gains over best-of-N (Figures 4, 8) translates directly to cost savings. The practical deployment would: (1) estimate each query's difficulty using a lightweight classifier (requiring, per the paper's own future-work call, a model that predicts difficulty from the question text alone — not the 2,048-sample method used in the paper), (2) route easy problems to a small model with minimal test-time compute (e.g., best-of-4 with majority voting), medium problems to a medium model with moderate search (e.g., beam search with M=4, N=64), and hard problems either to the largest available model or to human review. For a deployment with 1M queries per day, reducing the average generation budget from 64 to 16 on easy problems (which might constitute 60% of the query distribution) saves ~29M generations daily. The paper does not provide the difficulty distribution of real-world queries, but the efficiency ratio is established: Figure 4 shows that compute-optimal search at 16 generations matches best-of-N at 64 generations on the MATH test set.

2. Enriching general-purpose VLMs with targeted instructional pretraining. The paper's SFT transfer results (Table 5) demonstrate that textbook pretraining gains persist through standard instruction tuning — the textbook-pretrained LLaVA-1.5 achieves 28.7% on MathVista after SFT, compared to 23.2% for the baseline (no additional interleaved pretraining) and 25.6% for OBELICS-pretrained. This means a VLM provider (e.g., a company training a general-purpose multimodal assistant) can cheaply improve the model's STEM reasoning capabilities by inserting a textbook pretraining phase between standard paired-data alignment and instruction tuning. The cost is modest: 610K samples at typical VLM pretraining throughput is a small fraction of the total pretraining budget for large-scale models. The benefit is concentrated — the +5.5-point MathVista gain does not come at the expense of general VQA performance (OKVQA gains +1.1 points over baseline, Table 5) — meaning the intervention is Pareto-improving: better STEM capabilities at no detectable cost to general capabilities (and possibly a small benefit). This is a rare "free lunch" in model training, enabled by the textbook's complementary knowledge content relative to web-scale pretraining data.

3. Data generation for self-improvement pipelines with domain-specific interleaved data. When using VLMs to generate training data for themselves (analogous to STaR or ReST^EM in the text domain), the quality and diversity of generated solutions depend on the model's pretraining foundation. The paper shows that textbook pretraining dramatically improves the model's ability to solve reasoning-intensive problems (ScienceQA 0-shot: 26.3% for textbook vs. 2.8% for OBELICS). This means a VLM pretrained on the textbook would generate higher-quality solutions when used to create training data for science and math tasks, because it is more likely to produce correct solutions rather than fluent-sounding incorrect ones. In a self-improvement loop where the model generates solutions, filters for correctness (using a verifier or automated checking), and retrains on the correct solutions, the textbook-pretrained model would start from a higher base accuracy, producing more correct solutions per generation budget, which accelerates the self-improvement cycle. The specific multiplier: at 0-shot on ScienceQA, the textbook model is ~9× more likely to produce a correct answer than the OBELICS model (26.3% vs. 2.8%), meaning a self-improvement pipeline would need ~9× fewer generations to collect the same number of correct training examples.

4. On-device VLM deployment for educational applications. The paper demonstrates that a small model (LLaVA-1.5-7B) pretrained on the textbook can answer roughly 1 in 4 science questions correctly zero-shot (ScienceQA: 26.3%) and 1 in 3 with four examples (37.3%), compared to near-zero for the same model pretrained on webpage data. For educational applications where an on-device VLM needs to help students with homework, answer science questions, or provide step-by-step explanations without internet connectivity (privacy, latency, or access constraints), the textbook provides a path to packing more instructional knowledge into a small model without increasing model size. The practical deployment would involve: (1) pretraining a 1–3B parameter VLM (small enough for phones or tablets) on a combination of general paired data and the textbook, (2) instruction-tuning on educational question-answering data, and (3) deploying as an offline tutor app. The key metric from the paper supporting this use case is the 0-shot ScienceQA gap (26.3% vs. 2.8%) — on-device models cannot rely on few-shot examples (which require training set access) or test-time search (which adds latency), so the zero-shot knowledge density provided by the textbook is essential. This is a concrete instantiation of the paper's broader thesis: data quality can substitute for model scale in deployment-constrained settings.

When to Prefer This Method

The paper positions its multimodal textbook as a pretraining data source that complements, rather than replaces, existing datasets. The experimental evidence and design philosophy imply a specific decision framework for when to include the textbook in a VLM pretraining pipeline:

Prefer incorporating the multimodal textbook when:

The target application requires knowledge-intensive, reasoning-heavy multimodal tasks — math problem-solving (MathVista), science question-answering (ScienceQA), multi-step visual inference. The paper's largest performance gains (+20+ points on ScienceQA over webpage baselines, +5.5 points on MathVista after SFT) are concentrated here.
The deployment setting involves few-shot or in-context learning, where the model must attend to multimodal examples provided in the prompt. The cheat test and few-shot scaling results (Tables 2, 4) demonstrate that textbook pretraining specifically enhances this capability, while webpage-trained models largely fail to integrate their interleaved context.
The model is relatively small (7–8B parameters or less) and benefits disproportionately from high-quality, knowledge-dense pretraining data — analogous to the Phi-series findings in text-only LLMs. The paper only tests at this scale, so the claim is most grounded here, though the logic likely generalizes.
Video data in the target domain is available and exhibits the structural properties that make instructional video valuable — causally linked visual and textual content, logically ordered image sequences, high knowledge density in both visual (diagrams, formulas) and auditory (explanations, derivations) channels. The paper's curation pipeline provides a template but requires domain-matched tuning (e.g., SSIM threshold, OCR model choice).
The total pretraining compute budget is fixed and the practitioner must choose between scaling up web data quantity vs. adding a smaller curated instructional corpus. The paper's implicit scaling evidence (610K textbook samples outperforming matched quantities of web data) suggests the curated corpus is the higher-leverage investment within this budget constraint.

Prefer standard web-crawled interleaved data instead when:

The target tasks are general visual recognition or everyday visual question answering (OKVQA, TextVQA in zero-shot) where the textbook provides no advantage or a slight disadvantage (zero-shot OKVQA: 10.2% for textbook vs. 13.0% for OBELICS). Web data's diversity of visual scenes and object categories is essential for broad visual coverage.
The deployment setting is zero-shot only, with no few-shot examples provided at inference. The textbook's largest advantages manifest in few-shot settings where context-attention skills matter; in zero-shot, the model is limited to its parametric knowledge, and web data's broader coverage may be more valuable.
Inference latency is critical, since the textbook's keyframe extraction and multiple filtering stages add substantial upfront curation cost (though the trained model's inference cost is identical regardless of pretraining data source). For rapid prototyping where data curation time matters more than downstream accuracy, simpler web-crawled datasets are faster to acquire.
The application domain has no instructional video content available — for example, satellite imagery analysis, medical imaging, or specialized industrial inspection, where the "instructor explaining visual content" paradigm does not translate. The textbook's value comes from the pedagogical structure of its source; domains without that structure cannot benefit from this approach.
The model is extremely large (70B+ parameters) where the marginal benefit of additional curated data may be smaller because the model already acquires substantial knowledge from its web-scale pretraining. This is untested — the paper only experiments at 7–8B scale — but the Phi-series logic that small models benefit most from curated data suggests caution in extrapolating to much larger scales.