olmOCR 2: Unit Test Rewards for Document OCR

ArXiv: 2510.19817

🎯 Pitch

olmOCR 2 replaces conventional edit-distance rewards with 60,000 binary unit tests—automatically extracted from synthetically generated HTML pages—to train a 7B vision language model for OCR. This RLVR approach yields a +14.2 point jump on olmOCR-Bench, with the largest gains in math, tables, and multi-column layouts, while avoiding the pathological scoring problems of edit distance where two equally correct reading orders can receive wildly different scores. The synthetic pipeline renders real PDFs into ground-truth HTML and programmatically derives verifiable checks for text ordering, table cell coordinates, and LaTeX rendering, making unit-test rewards scalable without manual annotation.

1. Executive Summary

This paper introduces olmOCR 2, a state-of-the-art OCR system that converts digitized print documents into clean plain text by training a 7B vision language model using reinforcement learning with verifiable rewards (RLVR) — where the rewards are a diverse set of binary unit tests that check properties like text presence, reading order, table cell positions, and math formula rendering correctness. To scale unit test creation for RL training, the authors develop a synthetic document pipeline that samples real PDFs, generates corresponding HTML renderings with known ground truth, and programmatically extracts test cases from that HTML. On olmOCR-Bench, the resulting model achieves an overall score of 82.4±1.1, a +14.2 point improvement over the original February 2025 release, with the largest gains concentrated in math formula conversion, table parsing, and multi-column layouts — establishing that binary unit tests serve as an effective and scalable reward signal for OCR-specialized VLM training, particularly when the unit tests can be generated automatically from synthetic HTML documents rather than requiring manual verification.

2. Context and Motivation

The Core Problem: OCR Evaluation Metrics Don't Align With Practical Correctness

The fundamental problem this paper addresses is a mismatch between how OCR systems are commonly evaluated and what actually constitutes "correct" output in practice. The dominant evaluation paradigm for document OCR relies on edit distance — measuring the character-level or token-level differences between a model's output and a human-created ground truth transcript. This approach has been standard across benchmarks like OmniDocBench (Ouyang et al., 2024) and many others. The paper argues that edit distance, while convenient and continuous, suffers from two deep structural flaws that make it unsuitable for both evaluating and training OCR systems.

Problem 1: Edit distance penalizes equivalent-but-different outputs. Documents contain floating elements — tables, figures, captions, sidebars — whose linearization order in plain text is inherently ambiguous. A caption might reasonably appear before or after the paragraph it describes; a figure might interleave with text in multiple equally valid ways. The paper illustrates this with a concrete example in Figure 1: a document with green, yellow, and red text passages plus a floating caption. If the caption appears before the green passage, that is one valid ordering. If it appears after the yellow passage, that is another equally valid ordering. A well-designed unit test that checks "green text appears before yellow text, uninterrupted by red text" would correctly flag both orderings as correct. Edit distance, however, assigns wildly different scores to these two valid outputs — heavily penalizing one despite no actual error. Worse, edit distance sometimes partially rewards outputs that are genuinely wrong, such as when a caption appears in the middle of a paragraph (interrupting the reading order), because the character overlap is still high.

This is not a corner case. Scientific papers, legal documents, financial reports, and government records routinely mix multi-column layouts with floating tables, figures, captions, and footnotes. Any OCR system evaluated purely by edit distance is being measured on a metric that conflates "structurally similar to the ground truth" with "practically correct." The paper's objection is that these are not the same thing, and optimizing for edit distance can push models toward a specific linearization that may not match what a human reader would consider the natural reading order.

Problem 2: Edit distance doesn't measure semantic correctness for structured elements. The paper's second criticism is that edit distance treats all character differences equally, but some differences matter vastly more than others in OCR output. A LaTeX math formula like \frac{1}{2} and 1/2 have entirely different character sequences but render identically — edit distance sees a large difference where none exists semantically. Conversely, two LaTeX strings that differ by a single misplaced brace (like x^{2} vs. x^{2}) have small edit distance but one renders correctly and the other doesn't. Figure 2 demonstrates this concretely: Model A produces a LaTeX output more dissimilar to the reference, but its rendered formula matches perfectly, passing the unit test. Model B is closer to the reference in edit distance but fails rendering — the edit distance metric would incorrectly prefer Model B.

The same problem extends to tables, where a single misplaced cell value matters enormously for correctness but constitutes a tiny fraction of the total characters in a document. Edit distance would barely notice, but a user reading the extracted table would see an unambiguous error. The paper notes that while recent work has attempted to improve edit distance specifically for math formulas (Wang et al., 2025b, via Character Detection Matching), extending such calibrated continuous scores to tables, reading order, headers/footers, and other OCR targets remains unsolved.

The unit test solution — and its scaling problem. In their prior olmOCR-Bench work (Poznanski et al., 2025), the authors proposed an alternative: evaluate OCR output using a battery of binary unit tests that check specific properties — whether certain phrases appear, whether others are absent, whether sentences follow correct reading order, whether table cells maintain relative positions, whether math formulas render identically when passed through KaTeX. This approach elegantly solves both problems: it treats equivalent linearizations equally (any ordering that passes the reading order test gets full credit) and it measures semantic correctness (a formula either renders correctly or doesn't). The unit tests provide heterogeneous, interpretable signals about different failure modes — you can see whether a model improved on tables, math, or reading order independently.

But there was a critical scaling bottleneck: the original olmOCR-Bench unit tests were all manually verified. The abstract describes them as "checking for specific details of a PDF like formulas, tables, and text order" and notes they "took hours of work to create and check by hand." This meant that while unit tests were excellent as an evaluation framework, they could not practically be used as a training signal. For reinforcement learning, you need thousands of test cases generated across thousands of documents — far beyond what manual effort can produce.

The innovation in olmOCR 2 is closing this gap: if you can generate synthetic documents with known ground-truth HTML, you can programmatically extract unit tests from that HTML, making the unit test framework not just an evaluation tool but a scalable reward signal for RL training. This shifts the problem from "how do you write unit tests?" to "how do you generate high-quality synthetic documents with accurate HTML ground truth?" — a more tractable engineering challenge.

Why This Problem Matters: The Stakes of Document OCR at Scale

The paper exists in a context where document OCR has become a critical infrastructure component for large-scale language model training and deployment. The original olmOCR paper (Poznanski et al., 2025) was titled "Unlocking Trillions of Tokens in PDFs" — a nod to the fact that an enormous fraction of human knowledge exists in PDFs that are inaccessible to text-only language models. Scientific papers, legal documents, government records, technical manuals, financial filings, and historical archives are overwhelmingly distributed as PDFs containing complex layouts, tables, equations, and figures. Converting these into plain text that preserves semantic structure (correct reading order, intact tables, renderable equations) is a prerequisite for using them in training data pipelines or retrieval systems.

The quality of OCR directly limits the quality of downstream applications. A language model trained on OCR output where paragraphs are incorrectly ordered will learn corrupted language patterns. A retrieval system indexing OCR'd documents with scrambled table cells will return wrong information for factual queries. An assistant that reads a user-uploaded PDF and extracts wrong numbers from a financial table could cause real harm. The paper's focus on math formulas, tables, and multi-column layouts — the areas where olmOCR 2 shows its largest improvements — targets precisely the elements that are both most information-dense and most error-prone in OCR.

Beyond training data, there is a growing ecosystem of applications that process PDFs at scale: academic search engines, legal discovery tools, medical record digitization, government document declassification, library digitization projects. Each of these has different tolerance for different types of errors. A system that confuses reading order might produce plausible-looking but factually incorrect text — a particularly dangerous failure mode because it's hard to detect without comparing against the original PDF. Unit tests provide a way to measure and optimize for specific error types that matter for specific applications, rather than collapsing everything into a single edit distance number.

The paper also exists at a specific moment in the rapid evolution of OCR technology. The timeline from Table 1 tells the story: between February 2025 (olmOCR's initial release at 68.2) and October 2025 (olmOCR 2 at 82.4), the field moved from ~70 being competitive to ~80+ being necessary for state-of-the-art. This is a fast-moving, highly competitive landscape with multiple open-source projects (Marker, MinerU, PaddleOCR, dots.OCR, Chandra OCR), frontier model APIs (GPT-4o, Gemini, Mistral), and specialized OCR VLMs (Nanonets, MonkeyOCR, Infinity Parser, DeepSeek-OCR) all vying for leadership. The paper's contribution of a scalable RL training recipe using unit test rewards is not just an academic exercise — it's a practical method for staying at the frontier in a field where the bar rises every few months.

Prior Approaches and Their Limitations

The paper situates itself across three overlapping research traditions, each with distinct limitations that olmOCR 2 addresses.

Tradition 1: Pipeline-based OCR systems. Systems like MinerU (Wang et al., 2024a), Marker (Paruchuri, 2025a), and PP-OCRv5 (Cui et al., 2025b) compose multiple specialized models — one for layout segmentation, one for table detection, one for text recognition, one for reading order prediction — into a processing pipeline. These systems draw on decades of document analysis research and can achieve high accuracy on specific sub-tasks by training dedicated models for each. Their limitation is the compounding of errors: a mistake in the layout segmentation stage cascades into the text extraction stage, which cascades into the reading order stage, and there is no mechanism for the downstream stages to correct upstream errors. The modularity that makes them interpretable and debuggable also makes them brittle. Additionally, maintaining and updating multiple models is engineering-heavy, and each model is typically trained with its own dataset and objective function — there is no unified training signal that optimizes end-to-end correctness.

Tradition 2: End-to-end VLM-based OCR. The paper identifies a "paradigm shift" away from pipelines toward using vision language models that take a document image as input and produce structured text output in a single forward pass. Early examples include Nougat (Blecher et al., 2023) for academic documents and GOT-OCR 2.0 (Wei et al., 2024) for general documents. The release of GPT-4o (OpenAI et al., 2024) demonstrated that general-purpose frontier VLMs could achieve strong OCR performance without task-specific training, and the original olmOCR (Poznanski et al., 2025) showed that this capability could be distilled into a small 7B VLM through supervised fine-tuning on GPT-4o outputs. The subsequent explosion of OCR-specialized VLMs — Nanonets-OCR2-3B, MinerU 2.5, dots.OCR, Monkey OCR, Chandra OCR, DeepSeek-OCR, PaddleOCR-VL, Infinity Parser — represents the new normal: end-to-end models trained on large supervised datasets.

The limitation of this approach, as the paper sees it, is that supervised fine-tuning alone cannot easily optimize for the specific correctness properties that matter in OCR output. A model trained to imitate GPT-4o outputs (as in olmOCR's original recipe) will inherit GPT-4o's error patterns. A model trained on human-created ground truth transcripts will be biased toward matching the specific linearization choices made by the human annotators. In both cases, the training signal is a form of token-level cross-entropy loss that doesn't distinguish between a minor formatting variation and a catastrophic reading order failure — exactly the same problem that edit distance has at evaluation time, now baked into the training objective.

Tradition 3: RL with verifiable rewards for VLMs. The most recent development — and the one olmOCR 2 directly builds on — is using reinforcement learning with verifiable rewards (RLVR) to fine-tune VLMs beyond what supervised training achieves. The paper cites several concurrent works: DianJin-OCR-R1 (Chen et al., 2025) uses RL rewards to improve OCR via chain-of-thought reasoning; He et al. (2025) and Xiong et al. (2025) show that RL rewards improve visual document answering; and most directly, Infinity Parser (Wang et al., 2025a) develops a synthetic data pipeline around HTML renderings and trains an OCR-specialized VLM using GRPO with verifiable rewards.

The key distinction the paper draws versus Infinity Parser is in the choice of reward signal. Infinity Parser defines its reward based on edit distance, paragraph count, and structural consistency — essentially, a composite continuous score. olmOCR 2 instead uses binary unit tests. This is a fundamental design choice with consequences: binary unit tests don't partially reward partially correct outputs; they pass or fail based on specific, interpretable criteria. A model receives reward for getting the math formula rendering correct, regardless of whether its LaTeX matches the reference character-for-character. It receives no reward for a table that has all but one cell correct — unlike edit distance, which would give nearly full credit. The paper argues that this binary, criterion-referenced reward structure is better aligned with what users actually care about (did you get the table right? yes or no) than continuous scores that conflate importance-weighted and non-importance-weighted errors.

The Synthetic Data Pipeline as the Enabling Innovation

The paper's key insight — and what makes the unit test approach practical for RL training rather than just evaluation — is that if you can generate HTML that accurately represents a document page, you can programmatically extract unit tests from the HTML's semantics. The HTML structure encodes exactly the information needed: <header> and <footer> tags identify text that should be absent from body extraction (Text Absence tests); KaTeX-rendered equations provide both the LaTeX source and the rendered appearance (Math Formula Accuracy tests); <table> structures with known cell values enable Table Accuracy tests; and the document order of elements in the HTML source provides ground truth for reading order tests.

The synthetic document pipeline therefore becomes the critical subsystem. It works by sampling real PDF pages (providing layout diversity that template-based approaches lack), prompting a general VLM (Claude Sonnet) to produce HTML that renders to a visually similar page, and then using that HTML — not the original PDF — as ground truth for unit test generation. Crucially, any OCR errors Claude makes during HTML generation do not affect the unit tests: the tests are extracted from the HTML itself, not from a comparison between the HTML and the original PDF. The tests check whether olmOCR 2's output matches the HTML's structure, not whether that structure perfectly represents the source document. This makes the pipeline robust to VLM hallucination in a way that a pipeline requiring perfect ground truth would not be.

The cost is approximately $0.12 per document page for the Claude API calls, and the final training mix (olmOCR2-synthmix-1025) contains 2,186 PDF pages yielding 30,381 test cases — a scale that would be completely infeasible for manual creation. At this scale, the unit tests become dense enough to serve as a meaningful RL reward signal: each training document has enough tests that the fraction of passing tests (0.0 to 1.0) provides a reasonably smooth gradient for policy optimization.

How This Paper Positions Itself

The paper positions olmOCR 2 as addressing the intersection of two gaps:

The evaluation-training gap: Unit tests are a better evaluation framework than edit distance (established by olmOCR-Bench), but they couldn't previously be scaled for training. The synthetic HTML pipeline closes this gap, making the same binary test framework that evaluates the model also the reward signal that trains it.
The RL reward design gap: RLVR has been demonstrated for OCR (Infinity Parser), but the design of the reward function matters enormously. Binary unit tests provide a principled alternative to continuous edit distance scores — one that is interpretable, decomposable by error type, and aligned with practical correctness criteria.

The paper is explicit about its relationship to Infinity Parser: "A slight difference in our works is our use of sampled real content to seed generation of full HTML pages while their work injected sampled real content into pre-made HTML layouts. A more significant difference is that we use binary unit tests as our verifiable reward signal while they define their reward based on edit distance, paragraph count, and structural consistency." This framing highlights the paper's claim to novelty — not in using RLVR for OCR (which others have done), nor in using synthetic HTML for data generation (which others have done), but in the specific combination of programmatically generated binary unit tests as the reward signal for RL training of an OCR-specialized VLM.

The paper also positions itself within the broader open-source OCR movement. Table 1 is organized not just by benchmark score but by openness criteria: model weights, training data, training code, and inference code availability, plus license type. olmOCR 2 achieves "state-of-the-art performance while maintaining fully open data, model, and code" — a claim that distinguishes it from closed-source competitors like GPT-4o, Gemini, and Mistral OCR API, as well as from open-weight models that don't release training data. The authors present their work as advancing not just the technical state of OCR but the open ecosystem that enables reproducibility and further research.

3. Technical Approach

3.1 Reader Orientation

The paper presents a training system that takes a general-purpose 7B vision-language model already fine-tuned for document OCR and further improves it using reinforcement learning where the reward signal comes from programmatically generated, pass/fail checks on the model's output. It solves the problem of how to provide a scalable, interpretable, and semantically meaningful training signal for OCR — one that can distinguish between harmless formatting variations and genuine errors in reading order, table structure, and math formula rendering — by constructing a pipeline that synthesizes document pages with known HTML ground truth, extracts binary unit tests from that HTML, and then uses the fraction of passing tests as the reward for policy optimization.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a sequential pipeline:

PDF Sourcing Module — samples real PDF pages from diverse, challenging document sources (e.g., math-heavy arXiv papers) to serve as layout/style seeds for synthetic document generation.
VLM-Based HTML Generator — takes a rasterized image of a sampled PDF page and prompts a general VLM (Claude Sonnet) to produce HTML code that renders to a visually similar page, with semantically meaningful HTML tags (<header>, <footer>, <table>, KaTeX math).
Unit Test Extractor — programmatically scans the generated HTML to extract binary test cases of various types (text presence/absence, reading order, table cell positions, math formula rendering) without requiring manual verification.
Base OCR Model (Qwen2.5-VL-7B-Instruct + SFT) — the supervised-fine-tuned VLM that takes a document page image as input and outputs structured text (YAML format with Markdown content), serving as the policy to be optimized.
GRPO Training Loop — generates multiple completions per document from the base model, scores each using the unit test pass rate (plus format rewards), and updates the model using Group Relative Policy Optimization with KL regularization, producing the final olmOCR-2-7B-1025 model through weight averaging of multiple training runs.

Information flows as follows: real PDF pages are sampled → each page image is sent to Claude Sonnet with detailed layout analysis and HTML generation prompts → the resulting HTML is rendered and verified to match the original page appearance → unit tests are extracted from the HTML semantics → these tests (not the original PDF content) serve as ground truth for scoring → the OCR model generates completions for each document page image → each completion is scored by running the unit tests against it → the pass rate (0.0 to 1.0) is the reward → GRPO updates the model to maximize expected reward under a KL penalty.

3.3 Roadmap for the Deep Dive

First, the synthetic HTML pipeline (Section 3.1 of the paper): PDF sourcing, VLM-based HTML generation with iterative refinement, and how the HTML semantics enable programmatic test extraction. This is the foundational component — everything downstream depends on the quality of these synthetic documents.
Second, the unit test types and their extraction: how each category of test (Text Presence, Text Absence, Reading Order, Table Accuracy, Math Formula Accuracy, Baseline Robustness) is derived from HTML structure, and why this approach solves the "equivalent representations" problem that plagues edit distance.
Third, the base model and supervised fine-tuning: what model serves as the starting point (Qwen2.5-VL-7B-Instruct), what SFT data is used (olmOCR-mix-1025), and the output format (YAML) — establishing the policy that RL will optimize.
Fourth, the RLVR training setup: GRPO mechanics, how the reward is constructed (unit test pass rate plus format rewards), the KL penalty, generation budget per document, and the souping procedure that produces the final model.
Fifth, the incremental improvements that compound with the RL contribution: dynamic temperature scaling, bug fixes, image resizing, YAML output format, blank page handling — each represents a design choice that matters for the final system's performance.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core technical contribution is the pipeline that makes binary unit tests scalable as an RL reward signal for OCR training, plus the demonstration that GRPO with this reward produces state-of-the-art results.

3.4.1 Synthetic HTML Generation Pipeline

The synthetic data pipeline is the enabling infrastructure for the entire RLVR approach. Its purpose is to take real-world document pages — with all their layout complexity and diversity — and produce corresponding HTML representations where the semantics of the document (headers, footers, tables, equations, reading order) are explicitly encoded in the HTML tag structure, making it possible to programmatically extract unit tests without any manual annotation.

PDF Sourcing. The pipeline begins by sampling pages from real PDF documents that are selected to be challenging for OCR in specific ways. The paper states they "source from arXiv math-heavy papers" when they want to focus on unit tests for math equations, and presumably sample from other domains (multi-column layouts, documents with headers/footers, documents with complex tables) to cover the full range of test types. The key design decision here is using real-world documents rather than template-generated ones: "By sampling real-world documents, we create a high diversity of documents, instead of being restricted to just a handful of pre-made templates." Template-based approaches (like those used in Infinity Parser's "pre-made HTML layouts") can only generate as many distinct layouts as there are templates, whereas real PDFs provide essentially unbounded layout diversity — different column counts, margin sizes, figure placements, table styles, equation densities, and typographic conventions. This diversity is critical for RL training because the model must learn to handle layout variation as part of the OCR task, not just recognize text in a few fixed arrangements.

VLM-Based HTML Generation (Three Steps). For each sampled PDF page, the pipeline converts it to a rasterized image and sends it to a general VLM — specifically claude-sonnet-4-20250514 — in an iterative, three-step process:

Step 1: Layout Analysis. The VLM receives the page image and is prompted to analyze its layout structure. The exact prompt is not reproduced in the paper, but the authors reference a specific code location: github.com/allenai/olmocr/olmocr/bench/synth/mine_html_templates.py#L398-L420. The analysis covers "the general layout of the page, such as number of columns, presence of images or tables, headers and footers, and so on." This step is not the final HTML generation — it produces an intermediate structural description that guides the next step. The purpose is to improve coverage of unit test elements: by explicitly asking the VLM to identify headers, footers, tables, and equations in the layout, the system ensures that the subsequent HTML generation step encodes these elements with the appropriate semantic tags that enable unit test extraction.

Step 2: Initial HTML Generation. Using the layout analysis as guidance, the VLM generates HTML code that, when rendered in a browser, should produce a page visually similar to the original PDF page. The HTML is required to use specific semantic conventions that make unit test extraction possible: headers and footers must be placed in <header> and <footer> tags (enabling Text Absence tests), tables must use standard <table> structures with <tr> and <td> elements (enabling Table Accuracy tests), and math equations must be formatted using KaTeX with \( and \[ delimiters for inline and block math respectively (enabling Math Formula Accuracy tests). The body text content is placed in standard block elements like <p>, <div>, and <section> tags that encode the natural reading order in the document order of the HTML source.

Step 3: Refinement. The paper states that the VLM is prompted iteratively to "first create, and then refine, the HTML code." This implies at least a two-pass process where the initial HTML is fed back to the VLM along with the original page image for correction and improvement. The refinement step is important because the VLM may make OCR errors when reading text from the page image, and it may produce HTML that doesn't quite match the layout. The refinement pass gives the VLM a chance to correct discrepancies.

Why Claude Sonnet and Not a Smaller Model? The authors note that Claude Sonnet was "sufficiently accurate and cost effective, costing approximately $0.12 per document page." This is a one-time cost during data preparation, not an inference cost during model training or deployment. The choice of a frontier VLM reflects the fact that HTML generation from document images is itself a hard vision-language task — the generator must accurately read text, understand layout hierarchy, and produce syntactically valid HTML with semantically appropriate tags. Using a weaker model would introduce errors in the synthetic ground truth, which would then propagate into the unit tests and ultimately into the reward signal for RL training. The$ 0.12/page cost buys data quality.

Robustness to VLM Hallucinations. A subtle but critical property of this pipeline: "our pipeline is robust to hallucinations: even in cases where Claude makes an error when it is performing OCR, that does not affect our pipeline, as we use the HTML output alone to generate unit tests." This means the pipeline does not require Claude's HTML to be a perfect transcription of the original PDF. The unit tests are extracted from the HTML's own structure and content, and they test whether olmOCR 2 can produce output that matches the HTML's version of the document. If Claude misreads a word, the unit test will check for the misread word, and the model will be rewarded for reproducing that word — training it to output what the HTML says, not what the original PDF says. This decouples the synthetic ground truth quality from the original document fidelity, which makes the pipeline practical: you don't need perfect OCR from the VLM to generate useful training data.

However, this robustness has a cost: if Claude systematically hallucinates in some way (e.g., always placing captions in a particular position), the model will learn that pattern. The diversity of real-world source PDFs and the refinement step partially mitigate this by reducing the rate of hallucinations, but the fundamental property remains — the training signal teaches the model to match the synthetic HTML, not the source PDF. For the RL stage, this is an acceptable trade-off because the goal is to improve specific OCR capabilities (math, tables, reading order) rather than to achieve perfectly faithful transcription.

Scale of the Synthetic Data. The final synthetic data mix, olmOCR2-synthmix-1025, contains 2,186 PDF pages. From these pages, the unit test extractor produces 30,381 test cases — an average of approximately 14 tests per page. This density is important for the reward signal: with 14 binary tests per page, the pass rate can take on 15 discrete values (0/14, 1/14, ..., 14/14), providing enough granularity to differentiate partially correct from fully correct outputs while still being fundamentally a sparse, criterion-referenced signal rather than a dense continuous score.

3.4.2 Unit Test Types and Programmatic Extraction

The unit tests that form the reward signal are of six types, each extracted programmatically from the synthetic HTML's semantic structure. This section explains what each test type measures, how it is extracted from HTML, and what design choices make it more aligned with practical correctness than edit distance.

Text Presence. This test checks that a specific phrase appears verbatim in the model's output. The extraction procedure is straightforward: from the body text content of the HTML (paragraphs, headings, captions), random or strategically selected text spans are chosen as targets. The test passes if the exact string appears anywhere in the OCR output, and fails otherwise. This tests the most basic capability: can the model correctly read and transcribe text content? The binary nature means that a single character error in an otherwise perfectly transcribed sentence results in a test failure — which may seem harsh but accurately reflects the user experience: if you're searching for a specific quote or fact in the OCR'd document, a one-character error prevents you from finding it. The test is case-sensitive and requires exact match (including whitespace), as implied by the "binary" designation.

Text Absence. This test checks that certain phrases — specifically headers, footers, and page numbers — do not appear in the model's output. The extraction procedure leverages the <header> and <footer> HTML tags: any text content inside these tags in the synthetic HTML generates a Text Absence test. This is because olmOCR (like most document OCR systems) is expected to strip running headers and footers from the extracted body text — these are repetitive boilerplate that don't contain document content and would interfere with downstream processing (e.g., a phrase like "Chapter 3: Methods" at the top of every page would appear dozens of times in a document's OCR output, corrupting any analysis that assumes unique text). The test passes if the target header/footer text is absent from the output. This is an example of a test that would be very difficult to encode in an edit distance metric: the header text is genuinely present in the document image, so a transcription that includes it would have lower edit distance to a ground truth that includes headers, but it would be less useful for downstream applications.

Natural Reading Order. This test checks whether sentences or text spans appear in the correct sequential order in the model's linearized output. The extraction procedure leverages the document order of text elements in the HTML source: the HTML is assumed to encode the correct reading order through the sequence of its elements, so the test can check that, for example, a sentence from the first column appears before a sentence from the second column in a two-column layout, or that a caption appears either before or after its associated figure but not in the middle of a separate paragraph. The paper's Figure 1 provides the motivating example: a unit test checks that "green text appears before yellow text, uninterrupted by red text." The test does not require a specific absolute position — it only requires the relative ordering constraint. This is the key advantage over edit distance: two different linearizations that both satisfy the ordering constraint will both pass the unit test, whereas edit distance would assign one a higher score than the other based on character-level alignment with the ground truth.

The paper does not provide the exact algorithm for extracting reading order tests from HTML, but the principle is clear from the example: adjacent or nearby text elements in the HTML source that should be read in sequence are selected as test targets, and the test checks that they appear in the model's output in the same relative order (possibly allowing for intervening text that doesn't violate the ordering constraint).

Table Accuracy. This test checks the relative positioning of specific cells in a table. The extraction procedure leverages the HTML <table> structure: for each table in the synthetic HTML, the system identifies specific cell values and their row/column positions, then generates tests that check whether those values appear in the model's output in the correct table structure (i.e., in the correct row relative to other sampled cells). The paper describes this as checking "the relative position of cells (with specific values) in a table." For example, a test might check that the value "42.3" appears in the same row as "Experiment A" and to the right of the column header "Accuracy." The test passes if the model's table representation preserves this spatial relationship, regardless of how it formats the table (Markdown, HTML, or plain text with alignment) and regardless of minor formatting variations in the cell values.

This is substantially more robust than edit distance for tables: a table with N rows and M columns has NM cell values, and moving a single cell to the wrong position changes a tiny fraction of the total characters but constitutes a complete failure of that cell's information. Edit distance would barely penalize this error; the binary unit test unambiguously catches it.

Math Formula Accuracy. This test checks that a mathematical formula, when rendered by a LaTeX renderer (specifically KaTeX), produces the same visual output as the reference formula. The extraction procedure leverages the KaTeX markup in the synthetic HTML: the HTML contains math expressions in LaTeX format (delimited by \( for inline and \[ for block display), and the pipeline extracts these LaTeX strings as reference formulas. The test operates by taking the model's output for the formula, passing both the reference LaTeX and the model's LaTeX through KaTeX, and comparing the rendered output — not the LaTeX source code.

This is the most sophisticated test type and the one that most clearly demonstrates the advantage of unit tests over edit distance (Figure 2). Two LaTeX strings that are character-level different can produce identical rendered output (e.g., \frac{1}{2} and \frac{ 1 }{ 2 } with different spacing, or x^{2} and x^2 with equivalent exponent notation). Conversely, two strings that differ by a single character can produce radically different renderings (e.g., x^{2} vs. x^{2 with a missing closing brace). The KaTeX rendering comparison captures semantic equivalence while ignoring irrelevant formatting differences.

The paper references Wang et al. (2025b)'s Character Detection Matching (CDM) as prior work that improved math formula evaluation beyond simple edit distance, but notes that their approach only addresses math formulas — the unit test framework extends this semantic comparison approach to all types of OCR content.

Baseline Robustness. This test checks for two specific failure modes: (1) that long repeated n-grams — a sign of the model getting stuck in a repetition loop — do not appear in the output, and (2) that characters from non-target languages (i.e., not the document's declared language) do not appear. These are guardrail tests that catch catastrophic failures rather than measuring subtle quality differences. The test fails if the output contains, for example, the same 10-word phrase repeated 5 times, or if an English document's output contains Chinese characters (a sign of cross-lingual hallucination). Extraction from HTML is straightforward: the pipeline knows the document's language from metadata and can define repeated n-gram thresholds.

Binary Nature and Its Implications. All six test types produce binary pass/fail outcomes per test case. For RL training, these are aggregated at the page level:

$\text{reward}_{\text{page}} = \frac{\text{number of passing tests}}{\text{total number of tests for this page}}$

where the reward is a scalar in $[0.0, 1.0]$ representing the fraction of unit tests passed.

What it computes: For a given page, the system takes the model's complete OCR output (a string in YAML/Markdown format), runs each unit test extracted from that page's synthetic HTML against the output, and computes the ratio of passes to total tests. This single scalar is the primary reward for GRPO training.

Why this form (fractional rather than all-or-nothing page-level): A page-level binary reward (1.0 if all tests pass, 0.0 otherwise) would be too sparse — most outputs would score 0.0 during early training, providing no gradient for which partial improvements are better than others. The fractional reward provides a coarse but non-zero gradient: an output that gets the tables right but fails math still gets partial credit, allowing the policy to climb toward better performance incrementally. However, unlike edit distance (which is continuous and dense), the fractional reward still treats each test as binary — the model gets 1/6 for passing 1 of 6 tests regardless of whether the 5 failures were near-misses or complete disasters. This preserves the criterion-referenced philosophy: you either get the table right or you don't, and the reward reflects that.

3.4.3 Base Model and Supervised Fine-Tuning

The RLVR training does not start from a random or pretrained model. It starts from a model that has already been specialized for OCR through supervised fine-tuning (SFT), establishing a strong baseline that RL can further refine.

Starting Point: Qwen2.5-VL-7B-Instruct. The base architecture is Qwen2.5-VL-7B-Instruct (Bai et al., 2025), a 7-billion-parameter vision-language model from the Qwen family. This is an instruction-tuned VLM, meaning it has already been trained to follow text instructions and process images — it can take a document page image as input and produce text output describing or transcoding that image. The switch from Qwen 2 VL (used in the original olmOCR) to Qwen 2.5 VL is described in Table 3 as providing "a slight improvement in benchmark score." The 7B parameter scale is a deliberate choice for practical deployment: large enough to handle the complexity of document understanding, small enough to run on a single GPU node for inference and fine-tuning.

SFT Data: olmOCR-mix-1025. Before RL training, the model undergoes one epoch of supervised fine-tuning on olmOCR-mix-1025, a dataset containing 267,962 pages from over 100,000 PDFs sampled from diverse sources. This dataset is described as a "refreshed mix" compared to the original olmOCR-mix-0225, with several improvements:

Better base model for data generation: The supervision targets were generated using GPT-4.1 instead of GPT-4o. The authors don't detail the differences between these GPT versions, but the implication is that GPT-4.1 produces higher-quality OCR transcripts for training.
Consistent equation formatting: Math equations use \[ and \( delimiters for block and inline math. This standardization matters because inconsistent LaTeX delimiters in training data force the model to learn multiple formats for the same concept, wasting capacity on format variation rather than content accuracy.
HTML table format: Tables in the SFT targets use HTML markup rather than Markdown or plain text. This provides richer structural information — rowspan, colspan, alignment — that helps the model learn table structure.
Basic alt text for images: The targets include text descriptions of images and figures. This teaches the model to handle document elements beyond pure text (a capability that the unit test framework doesn't directly test but that matters for general-purpose OCR utility).

The SFT training follows the procedure described in the original olmOCR paper (Poznanski et al., 2025), which used standard next-token prediction (causal language modeling loss) on the model's output tokens conditioned on the input image and instruction prompt.

Table 2 Comparison. The paper provides an ablation comparing SFT on the old mix versus the new mix, evaluated on olmOCR-Bench:

Sub-benchmark	olmOCR-mix-0225	olmOCR-mix-1025
ArXiv	78.6	70.8
Old scans math	79.9	79.3
Tables	72.9	77.9
Old scans	43.9	45.6
Headers & footers	95.1	93.7
Multi column	77.3	81.3
Long tiny text	81.2	78.7
Base	98.9	99.3
Overall	78.5 ± 1.1	78.3 ± 1.2

The overall scores are nearly identical (78.5 vs. 78.3), but the sub-benchmark breakdown reveals tradeoffs: the new mix substantially improves Table performance (+5.0 points) and Multi-column performance (+4.0 points) while degrading on ArXiv (-7.8 points) and Long tiny text (-2.5 points). The ArXiv degradation is notable — it may reflect distribution shift in the GPT-4.1 training data or the HTML table format being less suitable for the math-heavy, single-column ArXiv documents. Regardless, the new mix is used as the SFT starting point for RL training.

Output Format: YAML with Markdown Content. The SFT model is trained to output documents in a specific structured format. The original olmOCR used JSON, but olmOCR 2 switches to YAML. The paper explains the rationale: "We speculate this is because the model does not need to remember how many open quotes there are currently in the JSON and can simply output an EOS token as soon as it is done. With JSON, we also found more incidences of repetition loops." This is a practical insight about token generation: JSON requires matching quotation marks, brackets, and braces across potentially long sequences, and the model's autoregressive generation can lose track of nesting depth (requiring it to "remember" the state of open delimiters). YAML's indentation-based structure is more forgiving — a newline followed by reduced indentation signals the end of the current block, and the model can terminate generation naturally without tracking delimiter state. The paper reports "no benchmark score difference" from the format switch, but "with fewer need for retries, this improved our inference efficiency" — a systems-level improvement that matters for production deployment even though it doesn't affect accuracy.

The output structure includes:

Document metadata at the top (primary language, rotation correction factor)
Body content in Markdown format with standardized equation delimiters
Tables in HTML format within the Markdown
Image alt text where applicable

3.4.4 GRPO Training with Unit Test Rewards

This is the core technical contribution: the RLVR training setup that takes the SFT model and optimizes it against the unit test reward signal.

Algorithm: Group Relative Policy Optimization (GRPO). The paper uses GRPO (Shao et al., 2024) as the RL algorithm, implemented via the Hugging Face TRL library (von Werra et al., 2020). GRPO is a variant of policy gradient methods designed for language model fine-tuning. Like PPO (Proximal Policy Optimization), it uses a clipped surrogate objective to constrain policy updates, but unlike PPO it doesn't require a separate value function (critic) network. Instead, for each prompt (document page), GRPO:

Samples $G$ completions from the current policy (where $G = 28$ in this paper's setup — "28 completions are generated" per document).
Computes the reward for each completion using the unit test pass rate.
Computes advantages by comparing each completion's reward to the group mean: for completion $i$ in a group of $G$ completions for the same prompt, the advantage is (roughly) the reward of completion $i$ minus the mean reward of all $G$ completions. This "group relative" normalization removes prompt-dependent baseline reward levels — a completion that scores 0.5 on a very hard document might get positive advantage if the group mean is 0.3, while a completion that scores 0.5 on an easy document might get negative advantage if the group mean is 0.7.
Updates the policy to increase the probability of completions with positive advantage and decrease the probability of those with negative advantage, subject to a KL divergence penalty.

The KL penalty parameter is $\beta = 0.01$ . This constrains how far the optimized policy can diverge from the reference policy (the SFT starting point), preventing catastrophic forgetting of the OCR capabilities learned during SFT while still allowing the model to adapt to the unit test reward.

Why GRPO over alternatives? The paper doesn't provide an ablation comparing RL algorithms, but GRPO has become popular for language model RL fine-tuning because: (1) it eliminates the need for a separately trained critic (value) network, reducing memory and computational cost; (2) the group-relative advantage normalization handles varying prompt difficulty without requiring a learned baseline; (3) the clipped objective provides stable training without the instability that can plague unconstrained policy gradients. The choice of GRPO is consistent with other recent OCR RL works, particularly Infinity Parser (Wang et al., 2025a).

Reward Construction. For each completion, the total reward is the sum of three components:

Unit Test Pass Rate (primary signal): The fraction of unit tests for that page that the completion passes, as described in Section 3.4.2. This is a scalar in $[0.0, 1.0]$ .
EOS Token Reward (binary): A reward of 1.0 if the completion ends with the end-of-sequence token (meaning the model terminated generation naturally), and 0.0 otherwise. This penalizes completions that hit the maximum generation length or that get stuck in repetition loops without producing an EOS token. The paper notes this as "ensuring correct output format."
Metadata Reward (scalar in $[0.0, 1.0]$ ): A reward for outputting document metadata at the top of the response — specifically, the primary language and rotation correction factor. The paper describes this as "a reward between 0 and 1" but doesn't specify the exact scoring rubric (presumably based on whether the metadata fields are present and correct).

These three components are additive (or possibly weighted, though the paper doesn't specify weights), producing a total reward that combines content correctness (unit tests), generation termination behavior (EOS), and output structure compliance (metadata).

Training Configuration. The key training hyperparameters:

Hardware: 8×H100 GPU node (one node)
Training data: olmOCR2-synthmix-1025 (2,186 pages, 30,381 test cases)
Training duration: One epoch over the synthetic data
Completions per document: 28 per training step
KL penalty coefficient: $\beta = 0.01$
Base model: Qwen2.5-VL-7B-Instruct after one epoch SFT on olmOCR-mix-1025

The paper doesn't specify learning rate, batch size, optimizer settings, or other GRPO hyperparameters beyond $\beta = 0.01$ and the TRL library implementation (which implies defaults from that library).

Model Souping (Weight Averaging). Rather than using a single training run, the final model is produced by training six independent runs with different random seeds and averaging their weights — a technique known as model souping (Wortsman et al., 2022; Matena and Raffel, 2022). The paper specifies: "We used importance sampling at both the token level (3 runs) and the sequence level (3 runs); more details on their difference in Zheng et al. (2025)." This refers to Group Sequence Policy Optimization (GSPO), a variant that uses importance sampling to reweight the policy gradient update — token-level importance sampling weights each token in a sequence by an importance ratio, while sequence-level importance sampling uses a single weight for the entire sequence.

The souping procedure is: train six models from the same SFT checkpoint, each with a different random seed; after training, compute the element-wise average of their weights to produce a single model. The paper reports that this improves performance, consistent with the broader finding in the model souping literature that averaging weights from multiple fine-tuning runs reduces variance and can produce better generalization than any individual run. Souping does not increase inference cost — the averaged model has the same architecture and parameter count as each individual model.

Why Multiple Seeds and Importance Sampling Variants? The paper doesn't deeply motivate this design choice, but the logic is likely: RL training is notoriously high-variance, with outcomes sensitive to random seed (which affects data ordering, dropout patterns, and initialization of any new parameters). Training multiple seeds and averaging reduces this variance. The split between token-level and sequence-level importance sampling (3 runs each) explores whether the choice of importance sampling granularity matters — and averaging across both variants hedges against one being worse.

3.4.5 Incremental Improvements (Pre-RL System Enhancements)

Table 3 in the paper documents a sequence of improvements between the original olmOCR (68.2 on olmOCR-Bench) and the SFT baseline before RL training (78.5). These improvements are not the main methodological contribution but represent important engineering choices that compound with the RL gains. Each is described briefly in Section 4 of the paper.

Dynamic Temperature Scaling (+4.6 points: 68.2 → 72.8). The original olmOCR used a fixed sampling temperature of 0.8. The authors found that lower temperatures produce better results but increase the risk of repetition loops — the model generating the same token or phrase repeatedly without producing an EOS token. The solution is dynamic temperature scaling: start generation at temperature 0.1, and if the model fails to produce an EOS token (i.e., starts repeating), increase the temperature to 0.2, then 0.3, and so on up to a maximum of 0.8. Each temperature increase is triggered by detecting a repetition failure. This allows the model to benefit from low-temperature (more deterministic, higher-quality) generation most of the time while having an escape mechanism when low temperature causes it to get stuck.

Why this works: Lower temperature sharpens the model's output distribution, making it more likely to sample high-probability (usually correct) tokens. But autoregressive generation can enter degenerate states where the model's own output creates a context that makes it predict the same token repeatedly — this is more likely at low temperatures because the model is less willing to sample alternative tokens that might break the loop. Dynamic temperature scaling provides an automatic circuit breaker: it runs at high quality by default and only sacrifices quality for diversity when necessary to escape a loop.

Better Prompting (+3.0 points: 72.8 → 75.8). The authors discovered a bug: "the order of image and the text was mismatched between training and inference prompts." In some configurations, the training prompt placed the image before the instruction text, while the inference prompt placed the instruction text before the image (or vice versa). Standardizing to always place text first (image second) in both training and inference substantially improved performance. The paper reports that experimenting with the reverse order (image first) showed "no meaningful difference in OCR performance," but consistent ordering matters because it allows "prompt caching by the inference engine." When the text portion of the prompt is fixed and comes first, the inference system can cache the key-value states for that part of the sequence across all documents, reducing per-document computation. This is an example of a seemingly minor implementation detail (prompt ordering) that has outsized effects because VLMs process image and text tokens differently and are sensitive to their relative positions.

New Trainer, YAML, Image Resize, Qwen 2.5 VL (+2.7 points: 75.8 → 78.5). These changes are batched in the paper's Table 3 as a single line, but they represent four distinct improvements:

New trainer: The SFT training code was reimplemented with minor hyperparameter tweaks, notably "avoiding weight decay on the bias and layer norm weights" — a standard practice in transformer fine-tuning that prevents the regularization from shrinking parameters that aren't supposed to be close to zero. The paper states this had "no meaningful benchmark score difference."
YAML output format: As discussed in 3.4.3, switching from JSON to YAML reduced retry rates and repetition loops. No score difference, but improved inference efficiency.
Image resizing: The original olmOCR resized input images to 1024 pixels on the longest edge. olmOCR 2 uses 1288 pixels instead. Larger images preserve more detail (especially for small text, subscripts, and superscripts common in math and tables), improving accuracy at the cost of more computation per image. The paper reports performing "a sweep of image sizes and picked this size as a reasonable balance between benchmark score and inference speed."
Qwen 2.5 VL base model: Switching from Qwen 2 VL to Qwen 2.5 VL provided a slight improvement, likely due to the newer model's stronger pretrained visual understanding capabilities.

Blank Page Handling (+0.0 points: stays at 78.5). The authors discovered that their data loader was skipping all blank pages during training — the model had never seen an empty or nearly empty document page. At inference time, on actual blank pages, it would hallucinate content. Fixing the data loader to include blank pages and retraining eliminated this hallucination behavior. The paper notes this "didn't impact benchmark scores" because olmOCR-Bench doesn't contain blank pages, but it improved real-world reliability.

Cumulative Effect. The progression from 68.2 (original olmOCR) to 78.5 (SFT baseline before RL) is a 10.3 point improvement from engineering and data improvements alone — larger than the gap from many competing models. The RL training then adds a further 3.9 points (78.5 → 82.4), demonstrating that the unit test reward signal provides gains on top of a strong SFT baseline. The paper doesn't provide an ablation showing how much of the 3.9-point RL gain comes specifically from the unit test design versus other aspects of the RL setup (GRPO, souping, etc.), which is a limitation of the reported results.

4. Key Insights and Innovations

Innovation 1: Binary Unit Tests Are a Scalable and Semantically Aligned Reward Signal for OCR Training

Prior to this work, the dominant paradigm for training OCR models—and the default assumption in the field—was that the training objective should mirror the evaluation metric. Since most OCR benchmarks use edit distance or its variants (OmniDocBench, Ouyang et al., 2024), training objectives were correspondingly continuous: cross-entropy loss against token-level reference transcripts, or continuous reward signals in RL setups (Infinity Parser, Wang et al., 2025a, uses edit distance, paragraph count, and structural consistency as its RL reward). The unstated assumption was that a good training signal should provide smooth, dense gradients that reward partial correctness proportionally to how close the output is to the reference.

olmOCR 2 challenges this assumption at a conceptual level. The paper's core argument—developed in Section 2 through Figure 1 and Figure 2—is that edit distance and continuous scores are not just inconvenient; they are misaligned with what "correctness" actually means in document OCR. They penalize equivalent linearizations differently (Figure 1: a floating caption before vs. after a paragraph gets wildly different edit distance scores despite both being valid). They reward semantically incorrect outputs that happen to share characters with the reference (Figure 2: a nearly-correct LaTeX string that fails to render gets higher edit distance than a character-different string that renders perfectly). And they weight all character differences equally, making a single misplaced table cell—a catastrophic failure for downstream use—disappear into the noise of hundreds of correctly transcribed words.

The conceptual move is to abandon the continuous, reference-matching paradigm entirely in favor of binary, criterion-referenced tests that check specific, interpretable properties: does this phrase appear? Does this header not appear? Does this table cell maintain its relative position? Does this equation render correctly? This is, at root, a measurement theory argument transported from software engineering (unit testing) to machine learning: define what correctness means in terms of pass/fail criteria, and optimize for the fraction of criteria satisfied, rather than optimizing for similarity to a single canonical output.

What makes this innovation distinctive beyond olmOCR's prior work (Poznanski et al., 2025, which already proposed unit tests for evaluation) is the scaling argument. The original olmOCR-Bench unit tests were manually created and verified—they took "hours of work to create and check by hand" (Section 3.1). This made them an excellent evaluation tool but completely impractical as a training signal. The insight of olmOCR 2 is that the same HTML structure used to represent a document semantically can be mined programmatically for unit tests: headers go in <header> tags (→ Text Absence tests), tables use <table> structure (→ Table Accuracy tests), math equations use KaTeX markup (→ Math Formula Accuracy tests), and the document order of elements encodes reading order (→ Reading Order tests). The HTML generation pipeline (described in Section 3.4.1 of the Technical Approach) doesn't need to produce perfect transcriptions of the source PDF; it only needs to produce internally consistent HTML whose semantics can be mined for tests. This decoupling of ground-truth fidelity from reward-signal quality is what makes unit tests scalable as a training resource.

A subtle but important implication: this design choice inverts the usual relationship between training signal density and signal quality. Edit distance provides a dense signal (every character contributes to the score) but a semantically noisy one. Binary unit tests provide a sparse signal (a page with 14 tests gives 15 possible reward values) but each test's outcome is semantically meaningful. The paper's empirical claim is that the sparsity is not a problem for GRPO training—the group-relative advantage normalization handles the coarse reward granularity—while the semantic alignment provides a cleaner optimization landscape. The 3.9-point improvement from RL over the SFT baseline (Table 3: 78.5 → 82.4) provides evidence for this claim, though the paper does not provide an ablation comparing unit-test rewards against a continuous reward function in the identical training setup, which would be needed to definitively isolate the effect of the reward design choice.

The significance of this innovation extends beyond OCR. The pattern—define correctness criteria as binary, semantically meaningful tests, then scale test creation through a synthetic data pipeline that encodes those criteria in its output format—is applicable to any structured prediction task where the space of valid outputs is larger than the space of reference outputs. Document parsing, code generation (unit tests as rewards), data extraction from forms, and structured information retrieval from semi-structured sources all share this property. The paper doesn't make this generalization argument explicitly, but the framework it demonstrates is more broadly significant than the specific OCR application.

Innovation 2: The Synthetic HTML Pipeline as a Robust, Self-Contained Ground Truth Generator

The second conceptual contribution is the design of a synthetic data pipeline that is robust to errors in the generator model itself. This is a subtle but important departure from how synthetic training data is typically created for supervised learning.

The standard approach to synthetic data generation in OCR and document understanding—used by the original olmOCR (Poznanski et al., 2025) and many subsequent systems (dots.OCR, Jian et al., 2025; Monkey OCR, Li et al., 2025; DeepSeek-OCR, DeepSeek-AI, 2025)—is to use a powerful model (typically GPT-4o or a comparable frontier VLM) to generate high-quality target outputs for real document images, then train a smaller model to imitate those outputs through supervised fine-tuning. In this paradigm, the quality of the training signal depends directly on the frontier model's OCR accuracy: if the frontier model misreads a word, the student model learns to misread that word. The student inherits the teacher's error distribution.

The olmOCR 2 pipeline works differently. Claude Sonnet is used to generate an HTML representation of a document page, but the HTML is not treated as a perfect transcription of the original PDF. Instead, the HTML becomes the ground truth in its own right—the unit tests are extracted from it, and the RL reward measures whether the OCR model's output matches the HTML's version of the content, not whether it matches the original PDF. As the paper states explicitly in Section 3.1: "our pipeline is robust to hallucinations: even in cases where Claude makes an error when it is performing OCR, that does not affect our pipeline, as we use the HTML output alone to generate unit tests."

This is a self-consistency approach rather than an accuracy-to-source approach. The reward signal asks: "Given this rendered page image (which was rendered from the HTML), can the model produce text that passes the tests extracted from that same HTML?" The reward is therefore always perfectly aligned with the synthetic ground truth, because the ground truth literally defines the test criteria. This eliminates the noise that would come from comparing the OCR model's output against the original PDF and finding discrepancies caused by Claude's OCR errors.

The tradeoff, which the paper acknowledges implicitly, is that the model is being trained to reproduce a stylized version of documents—the version that Claude Sonnet produces when converting PDF pages to HTML. If Claude has systematic biases (e.g., always linearizing multi-column text in a particular order, or always formatting tables in a particular way), the OCR model will learn those biases. The paper mitigates this through two mechanisms: (1) sampling real-world PDFs for layout diversity (so Claude's HTML must adapt to many different layouts, not just a few templates), and (2) using an iterative refinement process where Claude improves its HTML based on feedback. But the fundamental property remains: the training signal optimizes for consistency with the synthetic HTML, not for faithfulness to the original document.

This design choice is not a bug—it's a feature that makes the pipeline practical. Without it, every Claude hallucination would create a noisy or actively misleading reward signal, and the pipeline would need expensive human verification to filter out errors. By making the pipeline self-contained, the authors achieve a cost of approximately $0.12 per page and can generate 30,381 test cases from 2,186 pages without any manual checking. The significance is that this robustness property decouples the data generation cost from the generator model's accuracy, which is a general design principle for synthetic training pipelines that use one model to create supervision for another.

The closest prior work, Infinity Parser (Wang et al., 2025a), also uses HTML renderings for synthetic data, but the paper highlights a "slight difference": Infinity Parser "injected sampled real content into pre-made HTML layouts" while olmOCR 2 "used sampled real content to seed generation of full HTML pages." This distinction matters for diversity: template-based approaches can only generate as many distinct layouts as there are templates, while seeding from real PDFs provides effectively unbounded layout diversity. The paper doesn't provide an ablation comparing the two approaches, but the claim implicit in the design choice is that layout diversity from real-world seeding improves generalization.

Innovation 3: The RL Training Recipe Treats Diverse System-Level Improvements and the Core Algorithm as a Single, Cumulative Optimization Stack

A less flashy but practically important innovation is the paper's presentation of the full development trajectory as an integrated system rather than isolating the "academic contribution" from the "engineering improvements." Table 3 documents six steps from the original olmOCR (68.2) to the pre-RL SFT baseline (78.5), each with a specific benchmark impact: dynamic temperature scaling (+4.6), better prompting (+3.0), model migration + infrastructure improvements (+2.7). The RLVR + synthetic data + souping step then adds a final +3.9 points.

This is noteworthy because the paper resists the common academic temptation to claim that the RLVR contribution is the "real" innovation and the rest is implementation detail. The 10.3-point gain from engineering improvements is larger than the 3.9-point gain from the algorithmic contribution, and the paper documents it transparently. This matters for two reasons:

First, it provides a realistic picture of what it takes to build a state-of-the-art OCR system. The paper's journey from 68.2 to 82.4 involves fixing bugs (prompt ordering mismatch, blank page skipping), tuning inference hyperparameters (temperature, image resolution), switching output formats (JSON → YAML), upgrading base models (Qwen 2 VL → Qwen 2.5 VL), refreshing training data (GPT-4o → GPT-4.1), and then applying a novel RL training approach. Any one of these steps in isolation would be a modest improvement; the cumulative effect is what achieves state-of-the-art. This is an honest portrayal of how modern ML systems are built—through compounding marginal gains—that is often obscured in papers that report only the final number.

Second, it establishes a baseline that gives the RL contribution interpretable context. The RL training provides a 3.9-point gain on top of a 78.5 SFT baseline, not on top of the original 68.2 baseline. This means the unit test reward is providing signal that the SFT training did not already capture. The largest improvements from the RL step (visible in the final row of Table 3 in the breakdown columns) are in math equations (71.9 → 84.9, a 13-point jump), multi-column layouts (81.4 → 81.9, a slight improvement), and tables (72.9 → 84.9 in the "Tables" column, though this column includes both SFT and RL effects from the data mix and RL training combined). The paper does not provide a breakdown that isolates the RL gain per sub-benchmark (only the cumulative pre-to-post-RL comparison), which limits causal attribution, but the pattern is consistent with the claim that unit tests are particularly effective for structured elements (math, tables) where edit distance is most misaligned.

The paper's presentation of model souping as part of the final recipe—training six models with different random seeds and different importance sampling granularities, then averaging their weights—is also worth noting as a practical contribution. Souping is not novel (Wortsman et al., 2022), but its application to RLVR-trained OCR models, and the specific choice to split between token-level and sequence-level importance sampling (3 runs each, referencing Zheng et al., 2025's GSPO), represents a concrete recipe that practitioners can follow. The paper does not ablate souping vs. single-model, so the marginal contribution of averaging is unknown, but the technique is presented as "beneficial" and is part of the final state-of-the-art configuration.

Innovation 4: The Open Development Process as a Research Contribution

The paper's final distinctive contribution is not technical but methodological: it demonstrates that competitive, state-of-the-art OCR can be developed in a fully open manner, with all components—model weights, training data, training code, inference code—released under permissive licenses. Table 1 is structured around this claim, comparing olmOCR 2 not just on benchmark score but on openness criteria alongside 16 other OCR systems.

This matters because the OCR landscape in 2025 is fragmented along openness lines. At one extreme, GPT-4o, Gemini Flash, and Mistral OCR provide API-only access with no model weights, no training data, no training code, and usage governed by terms of service. At the other extreme, systems like PaddleOCR-VL, Qwen 2.5 VL, and Infinity Parser release model weights under open licenses (Apache 2.0) but may not release training data or code. In between are systems that release weights with usage restrictions (OpenRAIL-M, AGPL v3) or don't specify their license at all.

olmOCR 2 positions itself at the maximally open end: Apache 2.0 license, full release of olmOCR-mix-1025 (SFT data, 267,962 pages), olmOCR2-synthmix-1025 (RL data, 2,186 pages with 30,381 tests), training code implemented on the TRL library, inference code in the open-source olmOCR repository, and model weights on Hugging Face. The claim—implicit but clear from Table 1's structure—is that openness and state-of-the-art performance are not in tension. You don't need to compromise on benchmark scores to be fully open, and you don't need proprietary data or closed infrastructure to compete with API-only systems.

Whether this constitutes an "innovation" in the traditional sense is debatable—it's more of a demonstration and a norm-setting contribution. The paper shows that the approach works, establishes a template for others to follow, and implicitly argues (through its permissive license choices) that the field benefits from open releases that enable reproduction, inspection, and downstream innovation. This is significant because it counters the trend toward closing up systems as they become more capable (a pattern visible in the progression from fully open early models to API-only frontier systems). The paper's six-month development history with "frequent version updates accompanied by full data, model and code releases" provides a real-world case study that open development is compatible with rapid iteration.

5. Experimental Analysis

Evaluation Methodology

Dataset. All evaluation is conducted on olmOCR-Bench (Poznanski et al., 2025), the benchmark introduced in the original olmOCR paper. It consists of English-language document pages drawn from diverse sources including academic papers (ArXiv), old scans, documents with tables, multi-column layouts, headers and footers, and long documents with small text. The benchmark is organized into sub-categories that correspond to different OCR challenges: ArXiv, Old scans math, Tables, Old scans, Headers & footers, Multi column, Long tiny text, and Base. The paper does not specify the exact number of pages in the benchmark, but references it as the evaluation standard used consistently across all reported results in Tables 1, 2, and 3. The benchmark is under active development, with the paper noting contributions from "many developers of other open OCR systems" who have provided "usage of and feedback on olmOCR-Bench."
Base model(s). The primary model under development is Qwen2.5-VL-7B-Instruct (Bai et al., 2025), a 7-billion-parameter vision-language model from the Qwen family. The paper traces an evolution path: the original olmOCR (February 2025) used Qwen 2 VL (Wang et al., 2024b); an intermediate version switched to Qwen 2.5 VL, producing a slight improvement (Table 3). For the FLOPs-matched comparison implicit in Table 1, a wide range of baselines spanning different model families and scales are evaluated: GPT-4o (unknown parameter count, API), Gemini Flash 2 (unknown, API), Mistral OCR API (unknown, API), Qwen 2 VL 7B, Qwen 2.5 VL 7B, several 3B models (MonkeyOCR Pro 3B, Nanonets OCR2 3B), and several 7B models (Infinity-Parser 7B, DeepSeek-OCR). The 7B scale is chosen as a deliberate practical sweet spot — large enough for complex document understanding, small enough for single-GPU deployment.
Metrics. The primary metric is the olmOCR-Bench overall score, reported as a percentage with ±1 standard deviation error bars (e.g., 82.4 ± 1.1). This score is computed as the fraction of binary unit tests passed across all documents in the benchmark. The benchmark is organized into sub-categories, and Table 3 reports per-category scores as well as the overall aggregate. Each unit test is binary (pass/fail), and the per-document score is the pass rate across tests for that document. The overall score averages these document-level scores (or equivalently, computes the total pass rate across all tests). The six unit test types are: Text Presence (exact phrase matching), Text Absence (headers/footers stripped), Natural Reading Order (relative position constraints), Table Accuracy (cell relative positioning), Math Formula Accuracy (KaTeX rendering equivalence), and Baseline Robustness (no repeated n-grams, no non-target language characters). The benchmark's scoring rules — where each test case is "either a pass or fail" — are described as the same rules used for both evaluation and RL reward computation (Section 3.2).
Baselines. The paper evaluates against an extensive set of 16 comparison systems (Table 1), including:
- Frontier API models: GPT-4o (OpenAI et al., 2024), Gemini Flash 2 (Google, 2025), Mistral OCR API
- Open-source OCR systems: Marker 1.10.1 (Paruchuri, 2025a), MinerU 1.3.10 and MinerU 2.5.4 (Wang et al., 2024a; Niu et al., 2025)
- OCR-specialized VLMs: Nanonets OCR S and Nanonets OCR2 3B (Mandal et al., 2025), MonkeyOCR Pro 3B (Li et al., 2025), Infinity-Parser 7B (Wang et al., 2025a) — evaluated at two time points (June and October 2025), dots.OCR (Jian et al., 2025), Chandra OCR 0.1.0 (Paruchuri, 2025b), PaddleOCR-VL (Cui et al., 2025a), DeepSeek-OCR (DeepSeek-AI, 2025)
- Base VLMs (not OCR-specialized): Qwen 2 VL 7B (Wang et al., 2024b), Qwen 2.5 VL 7B (Bai et al., 2025)
- Prior olmOCR versions: The original February 2025 release (Poznanski et al., 2025) and several intermediate checkpoints
Results marked with * in Table 1 are "reported by their authors" (not independently reproduced). All other results — including all olmOCR variants, Marker, MinerU, DeepSeek-OCR, and the API models — are "fully reproduced by ourselves" according to the table notes. The paper also compares against different versions of itself in Table 3, tracing the development trajectory from the initial release through incremental improvements to the final olmOCR 2.
Generation budget / compute accounting. There is no explicit FLOPs counting or generation budget comparison in this paper — unlike the reference example paper which carefully accounts for test-time compute in "generations." This is a training methodology paper, not a test-time scaling paper. The relevant compute accounting is in the training setup: RL training uses 28 completions generated per document on an 8×H100 GPU node (Section 3.2), SFT training operates on 267,962 pages (Section 3.1), and the Claude Sonnet synthetic data pipeline costs approximately $0.12 per document page for HTML generation. For inference comparisons, the paper notes performance differences from image resolution (1024px → 1288px) and mentions sweeping image sizes to find a "reasonable balance between benchmark score and inference speed," but does not report inference latency or FLOPs for any system. This is, practically, a paper about how to train a better model, not about how to trade off compute at inference time. The closest the paper comes to a compute comparison is the implicit one in Table 1: models at 3B and 7B scales compete with API-based frontier models of unknown (presumably much larger) scale.
Cross-validation / statistical protocol. The paper does not describe cross-validation, train/test splits, or statistical significance testing beyond reporting ±1 standard deviation error bars on benchmark scores. For the RL training, the paper uses six independent training runs with different random seeds, all evaluated on the same olmOCR-Bench, and reports the souped (averaged) model as the final result. There is no held-out validation set described for the RL stage — the synthetic data (olmOCR2-synthmix-1025, 2,186 pages) appears to serve as both training data and the source of the reward signal, with evaluation on the separate olmOCR-Bench. The SFT data (olmOCR-mix-1025, 267,962 pages) is presumably also disjoint from the benchmark, though this is not explicitly stated. The paper's results are point estimates with error bars derived from multiple evaluation runs (the "±1.1" notation), not from cross-validation folds. For the development trajectory in Table 3, each row represents a distinct model checkpoint evaluated on the same benchmark — there is no correction for multiple comparisons or adaptive overfitting to the benchmark through iterative development.

Main Quantitative Results

Overall Benchmark Performance (Table 1)

The headline result appears in Table 1: olmOCR 2 achieves 82.4 ± 1.1 on olmOCR-Bench, which the paper claims as state-of-the-art among systems with fully open data, model, and code. The nearest competitors are Chandra OCR 0.1.0 at 83.1 ± 0.9 (open weights, training code unreleased, OpenRAIL-M license with usage restrictions) and Infinity-Parser 7B (October 2025 version) at 82.5 ± ? (Apache 2.0 license, unclear error bars). Among truly open systems, the next best is PaddleOCR-VL at 80.0 ± 1.0 (Apache 2.0, but training data not released per Table 1 footnote). Mistral OCR API scores 72.0 ± 1.1, and GPT-4o scores 68.9 ± 1.1 — both substantially below olmOCR 2, despite being much larger proprietary models (GPT-4o predates the October 2025 olmOCR 2 release by over a year and has presumably been updated since).

Comparing the original olmOCR (68.2 ± 1.1, February 2025) to olmOCR 2 (82.4 ± 1.1, October 2025), the improvement is +14.2 points. This is a relative improvement of approximately 21% over the original score, representing six months of development across both engineering improvements and the RLVR training contribution.

Development Trajectory and Incremental Gains (Table 3)

Table 3 breaks down the journey from 68.2 to 82.4 into six steps, with per-sub-benchmark scores at each stage:

Step 1: Original olmOCR — 68.2. The February 2025 release. Per-category strengths: Headers & footers (93.4) and Base (97.9) are already strong. Notable weaknesses: Multi column (54.8), Tables (62.3), and ArXiv (63.3). Math on old scans is at 67.5, old scans generally at 38.6.

Step 2: + Dynamic temperature scaling — 72.8 (+4.6). The improvement is broad across categories: ArXiv jumps from 63.3 to 71.4 (+8.1), Old scans math from 67.5 to 73.1 (+5.6), Multi column from 54.8 to 64.9 (+10.1), Tables from 62.3 to 65.6 (+3.3). Long tiny text shows a large gain from 81.2 to 76.6 — likely a typo or inversion in the table, since this would represent a degradation, but the paper doesn't flag it as such. Old scans are essentially flat (38.6 → 40.5, +1.9). Headers & footers actually decline slightly (93.4 → 93.2, -0.2), and Base drops noticeably (97.9 → 96.7, -1.2). The category-level pattern suggests that dynamic temperature scaling primarily helps with complex layouts (multi-column, ArXiv) and math, where low-temperature generation provides more deterministic, higher-quality output, while slightly hurting performance on simpler tasks where the original higher temperature was adequate and the restart mechanism may introduce variance.

Step 3: + Better prompting — 75.8 (+3.0). Gains are again broad: ArXiv 71.4 → 76.3 (+4.9), Old scans math 73.1 → 76.0 (+2.9), Tables 65.6 → 70.2 (+4.6), Multi column 64.9 → 71.9 (+7.0). Old scans improve from 40.5 to 43.2 (+2.7). Headers & footers recover to 94.1, and Long tiny text partially recovers to 77.5. The consistent prompt ordering between training and inference is credited for these gains — the paper states that the previous mismatch meant the model was processing images differently during training versus evaluation, and fixing this improved performance particularly on complex pages where prompt structure interacts with layout understanding.

Step 4: + New trainer, YAML, image resize, Qwen 2.5 VL — 78.5 (+2.7). These four changes are batched together. ArXiv reaches 78.8 (+2.5), Multi column jumps to 81.4 (+9.5) — the single largest sub-category gain in the entire trajectory — and Base recovers to 99.8. Tables improve slightly (70.2 → 71.9, +1.7), Old scans math improves (76.0 → 77.5, +1.5), Old scans improve (43.2 → 45.4, +2.2). Headers & footers are flat (94.1 → 94.2). The massive Multi column improvement likely reflects the image resize from 1024px to 1288px: multi-column documents pack more content per page, and higher resolution allows the model to better distinguish text in narrow columns, small fonts, and dense layouts. The YAML format switch, while not affecting accuracy directly, reduces retries and may improve effective throughput without quality loss.

Step 5: + Handle blank pages — 78.5 (+0.0). The overall score is unchanged because olmOCR-Bench doesn't contain blank pages. Sub-category scores shift slightly: ArXiv 78.8 → 78.6 (-0.2), Old scans math 77.5 → 79.9 (+2.4), Tables 71.9 → 72.9 (+1.0), Old scans 45.4 → 43.9 (-1.5), Headers & footers 94.2 → 95.1 (+0.9), Multi column 78.6 → 77.3 (-1.3), Long tiny text 81.4 → 81.2 (-0.2), Base 99.8 → 98.9 (-0.9). These are small fluctuations, likely within noise, and the paper correctly identifies this as a robustness fix rather than a performance improvement — the model no longer hallucinates on blank pages, which matters for real-world deployment but doesn't affect benchmark scores.

Step 6: + Synth data, RLVR, souping → olmOCR 2 — 82.4 (+3.9). This is the contribution of the synthetic data pipeline plus GRPO training plus weight averaging. The largest gains are in math-heavy and table-heavy categories: Old scans math jumps from 79.9 to 82.3 (+2.4), Tables from 72.9 to 84.9 (+12.0), and — critically — the "math" column (which is distinct from "Old scans math" and appears as a separate column in the "Tables" section of Table 3) jumps from 71.9 to 84.9 (+13.0). ArXiv (which contains math-heavy academic papers) improves from 78.6 to 83.0 (+4.4). Multi column improves from 77.3 to 83.7 (+6.4). Old scans improve from 43.9 to 47.7 (+3.8). Headers & footers and Long tiny text show modest gains (95.1 → 96.1, +1.0; 81.2 → 81.9, +0.7). Base recovers to 99.7.

The sub-category pattern strongly supports the paper's claim that the unit test reward signal is particularly effective for structured elements: math formulas, tables, and multi-column layouts see the largest RL-driven improvements. The +12.0 point gain on Tables and +13.0 on math (within the Tables sub-benchmark) are the largest single-category improvements in the entire development trajectory. This aligns with the conceptual argument: these are the elements where edit distance is most misaligned with practical correctness (Figure 2: semantically equivalent LaTeX gets different edit distances; table cell misplacement is a tiny fraction of total characters), and where binary unit tests that check rendering equivalence or cell positioning provide a cleaner optimization signal.

SFT Data Mix Comparison (Table 2)

The paper provides an ablation of the SFT training data by comparing two data mixes — the original olmOCR-mix-0225 used for the February 2025 release, and the refreshed olmOCR-mix-1025 — both trained for one epoch and evaluated on olmOCR-Bench:

	olmOCR-mix-0225	olmOCR-mix-1025
ArXiv	78.6	70.8
Old scans math	79.9	79.3
Tables	72.9	77.9
Old scans	43.9	45.6
Headers & footers	95.1	93.7
Multi column	77.3	81.3
Long tiny text	81.2	78.7
Base	98.9	99.3
Overall	78.5 ± 1.1	78.3 ± 1.2

The overall scores are nearly identical (78.5 vs. 78.3), but the sub-category shifts are substantial and in opposite directions. The new mix substantially improves Tables (+5.0) and Multi column (+4.0) while degrading ArXiv (-7.8) and Long tiny text (-2.5). This tradeoff is likely attributable to the SFT data changes: the new mix uses GPT-4.1 instead of GPT-4o for generating supervision targets, uses HTML table format (which may improve table learning but is less natural for the text-heavy, single-column ArXiv papers), and re-processes equation formatting with standardized \[ and \( delimiters. The paper does not ablate which of these changes causes which sub-category shift, but the pattern is consistent with a design choice that prioritizes table and layout accuracy at some cost to pure text transcription on simpler layouts.

This comparison is important context for interpreting the final RL gains: the SFT baseline that RL starts from already has strong table performance (77.9 vs. the original 72.9) due to data improvements, and the RL step then pushes tables further to 84.9. The 84.9 final table score represents a +12.0 improvement over the original SFT baseline — it's the combination of data mix improvements and RL training that achieves the total gain.

Comparison with Other Systems on Sub-Categories (Table 3)

The full Table 3 comparison against other OCR systems reveals where olmOCR 2's strengths and weaknesses lie relative to competitors:

ArXiv (olmOCR 2: 83.0): olmOCR 2 is competitive but not dominant. PaddleOCR-VL scores 85.7, Infinity-Parser 7B (October 2025) scores 84.4, Marker 1.10.1 scores 83.8. The margin between these systems is small (roughly 1–3 points). DeepSeek-OCR scores 77.2, Mistral OCR API scores 77.2.

Old scans math (olmOCR 2: 82.3): olmOCR 2 is competitive but trails Infinity-Parser 7B (83.8) and Chandra OCR (80.3). DeepSeek-OCR scores 73.6, Mistral scores 67.5.

Tables (olmOCR 2: 84.9): This is a strength for olmOCR 2. Nanonets-OCR2-3B scores 86.8 (highest in the comparison), Infinity-Parser 7B scores 85.0. The margin between the top systems is tight (within 2 points). MinerU 2.5.4 scores 84.9 (tied with olmOCR 2), PaddleOCR-VL scores 84.1. Mistral OCR API trails at 60.6 — a 24.3 point gap that highlights how table parsing remains a significant challenge for general-purpose API models.

Old scans (olmOCR 2: 47.7): This is a weakness across all systems. The highest score in this category is Chandra OCR at 50.4, followed by Infinity-Parser 7B at 47.9. olmOCR 2 at 47.7 is effectively tied with Infinity-Parser. Most systems score in the 33–45 range. Old scans clearly represent the hardest category — even the best systems get less than half of the unit tests correct. This likely reflects the combination of degraded image quality, non-standard typography, and historical document conventions that differ from the modern documents dominating training data.

Headers & footers (olmOCR 2: 96.1): Strong performance across the board for most systems. PaddleOCR-VL leads at 97.0, olmOCR 2 at 96.1, DeepSeek-OCR at 96.1, MinerU 2.5.4 at 96.6. Nanonets-OCR2-3B is a notable outlier at 32.1, suggesting a specific failure mode in header/footer detection.

Multi column (olmOCR 2: 83.7): olmOCR 2 is competitive. Infinity-Parser 7B leads at 84.2, followed by olmOCR 2 at 83.7, PaddleOCR-VL at 79.9, Marker at 80.0. DeepSeek-OCR trails at 66.4 — a gap of 17.3 points. Multi-column reading order is one of the canonical challenges that motivated the unit test approach (Figure 1), and olmOCR 2's strength here is consistent with the claim that binary unit tests provide better training signal for reading order than continuous scores.

Long tiny text (olmOCR 2: 81.9): olmOCR 2 performs well but trails some competitors. Nanonets-OCR2-3B leads at 93.0, Chandra OCR at 92.3, PaddleOCR-VL at 85.7, Marker at 85.7. Infinity-Parser 7B is at 86.4. The gap between olmOCR 2 (81.9) and the leaders (~93) is roughly 11 points. Long tiny text (small font sizes in dense documents) may be an area where higher-resolution processing or specialized small-text detection components provide advantages that the end-to-end VLM approach has not fully captured.

Base (olmOCR 2: 99.7): Most systems score near ceiling. DeepSeek-OCR and Chandra OCR reach 99.8 and 99.9; olmOCR 2 at 99.7 is essentially perfect on basic text extraction tasks. Mistral OCR API is at 99.4.

The overall pattern shows olmOCR 2 as a strong generalist: it's within 1–3 points of the best system on nearly every category, leads or ties on none (Table 3 footnotes attribute the highest scores to PaddleOCR-VL, Infinity-Parser, Chandra OCR, and Nanonets depending on category), but achieves the top overall score through consistent performance across categories. This is the advantage of the unit test reward signal: it simultaneously optimizes for multiple error types (six test categories) rather than overfitting to a single metric, producing balanced improvements.

Significance of the RL Contribution

The paper does not provide an ablation isolating the RLVR contribution from the synthetic data contribution — the final step in Table 3 combines "Synth data, RLVR, souping" as a single line. This makes it impossible to determine from the reported results how much of the 3.9-point gain (78.5 → 82.4) comes from:

The synthetic data itself serving as additional training examples (even without RL)
The RL training on top of SFT
The souping (weight averaging) procedure
The interaction between these components

The paper does state that "A single epoch of RL training over our synthetic data olmOCR2-synthmix-1025" and "Repeating the RL training for six random seeds and averaging (or 'souping') the checkpoints" are both part of the final recipe (Section 4), but the contributions are not disentangled. This is a significant limitation for interpreting the results — we can see that the full recipe works, but we cannot determine which components are necessary versus merely beneficial, and whether the RL training with unit test rewards specifically (as opposed to any RL training on this data) is responsible for the gains.

Similarly, the paper does not provide a direct ablation comparing unit-test-based RL rewards against a continuous reward function (e.g., edit distance, or the composite reward used by Infinity Parser) in the identical training setup. This is the experiment that would most directly test the paper's central claim — that binary unit tests are a better reward signal than continuous scores. Without it, the evidence is circumstantial: the gains on math and tables are consistent with the conceptual argument, but we cannot rule out that the same gains would have been achieved using any reasonable reward function on the synthetic HTML data.

Ablation Studies and Robustness Checks

The paper's ablation and robustness analysis is, candidly, minimal. This is a short technical report (6 pages) rather than a full research paper, and the experimental section prioritizes breadth (many baselines, many sub-categories) over depth (controlled ablations of the proposed method). Nevertheless, several embedded comparisons serve as implicit ablations:

SFT data mix old vs. new (Table 2): As discussed above, the comparison between olmOCR-mix-0225 and olmOCR-mix-1025 shows that the SFT data refresh changes the sub-category profile substantially (Tables +5.0, Multi column +4.0, ArXiv -7.8) while keeping overall accuracy nearly identical (78.5 vs. 78.3). This serves as an implicit ablation of the SFT data generation process, demonstrating that data quality improvements (GPT-4.1 supervision, HTML tables, standardized equation formatting) shift model capabilities in targeted ways rather than uniformly improving all tasks.

Qwen 2 VL vs. Qwen 2.5 VL (Table 3, step 4): The base model upgrade from Qwen 2 VL to Qwen 2.5 VL is bundled with other changes (new trainer, YAML, image resize), so its isolated effect is unclear, but the paper states it results in "a slight improvement in benchmark score." This is consistent with the general observation that newer base VLMs provide better pretrained representations.

Image resolution sweep (Section 4): The paper reports performing "a sweep of image sizes" and selecting 1288px on the longest edge (vs. the original 1024px) "as a reasonable balance between benchmark score and inference speed." No sweep results are reported, but the documented choice implies that larger images improve accuracy (consistent with the Multi column gains in Table 3, step 4) at a cost the authors deemed acceptable.

JSON vs. YAML output format (Section 4): The paper reports "no benchmark score difference" from switching from JSON to YAML, but notes "fewer need for retries" and "improved inference efficiency." This is important because it demonstrates that the output format choice — which might seem like a minor implementation detail — affects practical deployment characteristics (retry rate, latency variance) even though it doesn't affect accuracy. This ablation is implicit rather than controlled (no head-to-head JSON vs. YAML table is provided).

Blank page handling (Table 3, step 5): The 0.0-point overall score change from fixing the blank page data loading bug is not an ablation in the traditional sense (it's a bug fix), but it demonstrates that the benchmark is insensitive to a real-world reliability improvement. This highlights a limitation of the benchmark rather than the model — olmOCR-Bench apparently doesn't contain blank pages, so improvements to blank page handling are invisible in the reported metrics.

Multiple training seeds and souping (Section 4): The paper trains six models with different random seeds and averages their weights. No comparison is provided between the souped model and the individual models, so the marginal benefit of souping is unknown. The paper states it is "beneficial to train multiple models, and average, or soup, their weights," but the magnitude of benefit is not quantified. Similarly, the split between token-level and sequence-level importance sampling (3 runs each) is not ablated — there's no comparison showing whether one importance sampling granularity outperforms the other, or whether the 3+3 split is better than all-6 of one type.

Missing ablation: unit test reward vs. continuous reward. This is the most significant experiment that was not run. The paper's central methodological claim is that binary unit tests are a better reward signal than continuous scores (edit distance, structural consistency, etc.) for RL training of OCR systems. To test this, one would need to run the identical RL training setup (same data, same model, same GRPO hyperparameters, same souping) with different reward functions: (a) the binary unit test pass rate as used in the paper, (b) a continuous edit distance score against the HTML ground truth, (c) a composite continuous score similar to Infinity Parser's (edit distance + paragraph count + structural consistency), and potentially (d) no RL training (SFT only). With this comparison, one could determine: does RL training with any reward outperform SFT alone? Does the unit test reward outperform edit-distance-based rewards? Does it outperform a composite continuous reward? The paper provides none of these comparisons, leaving the central claim supported only by the conceptual argument (Figures 1 and 2) and the circumstantial evidence that gains are largest on math and tables (where the conceptual argument predicts unit tests should help most).

Missing ablation: synthetic data contribution vs. RL contribution. The current results combine synthetic data, RLVR, and souping into one step (+3.9 points in Table 3). One could run: (a) SFT on the synthetic HTML data (without RL), (b) RL on the original SFT data (without synthetic data), (c) both, and (d) neither. This would disentangle whether the gains come from additional training data (the 2,186 synthetic pages represent roughly 0.8% of the SFT data volume — a small addition if the benefit is purely from data quantity), from the RL optimization process, or from their interaction. Without this, we cannot tell if the synthetic data pipeline is primarily valuable as a source of unit tests (enabling RL) or as a source of additional training examples.

Missing ablation: per-category RL gains. Table 3 shows cumulative scores from the original olmOCR through to olmOCR 2, but the pre-RL to post-RL comparison is only available at the last two rows (78.5 → 82.4). What's missing is a row showing the SFT-on-new-mix model evaluated against the RL-trained model on the same benchmark version, broken down by sub-category. This would reveal whether the +3.9 overall gain is concentrated in specific categories (as the conceptual argument predicts: math and tables should benefit most) or diffuse across all categories. The cumulative Table 3 makes it impossible to isolate, because sub-category scores at step 5 (pre-RL) and step 6 (post-RL) reflect different benchmark versions and the cumulative effect of all prior steps.

Missing ablation: GRPO hyperparameters. Only the KL penalty coefficient β = 0.01 is reported. Learning rate, batch size, number of GRPO update steps, advantage normalization method, and PPO-style clipping parameters (if used) are not specified, making reproduction dependent on TRL library defaults. Sensitivity to β — which controls the tradeoff between reward optimization and staying close to the SFT policy — is particularly important: too high and the model doesn't learn from the reward; too low and it may catastrophically forget SFT capabilities.

Missing ablation: number of completions per document (28). The paper uses G = 28 completions per document for GRPO advantage computation. This is a key hyperparameter: too few completions and the group-relative advantage is noisy; too many and training is expensive. Sweeping this parameter (e.g., 8, 16, 28, 56) and measuring both final performance and training efficiency would provide practical guidance, but no such sweep is reported.

Critical Assessment

The paper makes three central claims that need to be evaluated against the experimental evidence:

Claim 1: "Binary unit tests serve as an effective and scalable reward signal for OCR-specialized VLM training." The evidence is that adding "Synth data, RLVR, souping" — which uses binary unit tests as the reward — improves the overall olmOCR-Bench score by 3.9 points (Table 3, step 5 → step 6). The gain is real and the final model achieves state-of-the-art performance among fully open systems. However, the evidence is weaker than it appears for several reasons:

First, the 3.9-point gain bundles synthetic data creation, RL training, and model souping into one step. Without an ablation isolating the reward signal (see "Missing ablation: unit test reward vs. continuous reward" above), we cannot attribute the gain specifically to binary unit tests rather than any RL training on this data. It is entirely possible that RL training with a continuous edit distance reward — or even supervised fine-tuning on the synthetic data — would have achieved similar or better gains. The conceptual argument (Figures 1 and 2) provides a rationale for why unit tests should be better, but the experiments do not test this hypothesis.

Second, the largest gains appear in math and table categories — where the paper's argument predicts unit tests should help most — which is circumstantially consistent but not conclusive. The SFT data refresh (step 1) already improved Tables by 5.0 points (Table 2), suggesting that data quality improvements alone can drive large gains in these categories. The RL gain might simply be additional training on the new SFT data mix plus synthetic data, with the reward function being incidental.

Third, the scalability claim rests on the synthetic pipeline: 2,186 pages yielding 30,381 test cases, at $0.12 per page. This is indeed orders of magnitude more scalable than manual creation. But the paper doesn't demonstrate that 30,381 tests is "enough" in any principled sense — would 10,000 tests have sufficed? Would 100,000 have been substantially better? The paper shows that this data volume works, not that it's optimal or that the approach scales monotonically with test count.

Verdict: Supported with qualifications. The unit test reward signal contributes to a working system that achieves state-of-the-art results, but the specific contribution of binary unit tests versus any RL reward on this data — and the scaling properties of the approach — are not experimentally established.

Claim 2: "The synthetic HTML pipeline is robust to VLM hallucinations and enables programmatic unit test extraction without manual verification." This claim is structural — it's about the pipeline design, not about downstream performance. The evidence is that the pipeline was built and used to generate 30,381 test cases, and that those test cases were used in RL training that improved model performance. The paper's claim that hallucinations "do not affect our pipeline" is definitionally true given the design: tests are extracted from the HTML, so any systematic OCR errors in the HTML become part of the ground truth, not noise in the reward signal.

However, there is an unexamined consequence: the model is being trained to match Claude Sonnet's OCR output, not the source PDF ground truth. If Claude has systematic biases — and it almost certainly does, as all OCR systems do — the trained model will learn those biases. The paper provides no analysis of whether the synthetic HTML contains systematic errors, what those errors look like, or how they might affect downstream model behavior. The diversity of source PDFs (100,000+ documents) provides some mitigation, but the generator model (Claude Sonnet) is a single system with a specific error profile. This matters because the model might learn to reproduce Claude-specific artifacts that don't generalize to real-world documents.

Additionally, the paper does not report any quality control metrics on the synthetic HTML. What fraction of generated HTML pages had rendering errors? What fraction had incorrect reading order? How often did the refinement step actually fix errors? Without these numbers, the claim of "robustness" is a design property, not an empirically verified one.

Verdict: Supported by design, but empirical verification is minimal. The pipeline is logically robust to VLM errors by construction, but the paper provides no measurement of how many errors exist in the synthetic data, what types they are, or whether they affect training outcomes.

Claim 3: "olmOCR 2 achieves state-of-the-art performance while maintaining fully open data, model, and code." This is the best-supported claim in the paper. Table 1 shows olmOCR 2 at 82.4 ± 1.1, competing with Chandra OCR (83.1 ± 0.9, but with usage-restricted license and unreleased training code) and Infinity-Parser 7B (82.5 ± ?, but with unreleased training code per Table 1). Among systems that release model weights, training data, training code, and inference code under permissive licenses, olmOCR 2 is clearly the leader. The table's footnotes document exactly which components are released or restricted for each system, making the comparison transparent.

The caveats: "state-of-the-art" is defined relative to a benchmark (olmOCR-Bench) developed by the same group. While the paper notes that other OCR developers have provided feedback and contributed to the benchmark, it is not independently developed or maintained by a neutral third party — there's an inherent risk that the benchmark captures what olmOCR is good at. The benchmark's coverage of document types, languages (English only), and failure modes reflects the priorities of the olmOCR team. This doesn't invalidate the results, but it means "state-of-the-art on olmOCR-Bench" shouldn't be conflated with "best OCR system in all contexts."

Additionally, the competitive landscape in Table 1 reveals that several systems are within error bars of each other: olmOCR 2 at 82.4 ± 1.1, Infinity-Parser 7B at 82.5 ± ?, Chandra OCR at 83.1 ± 0.9. With overlapping confidence intervals, the ranking among these top systems is not statistically distinguishable given the reported uncertainties. olmOCR 2 is tied for first among fully open systems rather than unambiguously state-of-the-art.

Verdict: Supported. The openness claim is demonstrably true (all components released under Apache 2.0), and the benchmark score claim is true within the reported error margins, with the caveat about benchmark independence and statistical distinguishability from close competitors.

Additional Strengths and Weaknesses in the Experimental Design

Strength: Breadth of baseline comparison. The paper evaluates against 16 other systems spanning API-only, open-weight, and fully open categories, with transparent documentation of which results are self-reproduced versus author-reported. This makes the competitive positioning credible and allows readers to assess whether stated claims hold under different interpretations of "open."

Strength: Development trajectory transparency. Table 3's step-by-step breakdown of how olmOCR went from 68.2 to 82.4 — including bug fixes, infrastructure changes, and data improvements alongside the algorithmic contribution — is unusually honest and provides practical guidance for practitioners building similar systems.

Weakness: Single benchmark, no out-of-domain evaluation. All results are on olmOCR-Bench, which is an English-language benchmark developed by the same team. There is no evaluation on OmniDocBench (Ouyang et al., 2024), DocVQA, or any other independently developed document understanding benchmark. This makes it impossible to assess whether the gains transfer to different document types, languages, or evaluation frameworks. Given that the paper's conceptual argument is about the superiority of unit tests over edit distance, evaluating on an edit-distance-based benchmark would provide a direct test of whether the approach produces models that are genuinely better or merely better at the unit test format.

Weakness: No test-time compute analysis. Unlike the reference example paper (which systematically studies how to allocate inference compute), this paper provides no analysis of how the model's performance scales with inference-time resources — beam search width, number of samples, temperature, etc. The dynamic temperature scaling improvement suggests that inference-time hyperparameters matter substantially, but this is treated as a one-time fix rather than a dimension to be systematically optimized.

Weakness: Error analysis is absent. The paper provides no qualitative examples of what olmOCR 2 gets right that competitors get wrong, or vice versa. There are no confusion matrices showing what types of unit tests are most commonly failed, no analysis of whether the RL training reduces specific error types at the expense of others, and no investigation of failure modes on the hardest sub-category (Old scans, where even the best systems score below 50). The conceptual argument in Section 2 (Figures 1 and 2) provides motivating examples, but these are hypothetical illustrations of the unit test philosophy, not analyses of the trained model's behavior.

Weakness: Statistical reporting is minimal. Error bars are reported as ±1 standard deviation (e.g., 82.4 ± 1.1), but it's unclear what the source of variance is — multiple evaluation runs? Different random seeds? Different prompt phrasings? The paper reports that six training seeds were used and souped, but doesn't report the variance across seeds, which would indicate how sensitive the final performance is to training stochasticity. There are no statistical significance tests comparing olmOCR 2 to its closest competitors.

Weakness: No scaling analysis. The paper does not investigate how performance scales with: amount of synthetic data (would 5,000 pages be substantially better than 2,186?), number of unit tests per page (is 14 tests per page optimal, or would 50 be better?), model size (how does a 3B or 13B variant perform?), or RL training duration (does a second epoch help or hurt?). These scaling analyses would transform the paper from a single-point demonstration into a more general study of how to train OCR models with unit test rewards.

6. Limitations and Trade-offs

6.1 The Synthetic Ground Truth Is Not Ground Truth — It's Claude Sonnet's OCR

The assumption. The pipeline treats the HTML generated by Claude Sonnet as ground truth for unit test extraction, explicitly decoupling synthetic data quality from fidelity to the original source PDF:

"our pipeline is robust to hallucinations: even in cases where Claude makes an error when it is performing OCR, that does not affect our pipeline, as we use the HTML output alone to generate unit tests" (Section 3.1)

This design choice makes the pipeline practical — you don't need human verification of the synthetic data — but it means the RL reward trains the model to reproduce Claude Sonnet's OCR output, errors and biases included, rather than the true content of the source documents.

The consequence. The trained model inherits whatever systematic biases exist in Claude Sonnet's document-to-HTML conversion. These could include: preferring a particular linearization strategy for multi-column text, making consistent errors on certain font families or character sets, formatting tables in idiosyncratic ways, or misreading specific mathematical notations. The paper provides no analysis of what error types or biases Claude Sonnet exhibits in the HTML generation task, making it impossible to assess what artifacts the trained model may have learned. In deployment, this could manifest as the model producing output that looks correct but contains subtle, systematic inaccuracies that trace back to Claude's generation patterns rather than the original documents. For high-stakes applications — legal document discovery, medical record digitization, financial statement extraction — this latent dependency on a frontier model's OCR error profile is a material risk that is invisible in the benchmark scores.

What evidence exists. None. The paper does not report any quality assessment of the synthetic HTML, no comparison between Claude Sonnet's HTML content and the source PDF content for a sample of pages, no measurement of hallucination rates in the HTML generation step, and no analysis of whether the refinement step (Section 3.1, step 3) actually corrects errors or merely reformats. The claim of robustness is a logical property of the pipeline architecture, not an empirically verified one.

Mitigation status. Not addressed in the paper. The diversity of source PDFs (2,186 pages from, presumably, many different document sources) provides some averaging of Claude's error profile across document types, but it does not eliminate systematic biases. The paper does not suggest any quality control mechanism, human auditing step, or data filtering procedure to catch synthetic HTML errors. This is an inherent property of the self-contained pipeline design, not a bug to be fixed — making it a fundamental limitation that users of the approach must accept.

6.2 Difficulty Estimation and Adaptive Compute Allocation Are Not Explored

The assumption. The entire training pipeline and evaluation treat all document pages as equally difficult and allocate identical computation to each — 28 completions per document during RL training (Section 3.2), and a single generation with dynamic temperature at inference. Unlike the reference example paper in the prompt — which builds an entire framework around estimating prompt difficulty and adaptively allocating test-time compute — olmOCR 2 provides no mechanism for understanding which documents it handles well and which it struggles with, beyond the coarse sub-category averages in Table 3.

The paper itself reveals dramatic difficulty variation across document types. On "Old scans," olmOCR 2 scores 47.7 (Table 3) — fewer than half of unit tests passed — while on "Base" (simple text extraction) it scores 99.7. The 52-point gap between these sub-categories is larger than the entire 14.2-point improvement from the original olmOCR to olmOCR 2. Yet the system treats a clean modern PDF and a degraded historical scan identically at inference time.

The consequence. In deployment, the model will silently produce low-quality output on difficult documents with no indication of reduced confidence. A downstream system consuming olmOCR 2's output has no signal about whether a particular extraction is likely to be reliable — the table might be perfectly parsed or completely scrambled, and the text output looks equally fluent in both cases. This is a practical liability for any application that processes heterogeneous document collections (which is essentially all real-world use cases): the user gets no warning when the system is operating outside its reliable performance envelope.

More subtly, the uniform training strategy may be inefficient. If the model receives equal RL updates from easy pages (where it already scores near-perfect and the unit test reward provides minimal signal) and hard pages (where most completions fail and the reward gradient may be noisy), the training budget is not being allocated where it can produce the most improvement. The reference example paper's finding that "the optimal way to use test-time compute is prompt-dependent" (Section 1 of that paper) likely has a training-time analog that this paper does not investigate.

What evidence exists. Table 3 provides per-category breakdowns showing the difficulty variation: Old scans (47.7), Headers & footers (96.1), Base (99.7). Table 1 shows that even the strongest system in the comparison (Chandra OCR) scores only 50.4 on Old scans. The difficulty is not unique to olmOCR — it's a property of the task — but the paper does not acknowledge it as a limitation or design around it.

Mitigation status. Not addressed. The paper does not discuss difficulty estimation, confidence scoring, adaptive compute allocation, or selective routing of difficult documents to human review or alternative processing pipelines. The dynamic temperature scaling mechanism (Section 4) is the closest the paper comes to adaptive behavior — it detects repetition loops and increases temperature to escape them — but this is a failure recovery mechanism, not a difficulty-aware allocation strategy.

6.3 The Central Ablation Is Missing: Unit Test Rewards vs. Continuous Rewards

The assumption. The paper's primary methodological claim is that binary unit tests provide a better RL reward signal than continuous scores (edit distance, structural consistency, paragraph count, etc.) for OCR training. This claim is argued conceptually in Section 2 (Figures 1 and 2) and is the primary point of differentiation from prior work — particularly Infinity Parser (Wang et al., 2025a), which the paper explicitly distinguishes by noting: "we use binary unit tests as our verifiable reward signal while they define their reward based on edit distance, paragraph count, and structural consistency" (Section 5).

The consequence. Without an experiment that runs the identical RL training pipeline with different reward functions — unit tests vs. edit distance vs. a composite continuous score — we cannot determine whether the 3.9-point gain from the "Synth data, RLVR, souping" step (Table 3) is attributable to the unit test design specifically, or whether any reasonable RL reward on the synthetic data would have produced similar improvements. The conceptual argument is plausible, but the experimental results do not test it. This matters because the synthetic data pipeline is expensive and complex: if a simpler continuous reward function (e.g., edit distance against the HTML ground truth) achieves comparable performance, the additional complexity of generating and maintaining unit tests would not be justified. Conversely, if unit tests dramatically outperform continuous rewards, that would be the paper's strongest empirical finding — but the experiment to demonstrate this was not run.

What evidence exists. Only circumstantial. The largest RL-driven gains appear in math and table categories (Tables +12.0, math within Tables +13.0, per Section 5 discussion of Table 3), which is consistent with the conceptual argument that these are the elements where edit distance is most misaligned. But these gains also coincide with the SFT data refresh that specifically improved table formatting and equation standardization (Section 3.1: "more consistent equation formatting," "uses HTML format for tables"), so we cannot disentangle whether the gains come from the reward function design, the data improvements, or their interaction.

Mitigation status. The paper does not acknowledge this as a missing experiment. The comparison to Infinity Parser is purely descriptive ("a more significant difference is that we use binary unit tests...") without an empirical test of which reward design produces better results. This is the most significant experimental gap in the paper, and it limits the strength of the central claim.

6.4 Single Benchmark, Single Model Family, Single Language

The assumption. All results are reported on olmOCR-Bench, an English-language benchmark developed by the same research group. The base model is Qwen2.5-VL-7B-Instruct throughout. No results are reported on any independently developed benchmark (OmniDocBench, DocVQA, FUNSD, SROIE), any non-English documents, any other VLM architecture, or any model scale other than 7B.

While the paper does not explicitly claim generalizability to other settings, the presentation of olmOCR 2 as a "state-of-the-art OCR system" (Section 1) and the competitive comparison against 16 other systems (Table 1) create an implicit expectation that the results reflect general OCR capability rather than benchmark-specific optimization.

The consequence. Several forms of uncertainty arise from this narrow evaluation:

Benchmark overfitting risk. The olmOCR team developed olmOCR-Bench, has iterated on multiple versions of their model against it over six months (Table 3 documents the full trajectory), and evaluated on it repeatedly. The benchmarks used for SFT data filtering, synthetic data selection, and RL reward design are all related to the same task distribution. Without an external benchmark, we cannot distinguish genuine OCR capability improvement from inadvertent overfitting to the specific document types, unit test formats, or evaluation conventions of olmOCR-Bench. The paper notes that other developers have contributed feedback, but the benchmark is not independently maintained.
Base model dependence. The pipeline's effectiveness may depend on properties specific to Qwen2.5-VL — its visual encoder resolution, its pretraining data distribution, its instruction-tuning format, its native output language tendencies (Chinese and English). The synthetic HTML pipeline uses Claude Sonnet, whose output style may align particularly well or poorly with Qwen's learned representations. The paper provides no evidence that the approach transfers to other model families (LLaVA, InternVL, Pixtral) or architectures (larger VLMs, different vision encoders).
Language and domain restriction. English-only evaluation means the system's handling of non-Latin scripts, right-to-left text, multilingual documents, and language-specific typographic conventions (CJK character spacing, Arabic ligatures, Devanagari conjuncts) is completely uncharacterized. Both the SFT data and the synthetic pipeline are English-focused. For the many real-world OCR use cases involving multilingual documents, the paper provides no guidance on expected performance or adaptation requirements.
Scale generalizability unknown. The 7B parameter scale is a deliberate practical choice, but larger models might benefit differently from the RL training (more capacity to learn from the sparse binary reward, or less need for it if SFT already achieves high accuracy). Smaller models (3B, 1B) might show a larger gap between SFT and RL, or might require different hyperparameters. The paper provides no investigation of scaling behavior.

What evidence exists. The paper evaluates only on olmOCR-Bench (English, in-house). Table 1 compares many other systems on this same benchmark, demonstrating competitive positioning but not cross-benchmark robustness. The base model migration from Qwen 2 VL to Qwen 2.5 VL (Table 3) shows a "slight improvement," providing minimal evidence of model-family sensitivity — both are Qwen architectures, so this is a weak test of generalizability.

Mitigation status. Not directly addressed. The paper's conclusion states interest in "further develop the synthetic data pipeline to cover more complicated document types and unit tests" (Section 6), suggesting awareness that the current coverage is limited, but there is no plan described for multi-benchmark or multi-lingual evaluation. The open release of all components implicitly enables others to run these experiments, but the paper itself does not provide them.

6.5 The Difficulty Estimation Cost Is Not Included, and There Is No Difficulty Estimation

The assumption. As discussed in Limitation 2, the paper does not estimate document difficulty at all — every page receives the same inference budget. This limitation examines the training-time analog of this issue: the cost of generating the synthetic HTML ground truth is treated as a one-time data preparation expense and is not factored into any efficiency comparison with alternative training approaches.

The pipeline costs approximately $0.12 per page for Claude Sonnet API calls (Section 3.1), totaling roughly$ 262 for the 2,186-page synthetic data mix. This is modest in absolute terms, but it represents an external dependency on a proprietary frontier model — the pipeline literally cannot function without Claude Sonnet (or an equivalently capable VLM) to generate the HTML. This is a more significant constraint than the dollar cost suggests.

The consequence. There are two distinct consequences:

First, reproduction requires API access to a frontier VLM. A research group or practitioner wanting to extend this work to a new document domain, language, or unit test type must have access to Claude Sonnet (or an equivalent model with comparable document-to-HTML capability). The paper's "fully open" claim (all components released under permissive licenses) is technically true of the released artifacts, but the pipeline that creates the training data depends on a proprietary, API-gated model. This creates a practical barrier: you can use the released model and data freely, but you cannot easily generate new synthetic data for your own documents without paying for and accessing a frontier VLM.

Second, the synthetic data cost is not amortized in any efficiency comparison. The paper compares olmOCR 2 against other systems purely on benchmark accuracy (Table 1), not on total cost of development including data generation. For a fair comparison of "cost to achieve 82.4 on olmOCR-Bench," one would need to account for the Claude API expense, the 8×H100 GPU node for RL training, and the human engineering effort across six months of iterative improvement (Table 3). A competing approach that achieves comparable accuracy through pure SFT on human-annotated data might have different cost characteristics. The paper provides none of this accounting, making cost-informed comparisons impossible.

What evidence exists. The per-page cost is explicitly stated ("approximately $0.12 per document page," Section 3.1). The reliance on Claude Sonnet is described in the pipeline (Section 3.1, step 2). The hardware for RL training is specified ("8×H100 GPU node," Section 3.2). But none of these costs are aggregated, compared against alternatives, or factored into any efficiency metric.

Mitigation status. The paper does not acknowledge the proprietary model dependency as a limitation or propose alternatives (open-weight VLMs with comparable document understanding capability, fully synthetic template-based pipelines that don't require a frontier model, etc.). The cost reporting is transparent but not contextualized.

6.6 The Revision/Correction Loop Problem Is Unaddressed — Outputs Have No Self-Correction Mechanism

The assumption. olmOCR 2 generates a single structured text output per page (using dynamic temperature scaling to escape repetition loops), and that output is final. Unlike the reference example paper, which studies iterative revision models that can refine their own incorrect outputs ("the revision model conditions on its own previous answers and produces an improved answer," Section 6 of that paper), olmOCR 2 has no mechanism for detecting errors in its own output, attempting corrections, or generating multiple candidate outputs to select among. The unit test reward signal is used only during training; at inference time, the model has no access to unit tests and no way to verify whether its output would pass them.

The consequence. The model's inference-time behavior is "one shot": it produces an output, and if that output contains errors — a misread equation, a scrambled table cell, an inverted reading order — there is no fallback, no verification, and no correction. This is a practical limitation for any deployment where OCR errors have meaningful costs:

In a document digitization pipeline, an incorrectly parsed financial figure or legal citation may propagate into downstream databases with no automated check.
In a retrieval system, a document with incorrectly ordered paragraphs will produce plausible-sounding but factually wrong search results, and the system has no internal signal that something went wrong.
In an interactive setting, a user who uploads a PDF and gets a badly parsed table has no recourse except to manually verify the output against the original — defeating much of the purpose of automated OCR.

This limitation is particularly acute given the difficulty variation documented in Table 3: on Old scans, olmOCR 2 fails more than half of unit tests (scores 47.7). Users processing historical documents — a common OCR use case — will encounter frequent errors with no automated recovery mechanism.

The paper's dynamic temperature scaling (Section 4) provides a limited form of error recovery for one specific failure mode: repetition loops. If the model starts repeating tokens, the temperature increases, eventually breaking the loop. But this addresses only the failure to produce a valid output format, not content errors. A table that is perfectly formatted but has the wrong numbers in the wrong cells will not trigger any retry or correction.

What evidence exists. The paper's architecture description (Sections 3.2, 4) makes clear that inference is single-pass with dynamic temperature as the only adaptive mechanism. There is no mention of multi-sample generation, verifier-based selection at inference time, iterative refinement, or confidence estimation. Unlike the original olmOCR paper — which likely had similar properties but predates the recent wave of RLVR work — olmOCR 2 exists in a context where techniques like best-of-N sampling with verifier selection and iterative revision are well-established for other domains, and their absence is notable.

Mitigation status. Not addressed. The paper's conclusion (Section 6) mentions future work on "further develop the synthetic data pipeline to cover more complicated document types and unit tests" but does not discuss inference-time error detection or correction. The model is presented as a drop-in OCR system that produces a single best-effort output, with the understanding that downstream applications may implement their own verification (e.g., human review for critical documents). This is a reasonable design choice for many applications, but it leaves a capability gap relative to systems that could, in principle, trade additional inference compute for higher accuracy on difficult pages.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual reframing of OCR training objectives — away from matching a reference transcript character-by-character and toward satisfying a set of binary, semantically meaningful correctness criteria. This is not a paradigm shift in the sense of making prior approaches obsolete; supervised fine-tuning on reference transcripts remains effective and widely used. Rather, it is a criterion-referenced turn in how the field thinks about OCR quality: instead of asking "how similar is the output to the ground truth?", the unit test approach asks "does the output satisfy the specific properties that users actually care about?"

The magnitude of this reframing should not be overstated. The paper demonstrates a concrete training recipe that improves performance by 3.9 points on olmOCR-Bench (from an already-strong 78.5 SFT baseline to 82.4) and produces state-of-the-art results among fully open systems. This is a meaningful but incremental gain — not a leap from non-functional to functional, but from competitive to best-in-class. The larger contribution is methodological: the paper shows that binary unit tests can be scaled to serve as an RL training signal, and that doing so produces measurable improvements specifically in the areas (math formulas, tables, multi-column layouts) where edit distance is most misaligned with practical correctness. The 3.9-point overall gain understates the category-specific impact: Tables improved by 12.0 points and math within Tables by 13.0 points from the pre-RL baseline to the final model, suggesting that the approach excels precisely where the conceptual argument predicts it should.

What makes this work genuinely novel — rather than an obvious extension of olmOCR-Bench — is the inversion of the relationship between evaluation and training. In the standard ML workflow, you design an evaluation metric (e.g., edit distance), build a training objective that approximates or aligns with it (e.g., cross-entropy loss against reference tokens), and hope that optimizing the training objective also optimizes the evaluation metric. The olmOCR 2 approach collapses this distinction: the evaluation metric (unit test pass rate) is the training objective. This is only possible because of the synthetic HTML pipeline, which makes unit test generation scalable — without it, unit tests would remain an evaluation-only tool, as they were in the original olmOCR-Bench.

The work also reconciles a latent tension in the OCR literature that the paper does not fully articulate but which its design choices resolve. On one hand, continuous evaluation metrics (edit distance, BLEU, ROUGE) have well-known pathologies: they penalize valid variations, reward structurally different but character-similar outputs, and are insensitive to semantically critical errors that affect few characters. On the other hand, categorical or structured evaluation (checking specific properties) has been considered too expensive and too sparse to serve as a training signal. The paper demonstrates that this tension is not fundamental — it is a scaling problem. With a pipeline that automatically generates structured ground truth (HTML) and extracts categorical tests from it, the sparsity of binary rewards becomes manageable for modern RL algorithms (GRPO with group-relative advantage normalization, 28 completions per prompt). This does not make continuous metrics obsolete, but it establishes that binary, criterion-referenced training signals are a viable alternative when you can programmatically define what "correct" means in terms of pass/fail criteria.

The finding has implications for which research directions become more or less attractive:

More attractive: Research on programmatic ground truth generation for structured prediction tasks. The paper's HTML pipeline is specific to document OCR, but the pattern — use a strong model to generate structured representations from which testable correctness criteria can be extracted — applies to any domain where: (a) there exists a structured intermediate representation (HTML, JSON, code, logical forms), (b) correctness criteria can be expressed as testable properties of that representation, and (c) a strong model can approximately generate the representation from raw inputs. Code generation (where unit tests are already standard), data extraction from forms (where field-level validation rules can be defined), and structured information retrieval (where entity-relationship constraints can be checked) are natural candidates.

More attractive: Research on RL reward design for VLMs that moves beyond continuous, reference-based scores. The paper demonstrates that a heterogeneous reward composed of multiple binary criteria — text presence, text absence, reading order, table structure, math rendering, robustness checks — can be effectively optimized jointly. This opens the door to reward functions that explicitly encode multiple desiderata (accuracy, formatting, safety, style) rather than collapsing them into a single scalar. The key enabler is having a ground truth representation rich enough to support extracting diverse tests — HTML works for documents, but other representations would work for other modalities.

Less attractive (or at least, no longer the only game in town): The assumption that continuous, token-level supervision (cross-entropy against reference transcripts) is the necessary and sufficient training objective for OCR. The paper shows that RL with binary rewards can produce gains on top of strong SFT baselines — meaning SFT alone leaves performance on the table that a criterion-referenced reward can capture. This does not make SFT obsolete (the paper starts from an SFT model, not a raw pretrained VLM), but it suggests that SFT should be viewed as initialization rather than the complete training recipe.

Less attractive: Heavily investing in better continuous OCR evaluation metrics (e.g., improved edit distance variants) if the goal is to use them as training signals. The paper's argument is not that continuous metrics are useless for evaluation — they may still have a role in tracking progress and comparing systems — but that for training, binary criteria aligned with user-relevant properties may be more effective. The paper does not prove this experimentally (see Limitation 3 in the prior section), but it provides a coherent alternative that shifts attention from refining continuous scores to defining better binary tests.

A subtle but important implication concerns open research norms. The paper's presentation in Table 1 — which ranks systems not just by accuracy but by openness — is a deliberate intervention in how the field compares OCR systems. By achieving state-of-the-art results while releasing model weights, training data, training code, and inference code under Apache 2.0, the paper demonstrates that openness and competitiveness are compatible in a domain where several leading systems (GPT-4o, Gemini, Mistral OCR) are API-gated. This does not change the technical landscape, but it changes the social landscape: it sets a precedent that "fully open" can mean "fully competitive," which may pressure other developers to release more components and elevate openness as a dimension of evaluation alongside accuracy.

Follow-Up Research This Work Enables

1. Direct comparison of binary unit test rewards versus continuous rewards in identical RL setups. The most important missing experiment is a controlled ablation: train the same Qwen2.5-VL-7B-Instruct SFT model using the identical GRPO setup, identical synthetic data (2,186 HTML pages), identical training hyperparameters (28 completions per document, β = 0.01, one epoch), but with different reward functions — (a) binary unit test pass rate (as in the paper), (b) character-level edit distance between the model output and the HTML text content, (c) a composite continuous reward mimicking Infinity Parser's (edit distance + paragraph count difference + table structure consistency), and (d) no RL (SFT only). The key measurement would be the final olmOCR-Bench score for each variant, plus per-category breakdowns (math, tables, multi-column, etc.), plus an evaluation on an external benchmark like OmniDocBench to test whether any performance differences are specific to the unit test evaluation format. This experiment would directly test the paper's central conceptual claim — that binary unit tests are a better reward signal, not just a different one — and would either substantially strengthen the paper's contribution (if unit tests outperform) or usefully bound its significance (if any RL reward on synthetic data performs similarly).

2. Unit test density scaling: how many tests per page are needed? The paper generates 30,381 tests from 2,186 pages — an average of ~14 tests per page — and uses the pass rate fraction as the reward. This is a specific choice, not a principled optimum. A follow-up could systematically vary the number of unit tests per page (by randomly sub-sampling tests at rates of 2, 4, 8, 14, 28, and 56 tests per page) and measure: (a) RL training convergence speed, (b) final olmOCR-Bench score, and (c) per-category gains. The hypothesis would be that too few tests create a reward signal that is too sparse (a page with 2 tests gives only 3 possible reward values: 0.0, 0.5, 1.0), making optimization difficult; too many tests might create a reward signal that is overly smooth (approximating a continuous score) and loses the binary criterion-referenced property. The sweet spot — and whether there is a sharp threshold or a broad plateau — would provide practical guidance for anyone applying this approach. A negative result (no sensitivity to test density within a wide range) would also be valuable: it would mean practitioners don't need to carefully tune this parameter.

3. Cross-model and cross-benchmark transfer of the RL training recipe. All results are on Qwen2.5-VL-7B-Instruct evaluated on olmOCR-Bench. A strong follow-up would determine whether the approach generalizes by: (a) applying the identical RL training recipe (same synthetic data, same GRPO hyperparameters) to a different VLM family — LLaVA-NeXT, InternVL2, or Pixtral, all at comparable 7B-8B scales; (b) evaluating on both olmOCR-Bench and an independently developed benchmark (OmniDocBench); and (c) measuring whether the per-category improvement pattern (largest gains on math and tables) replicates across model families. A finding that the approach transfers cleanly — same hyperparameters, same relative gains — would establish it as a general method rather than a recipe tuned to Qwen's specific properties. A finding that different model families require substantially different hyperparameters (β, completions per document, learning rate) or show different gain patterns would usefully bound the method's generality and motivate model-specific tuning studies.

4. Language and script extension via multilingual synthetic HTML pipelines. The current pipeline and evaluation are English-only. A natural extension is to build a multilingual version: sample real PDFs in Chinese, Arabic, Hindi, Japanese, and Korean (each with distinct typographic conventions, text directions, and script-specific OCR challenges — CJK character segmentation, Arabic ligature handling, Devanagari conjunct rendering), use Claude Sonnet (which supports these languages) to generate HTML with appropriate lang attributes, font specifications, and text direction declarations, and extract unit tests analogous to the English ones (with Text Presence tests checking for language-appropriate phrases, and a new Script Accuracy test checking that the output doesn't mix scripts inappropriately). The key experiment would train an olmOCR-style model on this multilingual synthetic data and measure: (a) per-language performance on a multilingual OCR benchmark (or an extended olmOCR-Bench with non-English documents), (b) whether the RL gains are language-dependent (do binary unit tests help more for some scripts than others?), and (c) whether multilingual training degrades English performance (catastrophic forgetting) or produces positive transfer. This would be particularly impactful because document OCR in non-English languages is substantially less developed than English OCR, and a demonstrated method for training competitive models without large human-annotated datasets would lower the barrier to entry.

5. Combining unit test RL rewards with iterative refinement at inference time. The paper identifies the absence of self-correction as a limitation. A direct follow-up would be: after RL training with unit test rewards, can the model be further trained (or prompted) to detect its own errors and produce revised outputs? A specific experiment: take the final olmOCR-2-7B-1025 model, and for a set of evaluation documents, generate an initial OCR output, then prompt the model a second time with both the original page image and its own previous output (similar to the revision model setup in the reference example paper), asking it to identify and correct errors. Measure whether a second pass improves unit test pass rates, particularly on the categories where the model is weakest (Old scans, where it scores 47.7). If it does, this opens the door to inference-time compute scaling in OCR — trading additional generation steps for higher accuracy on difficult pages, analogous to what the reference paper studies for math reasoning. If it doesn't (the model cannot self-correct OCR errors), that would be an informative negative result: it would suggest that OCR errors are "baked in" to the model's perception of the page image in ways that revision prompts cannot override, implying that improvements must come from better training or different architectures rather than inference-time iteration.

6. Synthetic data quality auditing: what biases does Claude Sonnet introduce? The paper's pipeline robustness claim — that VLM hallucinations don't affect the reward signal — is a design property, not an empirically verified one. A follow-up study could systematically audit the synthetic HTML quality by: (a) sampling 100-200 pages from the synthetic data mix; (b) having human annotators (or a panel of different OCR systems acting as judges) compare the HTML content to the original PDF content, flagging discrepancies in text accuracy, reading order, table structure, equation formatting, and header/footer identification; (c) categorizing the error types and frequencies in Claude Sonnet's output; and (d) measuring whether those error types appear in the trained olmOCR 2 model's output on real (non-synthetic) documents. This would quantify the "inherited bias" problem discussed in Limitation 1. A finding that Claude Sonnet's HTML is highly accurate (>95% of text content matches the source PDF) would validate the robustness claim and make the approach more compelling. A finding that Claude Sonnet makes systematic errors (e.g., consistently misordering captions, misreading certain math symbols) that then appear in the trained model would reveal a concrete failure mode and motivate either quality control mechanisms (filtering out low-quality synthetic pages) or ensemble-based generation (using multiple VLMs to cross-check HTML output).

Practical Applications and Downstream Use Cases

1. Training data preparation for large language models. The original olmOCR paper was titled "Unlocking Trillions of Tokens in PDFs," and olmOCR 2 represents a concrete improvement in the quality of that unlocking. For organizations building LLM training datasets — which increasingly involve web-scale document crawling and processing — the 14.2-point improvement in overall olmOCR-Bench score over the original olmOCR means substantially cleaner text extraction from PDFs. The specific gains on tables (+12.0 from pre-RL baseline to final model), math formulas (+13.0 within the Tables category), and multi-column layouts (+6.4) are particularly relevant: these are the document elements where OCR errors most corrupt the training signal. An LLM trained on text where table cells are scrambled learns incorrect factual associations; an LLM trained on text where paragraphs are incorrectly ordered learns corrupted discourse patterns. The fully open release (Apache 2.0, all components downloadable) means this quality improvement is available to any training data pipeline without API costs or usage restrictions — a practical advantage over Mistral OCR API (72.0 on olmOCR-Bench) or GPT-4o (68.9).

2. Scientific literature digitization and search. Academic search engines and literature review tools — Semantic Scholar, Google Scholar, institutional repositories — process millions of scientific PDFs, many containing dense math, tables, and multi-column layouts. olmOCR 2's category-specific strengths map directly onto these needs: ArXiv (83.0, competitive with the best systems), Tables (84.9), and Old scans math (82.3) are all strong. The old scans sub-category remains weak (47.7), but for current academic PDFs (which are born-digital, not scanned), the model is highly capable. A concrete deployment scenario: a university library digitizing its thesis collection (hundreds of thousands of documents, many with equations and tables) could run olmOCR 2 on a single H100 node (or multiple A100 nodes) with no per-page API costs, achieving accuracy comparable to or better than commercial API solutions (Mistral OCR API at 72.0, GPT-4o at 68.9) at a fraction of the cost, while maintaining full control over data privacy and processing pipeline.

3. Legal and financial document processing with targeted accuracy requirements. Legal discovery, contract analysis, and financial statement processing have asymmetric error costs: misreading a dollar amount, a date, or a party name is a critical failure, while minor formatting variations in paragraph text are often acceptable. The unit test framework's decomposability is directly useful here: an organization could define domain-specific unit tests (e.g., "all monetary amounts in this table are correctly extracted," "no clause numbers are omitted") and train or fine-tune olmOCR 2 using the same RLVR recipe on synthetic legal or financial HTML documents. The paper's demonstrated pipeline — sample real documents from the target domain, generate HTML with Claude Sonnet, extract unit tests, run GRPO — is domain-agnostic. The $0.12 per page cost for HTML generation means that creating a domain-specific training set of 2,000 pages would cost roughly$ 240 in API calls, plus the GPU compute for RL training, which is economically viable for organizations that process documents at scale and need high accuracy on specific structured elements.

4. On-device or air-gapped OCR for sensitive documents. The 7B parameter scale and Apache 2.0 license enable deployment scenarios where documents cannot leave a secure environment due to privacy, classification, or regulatory requirements. Government agencies processing classified documents, hospitals digitizing patient records under HIPAA, and law firms handling privileged client communications all need OCR that runs locally without sending data to external APIs. olmOCR 2's competitive accuracy (82.4 on olmOCR-Bench, compared to 72.0 for Mistral OCR API and 68.9 for GPT-4o — both of which require sending documents to external servers) means that local deployment does not require a substantial accuracy sacrifice. The model runs on a single 8×H100 node for training but a single GPU for inference, making deployment feasible on institutional hardware. The blank page handling fix (step 5 in Table 3) specifically improves reliability on real-world document collections that include empty pages — a practical concern for batch processing of scanned archives that researchers would otherwise need to manually filter.