DocReward: A Document Reward Model for Structuring and Stylizing

🎯 Pitch

DOCREWARD outperforms GPT-5 by 14.6 points in evaluating document layout, judging structure and style while ignoring identical textual content. Trained on 117K content-identical pairs, it guides agents to produce professionally formatted documents, making visually amateurish AI outputs a solvable problem.

1. Executive Summary

This paper introduces DOCREWARD, a Document Reward Model specialized in assessing the structural and stylistic professionalism of documents — evaluating formatting elements such as white space, margins, font choices, heading consistency, and alignment — while remaining agnostic to textual content quality through a textual-quality-agnostic framework that evaluates documents sharing identical text but differing in layout. Trained via Bradley-Terry preference optimization on DOCPAIR, a constructed dataset of 117K document pairs spanning 32 domains and 267 types, DOCREWARD outperforms GPT-5 by 14.6 percentage points on a human-annotated benchmark of 1,443 pairs (achieving 82.3% accuracy versus GPT-5's 67.7% in the same pointwise setting). Reinforcement learning experiments further demonstrate that DOCREWARD serves as an effective reward model for aligning both open-source (Qwen2.5-Coder) and closed-source (GPT-4o) document generation agents toward producing documents with consistently higher structural and stylistic professionalism, establishing its utility as a plug-in reward signal for agentic workflows that generate professional formatting.

2. Context and Motivation

The Core Problem: We Can Generate Document Content, But Not Professional Formatting

The fundamental gap this paper addresses is deceptively straightforward: current AI systems can write the words, but they cannot design the document. Agentic workflows — multi-step AI pipelines that automate complex tasks — have made remarkable progress in generating textual content for professional documents. Systems can now produce deep research reports, technical documentation, and formal business writing with impressive textual coherence. However, the visual presentation of these documents — their structure and style — remains neglected, producing outputs that are technically correct in content but visually amateurish in execution.

This gap matters for a simple reason that anyone who has received an unformatted wall of text knows: readability depends as much on presentation as on content. A well-organized structure with clear headings, appropriate white space, and logical section breaks enables readers to navigate and comprehend information efficiently. Consistent styling — proper font choices, heading hierarchy, emphasis techniques, bullet points, and numbering — makes content engaging and scannable rather than intimidating. The paper positions these as co-equal pillars of document professionalism alongside textual quality, not as secondary aesthetic concerns.

Why This Gap Exists: The Missing Reward Signal

The paper identifies a specific technical bottleneck: we lack effective reward models capable of evaluating structural and stylistic professionalism. Reward models are the component in modern AI pipelines that provide feedback signals — they judge whether an output is good, enabling both selection (picking the best among multiple candidates) and training (reinforcement learning to improve the generator). For textual content, large language models like GPT-4o and GPT-5 serve as reasonably effective reward models because they can read text and assess its factual accuracy, logical coherence, and writing quality.

But structure and style live in the visual domain — they are properties of the rendered document as seen by a human reader, not properties of the underlying text string. A paragraph that reads identically as plain text can appear professional with proper margins and readable fonts, or amateurish with cramped spacing and inconsistent styling. Current LLMs, even multimodal ones, have not been trained to distinguish these visual quality differences reliably. As the paper demonstrates empirically (Table 2), GPT-5 achieves only 67.7% accuracy on a pairwise document comparison task that DOCREWARD handles at 82.3% — a 14.6-point gap that represents the difference between guessing correctly two-thirds of the time versus more than four-fifths of the time.

Two Requirements That Make This Hard

The paper articulates two non-trivial requirements that any effective document reward model must satisfy, and explains why they have not been jointly achieved before:

1. Comprehensiveness. The reward model must work across diverse document types, structures, styles, and quality levels. A reward model that evaluates only government forms or only academic papers is not practically useful — real-world document generation spans reports, manuals, proposals, meeting minutes, press releases, syllabi, application forms, and hundreds of other specific types (the paper documents 267 distinct types across 32 domains in its training corpus). Building a single model that generalizes across this diversity requires training data with comparable breadth, which did not exist before this work.

2. Textual-quality-agnosticism. This is the more subtle and critical requirement. The reward model must evaluate structure and style independently of whether the textual content is well-written. If you give a reward model two documents that differ in both writing quality and formatting, it will conflate the two — preferring the document with better writing even if you want to optimize for layout. To isolate the structure/style signal, you need document pairs that share identical textual content but differ in formatting. Constructing such pairs at scale — thousands of them across hundreds of document types — is the key engineering challenge the paper tackles.

The paper is careful to clarify what "textual-quality-agnostic" means. It does not mean the model ignores text entirely — the model still reads and understands content to judge whether the structure and style are contextually appropriate (e.g., whether headings are used correctly for the document's logical organization). Rather, it means the model's judgment is not confounded by how well the text itself is written. A document with awkward prose but excellent formatting should score higher than one with eloquent prose but terrible formatting, when evaluated purely on structure and style.

Where Existing Approaches Fall Short

The paper situates its contribution against three strands of related work, each of which partially addresses the problem but leaves the core gap open:

Aesthetic and professionalism assessment research (Section 5) has developed models for evaluating visual quality, but primarily in domains other than multi-page documents. AesthetiQ trains multimodal LLMs to assess graphic layouts using preference optimization; LACE uses diffusion models with differentiable aesthetic constraints for layout generation; Calista evaluates website visual appeal using explicit ratings and pairwise comparisons; A-Lamp and related work assess photo aesthetics using layout-aware convolutional networks. These approaches demonstrate that learned models can capture visual quality signals, but they focus on images, single-page graphic designs, or web interfaces — not multi-page professional documents where professionalism depends on the interplay of structure (section organization, white space allocation, margins, headers/footers) and style (font consistency, heading hierarchy, emphasis conventions) across potentially many pages.

Document AI research has addressed document understanding — LayoutLM, ReLayout, and OCR-based pipelines identify headings, tables, and semantic elements within documents for information extraction and classification tasks. Recent work has even explored automatic document and layout generation. However, as the paper notes, evaluation in these systems has "primarily been limited to content correctness or basic formatting." The assessment of whether a generated document looks professional — whether the visual presentation meets professional standards — remains "largely unexplored." Document AI can parse a document; it cannot judge whether that document is well-designed.

Preference learning and reward models (RLHF, DPO) provide the training methodology that makes DOCREWARD possible, but prior work has applied these techniques to alignment tasks (helpfulness, harmlessness, summarization quality) rather than document professionalism. The Bradley-Terry preference optimization loss that DOCREWARD uses is standard in the RLHF literature, but the paper adapts it to a novel domain with a novel data construction pipeline.

A Confounding Factor: Content and Formatting Are Entangled in Natural Data

A deeper motivation the paper implicitly addresses: you cannot simply scrape the internet for document pairs where only the formatting differs. In naturally occurring documents, content quality and formatting quality are strongly correlated — well-written documents tend to be well-formatted, and poorly written documents tend to be poorly formatted. If you train a reward model on natural document pairs (e.g., a professionally formatted government report versus a hastily written student essay), the model will learn to prefer the government report, but it will learn a mixture of signals: better writing, better organization, better formatting, more authoritative tone. It will not learn to isolate formatting quality from content quality.

The paper's solution — constructing artificial document pairs that share identical text but differ in formatting — is the enabling innovation that makes textual-quality-agnostic training possible. The three-phase data construction pipeline (curate → expand → rank) described in Section 3.1 is not just a data collection exercise; it is a carefully designed procedure for disentangling structure/style from content, and understanding why this disentanglement is necessary is essential to appreciating the paper's contribution.

The Practical Stakes: Agentic Document Generation Without a Quality Filter

The paper motivates the problem through a specific use case that has become increasingly relevant: agentic workflows for document generation. Systems like OpenAI's Deep Research and various technical documentation generators can produce multi-page reports automatically. But these systems face a quality control problem — they can generate many candidate documents, but they cannot reliably determine which one looks most professional. A human reviewer could make this judgment, but the point of automation is to reduce human involvement.

Without a reliable reward model for structure and style, agentic document generation faces two specific limitations:

Best-of-N selection is blind to formatting. An agent can generate 8 candidate documents from the same content, but picking the best one requires a quality signal. Using GPT-5 as the judge (as the paper's baseline does) yields only 67.7% alignment with human preferences on structure/style — meaning roughly one-third of the time, the "best" document selected by GPT-5 is not the one a human would prefer.
Reinforcement learning cannot optimize for formatting. RL-based training of document generation agents requires a reward function. Without a reward model that captures structural and stylistic quality, the agent's optimization signal is limited to basic rule-based checks (did the code execute? does the output text match the input?) that do not distinguish professional formatting from amateur formatting.

The paper's extrinsic evaluations (Best-of-N in Section 4.3 and RL experiments) directly address these practical gaps, demonstrating that DOCREWARD provides the missing quality signal.

How This Paper Positions Itself

The paper's positioning is clear and specific: it does not claim to solve document generation, document understanding, or content quality assessment. It claims to solve a narrow but critical subproblem — evaluating the structural and stylistic professionalism of documents in a way that is not confounded by content quality — and to demonstrate that this capability is sufficient to significantly improve document generation when plugged into existing agentic workflows.

The relationship to prior work is additive rather than competitive. The paper builds on:

Multimodal vision-language models (Qwen2.5-VL) as the architectural backbone
Preference optimization (Bradley-Terry loss) as the training objective
Agentic document generation approaches as the downstream application

What is new is:

The textual-quality-agnostic framework that formalizes the training objective (Equation 1) — maximizing rank correlation with true preferences under the constraint of identical textual content
The DOCPAIR dataset constructed through a three-phase pipeline (curation, agent-based expansion, oracle-anchored ranking) that operationalizes this framework at scale (117K pairs across 32 domains and 267 types)
The empirical demonstration that a specialized 3B/7B parameter model trained this way substantially outperforms GPT-5 (a model likely orders of magnitude larger) on the specific task of structure/style assessment
The end-to-end validation that this reward model is practically useful for both selection-based and RL-based improvement of document generation agents

The paper's framing in Section 2 (Equation 1) casts this as a constrained optimization problem — learn a scoring function that orders documents correctly with respect to structure/style, under the constraint that all compared documents share the same text — and this formalization distinguishes it from prior work that either ignored the content-formatting entanglement or operated in different domains (images, UI, single pages) where the constraint is less relevant.

3. Technical Approach

3.1 Reader Orientation

DOCREWARD is a learned scoring function (a "reward model") that takes rendered images of a document as input and outputs a single number reflecting how professional the document's structure and style are. The paper tackles the problem that current AI systems can generate document text but cannot evaluate — and therefore cannot optimize for — whether the resulting document looks professional in terms of formatting, layout, spacing, font choice, and other visual elements that determine readability. The solution has three parts: (1) a framework that defines precisely what it means to evaluate structure/style independently of content quality, (2) a large-scale dataset of paired documents constructed specifically to train models under this framework, and (3) a model trained via preference optimization to order documents by their structural and stylistic professionalism.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components, connected in a pipeline that produces a trained reward model:

Source Document Corpus — A curated collection of high-quality, human-authored documents (government reports, academic papers, business forms, etc.) spanning 32 domains and 267 document types. These serve as the "ground truth" examples of professional formatting.
Document Expansion Pipeline — A set of LLM-powered agents that take a source document and produce multiple synthetic variants: documents that preserve the exact same textual content but explore different structural and stylistic choices. One agent generates documents from scratch given only the plain text; another refines synthetic documents by comparing them against the human-authored original.
Ranking Procedure — A cost-effective labeling strategy that assigns relative quality judgments to pairs of documents sharing the same text. For human-authored versus synthetic pairs, the human version is designated the winner. For synthetic-versus-synthetic pairs, an oracle-anchored method uses a large closed-source LLM (GPT-5) with the human-authored document as a reference to determine which synthetic variant is closer in quality.
DOCREWARD Model — A Qwen2.5-VL vision-language model with a regression head that ingests rendered document pages and outputs a scalar professionalism score. It is trained with the Bradley-Terry pairwise preference loss to assign higher scores to documents ranked as more professional.

Information flows as follows: human-authored documents → text extraction and agent-based regeneration → grouping of documents sharing identical text → pairwise ranking using heuristics and oracle anchoring → Bradley-Terry preference training → a model that scores any rendered document on structure/style professionalism.

3.3 Roadmap for the Deep Dive

First, the textual-quality-agnostic framework (Equation 1), which defines the mathematical objective and the constraint that makes the problem well-posed — all compared documents must share identical text so the model cannot cheat by evaluating content quality.
Second, the three-phase DOCPAIR data construction pipeline (curation → expansion → ranking), since the dataset is the primary enabler — no prior dataset disentangled content from formatting at this scale and diversity.
Third, the model architecture and training objective (Bradley-Terry loss), which form the learning machinery that converts the ranked pairs into a calibrated scoring function.
Fourth, how the trained model is deployed in downstream applications — the Best-of-N selection protocol and the reinforcement learning reward formulation — since the paper's contribution is validated through these extrinsic use cases.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset engineering paper with a systems-motivated model training component: its core idea is that a reward model for document formatting must be trained on data where textual content is held constant while formatting varies, and the paper's primary contribution is the pipeline for constructing such data at scale, along with the demonstration that the resulting model is practically useful.

The Textual-Quality-Agnostic Framework (Section 2)

The framework formalizes the training objective as a constrained optimization problem. The central constraint — that all compared documents share the same textual content — is what distinguishes this work from generic preference learning.

Let $\{D_i\}_{i=1}^N$ denote a set of $N$ documents, where each document $D_i$ consists of two components: its textual content $D_{\text{text},i}$ (the words, stripped of all formatting) and its rendered images $D_{\text{img},i}$ (what the document actually looks like as pages). The reward model $R_\theta$ is a function parameterized by weights $\theta$ that takes rendered document images as input and produces a scalar score:

$R_\theta: D_{\text{img}} \rightarrow \mathbb{R}$

The framework defines the optimization objective as:

$\max_\theta \ \text{Sim}\left(\pi^*, \ \text{Argsort}(R_\theta(D_{\text{img},1}), \ldots, R_\theta(D_{\text{img},N}))\right) \quad \text{s.t.} \ D_{\text{text},i} = D_{\text{text},j}, \ \forall i,j$

Where $\text{Sim}$ is a predefined similarity function that measures agreement between two orderings, $\text{Argsort}$ returns the indices of documents sorted by their predicted scores from the reward model, and $\pi^*$ represents the ground-truth ranking — the true ordering of these documents by structural and stylistic professionalism.

What this equation computes: given a set of documents that all contain exactly the same words (the constraint), the reward model assigns a score to each document's visual rendering. The $\text{Argsort}$ function converts these scores into a predicted ranking (highest score first, lowest score last). The $\text{Sim}$ function measures how closely this predicted ranking matches the true ranking $\pi^*$ . Training maximizes this similarity — the model learns to produce scores whose induced ordering matches human judgments of formatting quality.

Why this form: the constraint $D_{\text{text},i} = D_{\text{text},j}$ is the critical design element. Without it, the model could achieve high similarity by evaluating writing quality rather than formatting — if better-written documents tend to be better-formatted in the training data, a model that scores based on prose style would accidentally correlate with formatting quality without actually learning to assess visual structure. By enforcing identical text across all compared documents, the framework isolates the signal: any difference in predicted score must come from differences in rendered appearance, since the underlying words are identical. This transforms the learning problem from "what makes a good document overall?" to "given fixed content, what makes the presentation professional?"

The paper emphasizes a nuance: this is a content-quality-independent framework, not a content-agnostic one. The model still reads and processes the textual content — it needs to understand what the document says to judge whether the formatting is contextually appropriate. For instance, it must recognize that a 40-point bold centered line is appropriate as a document title but inappropriate as body text. The independence is about the evaluation signal, not the input processing: the model's score should not be higher simply because the prose is eloquent, even though the model can read and comprehend that prose.

Phase 1: Curating High-Quality Professional Documents (Section 3.1)

The data pipeline begins with sourcing real-world documents that exemplify professional formatting. This phase establishes the "ground truth" of what professional structure and style look like.

Source corpora. The paper draws from two complementary sources to ensure diversity:

Government corpora: GovDocs is a corpus of approximately 1 million documents obtained from government websites using web crawling, covering reports, forms, guidelines, and other institutional materials. NapierOne is a modern mixed-file dataset designed as an alternative to GovDocs, providing additional diversity. Government documents are valuable because they typically follow strict formatting standards — they are templates for what "professional" means in formal institutional contexts.
Web document corpus: Documents from CommonCrawl, a massive web crawl dataset, spanning business, education, law, healthcare, and other professional domains. These provide everyday professional communication examples that complement the formality of government documents.

Preprocessing and quality filtering. Raw web documents are noisy. The paper applies a multi-stage filtering pipeline to retain only high-quality professional samples. First, all files are converted to .docx format to enable programmatic access and modification via the python-docx library. This is a practical engineering choice — .docx is an XML-based format that can be parsed, analyzed, and generated systematically, unlike PDF which represents documents as fixed-layout primitives.

Second, extreme cases are discarded based on quantitative heuristics:

Documents exceeding 20 pages are removed (too long to reasonably render and annotate)
Files larger than 1 MB that are dominated by images are removed (image-heavy documents likely reflect content where text extraction is unreliable)
Files smaller than 10 KB with trivial content are removed (these likely contain too little text to meaningfully evaluate structure)

Third, the paper employs GPT-5 as an automated quality filter. GPT-5 is prompted to assess each document's structure and style on a [0, 10] scale, with documents scoring above 8 being retained. This is a pragmatic use of a large model's general visual judgment — GPT-5 is not specialized for document formatting, but it can roughly distinguish professional layouts from amateurish ones at a level sufficient for filtering. A manual inspection of 200 randomly sampled retained documents confirmed that this automated filter preserves high-quality professional samples, validating its reliability as a coarse filter.

The resulting corpus spans 32 domains and 267 document types, with the top domains being government (32.2%), education (28.6%), non-profit (9.6%), medical (5.7%), and scientific (5.0%), as shown in Figure 3. This breadth is essential for training a reward model that generalizes — the model must learn universal principles of professional formatting (appropriate white space, consistent fonts, clear hierarchy) rather than domain-specific conventions, and exposing it to documents from government, education, business, legal, medical, and technical domains forces it to extract the common elements that define professionalism across contexts.

Phase 2: Expanding Source Documents via Agents (Section 3.1)

This is the phase where the textual-quality-agnostic constraint is operationalized. Starting from a high-quality source document, the goal is to produce multiple variant documents that preserve the exact same textual content but differ in structure and style.

Text extraction. The first step is to strip all formatting from the source document, extracting only the plain text content. This discards font specifications, spacing, margins, alignment, heading styles, page breaks, headers, footers — everything that constitutes structure and style. What remains is a string of words that encodes what the document says but not how it looks. This text serves as the shared input to all downstream agents, ensuring the content-is-fixed constraint from Equation 1.

Agent Type 1: Textual Content to Document. This agent takes the extracted plain text and generates a completely new .docx document from scratch using python-docx. The agent is an LLM (GPT-4o, OpenAI o1, Claude Sonnet 4, or GPT-5) prompted to:

Analyze the text content to infer document structure (what should be a heading? what should be body text? where should lists or tables appear?)
Create a new DOCX document using appropriate styles and formatting
Apply visual hierarchy and professional appearance
Preserve all text content exactly — no omissions, modifications, or additions

The generated output is Python code that, when executed, produces a .docx file. The prompt (reproduced in Appendix A.8) is detailed and explicit about requirements: the code must be complete and executable, must not use placeholders or omit sections, must include document.save() to the specified output path, and must follow a specific code structure template with standardized imports and error handling.

The key insight: different LLMs, and even the same LLM with different sampling, will make different design choices about how to format the same text. One run might choose 12-point Times New Roman with 1-inch margins and numbered headings; another might choose 11-point Calibri with 1.15 line spacing and bullet-style headings. Both preserve the same words, but they produce visually different documents. This variation is exactly what the training framework needs — it creates pairs where content is fixed but formatting varies, enabling the model to learn which formatting choices are more professional.

Agent Type 2: Refinement for Better Structure and Style. This agent takes a previously synthesized document and improves it by comparison with the original human-authored source document. The refinement process is a two-stage procedure:

Stage 1 — Plan generation: The agent receives three inputs: the python-docx code that generated the synthetic document, rendered screenshots of the synthetic document, and rendered screenshots of the original human-authored document along with a structured textual representation of that document. The agent is prompted to identify the five most important differences between the synthetic and original documents, focusing on specific, actionable formatting changes. For each difference, it must specify: the location in the document, the current state, the target state, and the exact implementation (e.g., "set run.font.size = Pt(14)" rather than "make the font bigger"). The output is a structured refinement plan.

Stage 2 — Code generation: The agent receives the original synthetic document's code and the refinement plan from Stage 1, and generates improved Python code that implements all the specified improvements. The code must be a complete, standalone script — not just modifications to the existing code, but a full regeneration that applies the refinement while preserving all text content.

The refinement agent is important because the "Textual Content to Document" agent generates documents from scratch without any visual reference for what good formatting looks like for that specific document type. The refinement agent introduces a learning signal: it can see the gap between what was generated and what the original looks like, and it can make targeted improvements. This produces synthetic documents that are closer in quality to the human originals, which is valuable because the training data needs a range of quality levels — some synthetic documents will be poor (providing negative examples), while others refined through this process will be closer to professional (providing harder discrimination examples).

Filtering to ensure content preservation. After document generation, a rigorous filtering step verifies that the synthetic documents truly preserve the original textual content. Using python-docx, text is extracted from both the original and synthetic documents, and their word counts are compared. Only documents where the word count difference is no more than 20 words from the original and the ROUGE-L score (a measure of longest common subsequence overlap) exceeds 0.95 are retained. This dual threshold ensures that content is comparable while allowing for minor differences that arise from formatting choices (e.g., whether "Section 1" in the original becomes "1." in the synthetic — same meaning, slightly different token sequence). Pairs that fail this check are discarded, since they violate the textual-quality-agnostic constraint.

Scale and diversity. The expansion phase produces, for each source document, multiple synthetic variants — documents generated by different base LLMs (GPT-4o, o1, Claude Sonnet 4, GPT-5) using the "Textual Content to Document" agent, and refined variants produced by the refinement agent. These are grouped by shared textual content, creating sets of documents that vary only in structure and style. The final DOCPAIR dataset contains 69,137 source documents that were expanded into 117,108 document pairs (each pair consisting of two documents with identical text but different formatting), with an average document length of 3.2 pages. The paper does not provide a detailed breakdown of how many documents each agent type produced, but notes in Appendix A.3 that "both GPT-4o and GPT-5 serve as the base models of agents" for the training dataset construction, and in Section 4.1 mentions that the evaluation benchmark includes documents from four different Textual Content to Document agents (using GPT-4o, o1, Claude Sonnet 4, and GPT-5) plus refinement outputs and human-authored originals.

Phase 3: Ranking Documents (Section 3.1)

Given a set of documents sharing identical text, the final phase assigns a relative ranking $\pi^*$ — an ordering from most to least professional in structure and style. This ranking provides the supervision signal for preference-based training.

The paper handles two distinct cases, since different strategies are appropriate depending on whether a document pair includes the human-authored original.

Case 1: Real vs. Synthetic. When comparing an original human-authored document with an agent-generated synthetic variant, the human-authored version is designated as the winner (preferred). This heuristic is grounded in a validation check: human inspection of 100 randomly sampled pairs confirmed that current state-of-the-art models (GPT-5, Claude Sonnet 4) produce documents with structural and stylistic quality inferior to the high-quality human-authored documents that survived the rigorous filtering pipeline in Phase 1. In other words, even the best LLM-generated formatting is not yet as professional as well-crafted human formatting, at least for the curated corpus used in this work. This makes the real-versus-synthetic case straightforward — the real document always wins, providing a strong and reliable training signal.

Case 2: Synthetic vs. Synthetic. When comparing two agent-generated synthetic documents, there is no ground-truth "correct" formatting — both are artificial constructions. Manual annotation of thousands of such pairs would be prohibitively expensive. The paper therefore adopts an oracle-based annotation method that leverages a closed-source LLM (GPT-5) as a proxy for human judgment, but with a critical design choice that reduces the LLM's inherent biases.

The standard approach would be to show GPT-5 two synthetic documents and ask "which is more professional?" However, this is subjective — GPT-5's preferences may reflect its own idiosyncratic training biases rather than genuine formatting quality. The paper's innovation is to transform this from a subjective preference task into an objective similarity matching task by providing a reference anchor.

Specifically, the LLM is presented with a document triplet: {D_real, D_synth1, D_synth2}, where D_real is the human-authored original. The prompt (reproduced in Appendix A.8) asks the model to compare D_synth1 and D_synth2, "taking the Original document as reference for quality standards." The model does not judge which synthetic document it "prefers" — it judges which synthetic document is more similar in professionalism to the human-authored reference. This anchoring transforms the task: rather than relying on the LLM's internal aesthetic sense, the model simply needs to recognize which of two candidates more closely matches the formatting exemplar.

The paper validated this oracle-anchored approach on 120 pairs where human experts also provided judgments, achieving 92.5% alignment between the oracle method and human experts. This high agreement justifies using the oracle method for the large-scale annotation of synthetic-versus-synthetic pairs, making the data construction pipeline scalable without sacrificing annotation quality.

Converting rankings to pairwise preferences. The ranking labels $\pi^*$ define a total order over a group of documents sharing the same text. For training, these total orders are decomposed into pairwise comparisons: for any two documents where $D_i$ is ranked higher than $D_j$ in $\pi^*$ , a training pair $(D_{\text{img},i}^w, D_{\text{img},j}^l)$ is created, labeling $D_i$ as the winner ( $w$ ) and $D_j$ as the loser ( $l$ ). This decomposition into binary preferences is necessary because the Bradley-Terry training objective (Equation 2) operates on pairs, not rankings.

Data statistics. The final DOCPAIR training set comprises 117,108 pairs drawn from 69,137 unique source documents spanning 32 domains and 267 document types, with an average document length of 3.2 pages. This scale — over 100K training pairs across hundreds of document types — is what enables the model to learn generalizable principles of professional formatting rather than overfitting to specific document templates.

Model Architecture and Training (Section 3.2)

Base architecture. DOCREWARD is built on Qwen2.5-VL, a multimodal vision-language model that can process both images and text. The choice of a vision-language model (rather than a text-only model that would receive formatting as structural markup) reflects the nature of the task: structural and stylistic professionalism is inherently visual. A human judges a document by looking at it, not by reading its XML markup. Using rendered document pages as input ensures the model learns from the same visual signals that a human reader perceives — font sizes, white space distribution, alignment, visual hierarchy, and the overall visual balance of the page.

A regression head is added to the base Qwen2.5-VL model — this is a small additional neural network layer that maps the model's final hidden state to a single scalar value, the professionalism score. The regression head is the only component trained from scratch; the base vision-language model provides the visual understanding capabilities that have been pretrained on large-scale multimodal data.

Input processing. Documents are rendered as images and fed into the model. The maximum input resolution is set to 300,000 pixels (not the image size in pixels, but the total pixel budget for all pages combined — the model can process multi-page documents within this budget). The maximum context length is 16,000 tokens, which includes both the visual tokens from the rendered pages and any text tokens from the prompt. Multi-page documents are processed by including rendered pages sequentially within the input.

An ablation study in Appendix A.7 investigates whether providing additional OCR-derived information (text content of spans and their bounding box coordinates) improves performance. The results show that image-only input performs better than image + OCR text and bounding boxes (85.00% vs. 80.30% for the 3B model; 87.94% vs. 84.41% for the 7B model, on a subset of the test set). This finding supports the visual-only design: the model learns better from raw rendered pages than from structured representations of text positions, perhaps because raw images preserve subtle visual cues (font rendering quality, color, spacing nuances) that discretized bounding box coordinates cannot capture.

Training objective. The model is trained using the Bradley-Terry (BT) loss, a standard preference optimization objective from the RLHF literature. Given a preference pair consisting of a winner document $D_{\text{img}}^w$ and a loser document $D_{\text{img}}^l$ (where the winner has higher structural and stylistic professionalism according to the ranking $\pi^*$ ), the loss function is:

$\min_\theta \ -\log \sigma\left(R_\theta(D_{\text{img}}^w) - R_\theta(D_{\text{img}}^l)\right)$

where $\sigma(x) = \frac{1}{1 + e^{-x}}$ is the sigmoid function, and $R_\theta(D_{\text{img}})$ is the scalar score assigned by the reward model to document $D_{\text{img}}$ .

What this equation computes: the model produces a score for the winner and for the loser. The difference $R_\theta(D_{\text{img}}^w) - R_\theta(D_{\text{img}}^l)$ is the model's predicted score gap — if the winner truly deserves a higher score, this difference should be large and positive. The sigmoid converts this difference into a probability: $\sigma(\Delta)$ is the model's estimated probability that the winner is indeed preferred. The negative log of this probability is minimized — when the model correctly assigns a much higher score to the winner, the sigmoid approaches 1, the log approaches 0, and the loss is small. When the model incorrectly assigns a higher score to the loser (or assigns similar scores), the loss is large.

Why this form: Bradley-Terry models preferences as arising from an underlying latent quality score: the probability that item A is preferred over item B is proportional to $\exp(\text{score}_A) / (\exp(\text{score}_A) + \exp(\text{score}_B))$ , which is equivalent to $\sigma(\text{score}_A - \text{score}_B)$ . The negative log-likelihood of this probabilistic model is exactly the BT loss. This formulation has two desirable properties. First, it only depends on score differences, not absolute score magnitudes — the model can learn relative ordering without needing calibrated absolute scores. Second, it handles uncertainty gracefully: when the score difference is small, the sigmoid is near 0.5 and the loss is $-\log(0.5) \approx 0.69$ , which is bounded; when the model confidently reverses the preference order (large negative score difference), the loss grows rapidly toward infinity, strongly penalizing such errors.

Training hyperparameters. The paper uses the following configuration:

Optimizer: AdamW
Learning rate: 1 × 10⁻⁶ (notably lower than the 3 × 10⁻⁵ typical for LLM fine-tuning, since the regression head is being trained and the base model's visual features should remain stable)
Batch size: 256 (pairs per training step)
Training epochs: 3
Hardware: 8 NVIDIA A100 GPUs (80GB each)
Training code: based on LLaMA-Factory, an open-source fine-tuning framework
Maximum input pixels: 300,000
Maximum context length: 16,000 tokens

Model variants. The paper trains two sizes: DOCREWARD-3B (based on Qwen2.5-VL-3B) and DOCREWARD-7B (based on Qwen2.5-VL-7B). Both use the same training procedure and data, differing only in the base model capacity. The 3B model serves as a lower-cost demonstration, while the 7B model represents the primary contribution.

Deployment as a Reward Model (Section 4.3)

Once trained, DOCREWARD is used as a plug-in reward signal in two modes that demonstrate its practical utility.

Best-of-N selection. In this mode (illustrated in Figure 4, second panel), a document generation agent produces $N$ candidate documents from the same textual content. DOCREWARD scores each candidate independently, and the document with the highest score is selected as the output. The paper's Best-of-N experiment (Section 4.3) uses $N = 8$ and compares DOCREWARD against two baselines: random selection and GPT-5-based selection. Human annotators then rank the selected documents from each reward model, and win/lose/tie rates are computed.

This mode is the simplest form of integration — no model training or modification is needed, only the insertion of DOCREWARD as a re-ranking step after candidate generation. It works whenever the agent can produce multiple outputs, which is feasible in practice through sampling with non-zero temperature.

Reinforcement learning reward. In this mode (illustrated in Figure 4, third panel), DOCREWARD provides the reward signal that trains a document generation model to produce more professional formatting. The generation model takes plain text as input and produces Python code (using python-docx) that renders a formatted document. The total reward combines two components:

$R = R_{\text{rule}} + \alpha \cdot \mathbb{I}_{\text{rule}} \cdot \sigma(R_{\text{DOCREWARD}})$

Where $R_{\text{rule}}$ is a rule-based reward that handles execution correctness: if the generated Python code executes successfully, $R_{\text{rule}} = \text{ROUGE}(doc_{\text{ori}}, doc_{\text{gen}})$ — the ROUGE similarity between the original plain text and the generated document's text content (measuring whether the content was preserved); if execution fails, $R_{\text{rule}} = 0$ . $R_{\text{DOCREWARD}}$ is the raw output of the DOCREWARD model. $\sigma(\cdot)$ is the sigmoid function mapping this raw score to $(0, 1)$ . $\mathbb{I}_{\text{rule}}$ is an indicator that is 1 when $R_{\text{rule}}$ exceeds a threshold (set to 0.8), and 0 otherwise. $\alpha$ is a balancing hyperparameter (set to 1).

What this reward formulation does: the rule-based component $R_{\text{rule}}$ ensures the model first learns to produce valid, content-accurate documents — if the code doesn't run or the text is garbled, the reward is zero regardless of formatting. The DOCREWARD component $\sigma(R_{\text{DOCREWARD}})$ provides a formatting quality bonus that is only active (via the indicator $\mathbb{I}_{\text{rule}}$ ) when basic correctness is achieved. This staged rewarding prevents the model from optimizing for formatting at the expense of content accuracy — it must first pass the execution-and-content barrier before earning formatting bonuses.

The sigmoid $\sigma$ regularizes DOCREWARD's raw score to $(0, 1)$ , preventing its magnitude from dominating $R_{\text{rule}}$ (which is already in $[0, 1]$ since ROUGE is bounded). The hyperparameter $\alpha = 1$ gives equal weight to content preservation and formatting quality in the combined reward.

RL algorithms. For open-source models (Qwen2.5-Coder-1.5B), the paper uses GRPO (Group Relative Policy Optimization) — a reinforcement learning algorithm that optimizes a policy (the document generation model) using relative comparisons among groups of sampled outputs, rather than requiring an absolute value function. For closed-source models (GPT-4o) where parameter updates are not possible, the paper uses training-free GRPO — a lightweight adaptation that adjusts generation behavior without modifying model weights, presumably through prompt optimization or output filtering guided by the reward signal.

The RL experiments (Table 4 and Figure 5) demonstrate that adding DOCREWARD to the rule-based reward consistently improves document generation quality across both open- and closed-source models, as measured by success rate (did the code execute?), ROUGE-L (did it preserve content?), DOCREWARD score (did it produce professional formatting?), and human ranking (overall preference).

Evaluation Protocols

The paper creates a separate human-annotated benchmark DOCPAIRBENCH for evaluation, distinct from the training data. This benchmark is constructed from documents that were set aside during curation (Section 4.1). It comprises 1,443 comparison pairs, produced from six types of document origins: four from different Textual Content to Document agents (using GPT-4o, o1, Claude Sonnet 4, and GPT-5 as the base LLM), one from the Refinement agent (GPT-5 refining synthetic documents), and one from the curated human-authored originals.

Three human annotators (Ph.D. students with expertise in computer science/math, marketing, and design) independently ranked documents within each group based on the professionalism criteria defined in the annotation guidelines (Figure 8). Inter-annotator reliability reached a Cohen's Kappa of 0.834, indicating high agreement despite the annotators' different professional backgrounds — evidence that the annotation criteria capture general, objective principles of professional formatting rather than subjective aesthetic preferences.

The benchmark supports two evaluation protocols:

Pointwise: each document is scored independently, and accuracy measures whether the higher-scored document matches the human preference for each pair. This is the more challenging setting because the model must produce calibrated absolute scores.
Pairwise: the model is shown both documents side-by-side and asked to select the preferred one. This provides context that may help judgment but is less efficient at scale.

The paper reports results in both settings (Table 2), with DOCREWARD-7B achieving 82.3% overall accuracy in the pointwise setting (its primary evaluation), outperforming GPT-5's 67.7% by 14.6 percentage points. In the pairwise setting, DOCREWARD-7B reaches 82.3% as well (the same, since the model is used pointwise in both cases — the paper's scores in the pairwise setting in Table 2 appear to reflect pointwise model scoring applied to pairs rather than a separate pairwise inference protocol).

4. Key Insights and Innovations

Innovation 1: Disentangling Content Quality from Formatting Quality as a Formal Training Constraint

The paper's deepest conceptual move is its reframing of document professionalism assessment from a holistic judgment ("is this a good document?") into two separable dimensions — textual quality and structural/stylistic quality — and the recognition that isolating the second dimension requires a specific constraint during training. This is not a new model architecture or loss function; it is a diagnostic insight about data construction that makes the problem tractable.

Prior work in document generation and assessment treated "document quality" as an undifferentiated whole. When researchers built systems to evaluate or improve documents, they either focused exclusively on textual content (writing quality, factual accuracy, logical coherence — the domain of standard LLM evaluation) or they worked in domains where content and formatting were not entangled in the first place — single-page graphic layouts (AesthetiQ, LACE), website interfaces (Calista), or photographs (A-Lamp). In all these latter cases, there is no "textual quality" to confound because the object being evaluated is purely visual. But a multi-page professional document inextricably combines text and formatting: the words matter, and how they look matters, and in natural data these two qualities are strongly correlated — well-written documents tend to be well-formatted.

The paper's contribution is the explicit formalization of this entanglement as the central obstacle, and the construction of a training framework that breaks it. The constraint $D_{\text{text},i} = D_{\text{text},j}$ in Equation 1 is not a mathematical convenience — it is the conceptual engine of the entire approach. By requiring that all compared documents share identical text, the framework forces the reward model to learn from differences that can only be formatting differences. This transforms a confounded learning problem into a clean one: the model cannot succeed by learning to prefer eloquent prose over clumsy prose, because the prose is identical across the pair. It must learn to attend to margins, font choices, spacing, alignment, heading hierarchy — the visual elements that constitute structure and style.

This insight is significant beyond the specific model it enables. It articulates a general principle for reward model design in domains where multiple quality dimensions are correlated in natural data: construct training pairs that hold the confounding dimension constant while varying the target dimension. This principle could apply to other multimodal generation tasks — evaluating slide deck design independently of content quality, assessing data visualization clarity independently of the underlying data correctness, or judging code documentation layout independently of code correctness. The paper does not make this generalization explicit, but the framework it provides is transferable.

Evidence for the validity of this disentanglement comes from two sources. First, the ablation in Appendix A.7 showing that adding OCR text and bounding boxes reduces performance (85.00% → 80.30% for the 3B model) suggests the model has genuinely learned visual assessment rather than shortcutting through text-based heuristics — if it were relying on text content to judge quality, providing explicit text would help rather than hurt. Second, the model's cross-lingual robustness (Table 6), where performance on non-English documents (77.9%) remains close to English performance (82.3%), indicates that the learned signal is visual and structural rather than linguistic — the model is not reading the text for quality cues but perceiving formatting patterns that transcend language.

This is a fundamental shift in how to approach document professionalism assessment, not an incremental improvement over prior methods. Before this work, the field had no clear framework for separating structure/style from content in document evaluation. After it, there is a specific, operationalizable constraint that makes the separation possible, along with a demonstrated pipeline for constructing data that satisfies it.

Innovation 2: Oracle-Anchored Ranking as a Scalable Alternative to Human Annotation for Visual Preferences

The paper's second conceptual contribution is a methodological innovation in preference data collection that addresses a practical bottleneck: how do you obtain reliable pairwise quality judgments for visual formatting without paying for thousands of hours of human annotation?

The standard approaches in the preference learning literature have two extremes. At one end, full human annotation (as used in RLHF for language models, or in the DOCPAIRBENCH construction for evaluation) provides high-quality labels but is expensive and slow — annotating 117K pairs would be prohibitive. At the other end, using an LLM as a direct judge ("which of these two documents is more professional?") provides cheap labels but introduces unknown biases — the LLM may prefer formatting styles that happened to appear more frequently in its training data, or it may be influenced by spurious visual features that humans would ignore.

The paper's innovation is a third approach: oracle anchoring. Rather than asking GPT-5 for a subjective preference, the annotation prompt provides a reference document — the human-authored original — and asks which of two synthetic candidates is closer in quality to this reference. This transforms the task from "what do you like?" to "which one more closely matches this exemplar?" — an objective similarity judgment rather than a subjective preference judgment.

The conceptual move is subtle but important. Subjective preference annotation is inherently noisy because it depends on the annotator's personal aesthetic standards. Oracle anchoring replaces this with a criterion that is, in principle, objectively verifiable: given a high-quality reference document, determining which of two candidates is more similar to that reference is a perception task, not a preference task. The reference serves as an external standard that stabilizes the judgment.

This is significant because it provides a principled middle ground between expensive human annotation and unreliable LLM-as-judge approaches. The paper validates the method with 92.5% alignment against human experts on 120 pairs, demonstrating that the oracle-anchored annotations are not merely cheaper than human labels — they are nearly as accurate. This opens the door to large-scale preference data collection in any visual domain where high-quality reference examples exist, which is a common scenario (professional designs, well-formatted documents, high-quality visualizations).

The innovation is incremental in mechanism (it uses an existing LLM with a modified prompt) but fundamental in implication: it shows that the quality ceiling of LLM-based annotation can be raised substantially by changing the task framing rather than by improving the underlying model. A field that has been debating "LLMs cannot/ can replace human annotators" receives a third answer: they can, if you give them a reference standard rather than asking for a subjective opinion.

Innovation 3: A Reward Model for the Visual Domain That Outperforms General-Purpose Frontier Models

The empirical result that a specialized 7B-parameter model (DOCREWARD-7B) with 82.3% accuracy substantially outperforms GPT-5 at 67.7% — a 14.6-percentage-point gap — on a task within GPT-5's nominal multimodal capabilities is not just a performance claim. It is evidence for a specific hypothesis about specialization versus scale that runs counter to a common assumption in the current LLM era.

The dominant narrative in 2024–2025 is that very large, general-purpose models (GPT-4o, GPT-5, Claude 4) are approaching human-level performance across such a wide range of tasks that specialized smaller models are becoming obsolete — why train a task-specific model when a frontier model can do the task adequately with zero-shot prompting? This paper provides a clear counterexample: a specialized 7B model, trained on domain-specific preference data, dramatically outperforms a frontier model that is likely 1–2 orders of magnitude larger, on a task that falls squarely within the frontier model's advertised multimodal capabilities.

What makes this result intellectually interesting is not the raw numbers but what it reveals about the frontier model's failure mode. GPT-5 achieves 67.7% accuracy on a binary preference task — better than random (50%) but far from reliable (85%+ being the threshold where you would trust the model to make unsupervised quality decisions). This means that on roughly one-third of document pairs, GPT-5's judgment of which document has better formatting does not align with human judgment. The paper's analysis does not deeply diagnose why GPT-5 fails (a missed opportunity), but the existence of a 14.6-point gap between a specialized small model and a general large model on a perception task is itself a significant finding. It suggests that general-purpose multimodal training — even at massive scale — does not automatically confer fine-grained visual discrimination abilities for specific domains. The model sees millions of documents during pretraining but does not learn to distinguish professional from unprofessional formatting as reliably as a model explicitly trained to make that distinction.

The FLOPs and parameter efficiency implications are stark but unstated by the paper. DOCREWARD-7B likely requires orders of magnitude less compute than GPT-5 for both training and inference, yet provides substantially better performance on this task. If the task matters (and the RL experiments suggest it does for downstream document generation), this creates a strong economic argument for specialized reward models over general-purpose judges — at least in domains where large-scale preference data can be constructed.

This innovation is incremental in methodology (specialized fine-tuning of a base VLM on domain-specific preference data is a standard recipe) but fundamental in its implications for the specialization-versus-scale debate. It provides a concrete data point in a discussion that has been largely theoretical: for narrow, well-defined evaluation tasks, specialized training on curated preference data beats general-purpose scale, even when the scale difference is massive.

Innovation 4: Demonstrating That a Plug-in Visual Reward Model Drives Meaningful RL Improvement in Document Generation

The paper's final conceptual contribution is closing the loop from evaluation to generation: showing that a reward model trained for visual assessment can serve as an effective training signal for improving document generation agents via reinforcement learning. This is not a new idea in the abstract — using reward models for RL is the core of RLHF — but applying it to the visual formatting domain and demonstrating consistent improvement across both open-source and closed-source models is novel and practically significant.

Prior work on document generation focused primarily on content correctness and basic formatting success ("did the code execute?" "does the output match the input?"). The RL experiments in this paper (Table 4) show that adding DOCREWARD to a rule-based reward function produces documents that human annotators rank higher — not just because the documents are more likely to render successfully, but because their visual presentation is more professional. The Qwen2.5-Coder model with DOCREWARD achieves a DOCREWARD score of 0.3046 and an average human ranking of 2.84, compared to 0.1785 and 4.06 with only rule-based rewards. The GPT-4o results show an even larger gap: 0.4486 and 2.02 with DOCREWARD versus 0.3189 and 2.70 with rules only.

What makes this significant is that it demonstrates a pathway for iterative improvement through learned visual feedback. Before DOCREWARD, an agentic document generation system had no reliable way to self-improve on structure and style — it could check whether its code executed and its content matched the input, but it could not determine whether its formatting looked professional. With DOCREWARD, the optimization loop becomes: generate → score formatting → update toward higher-scoring formatting choices. This turns document generation from a one-shot generation problem into an optimizable process where the system can learn formatting principles through trial and error, guided by the reward signal.

The fact that this works for GPT-4o via training-free GRPO (no weight updates, only output selection and prompt optimization) is particularly noteworthy. It means the reward model can improve document generation even when the underlying generator is a black-box API — no fine-tuning access required. This dramatically expands the practical applicability: any document generation agent that can produce multiple candidates can be improved by DOCREWARD-as-reranker, and any agent accessible through an API that supports iterative refinement can benefit from DOCREWARD-as-reward-signal.

This innovation is systems-level and practical rather than theoretical. It takes a known paradigm (reward model → RL training) and demonstrates that it works in a new domain (visual document formatting) with non-trivial domain-specific challenges (the reward must evaluate visual quality from rendered images, not text). The contribution is in the validation that this pipeline is viable and produces documents that humans genuinely prefer — closing the gap between "we can evaluate formatting" (the intrinsic result) and "evaluating formatting leads to better documents" (the extrinsic result).

The visualization in Figure 5 provides qualitative evidence that the improvements are real: the base model produces documents with execution errors or amateurish formatting, the rule-based reward fixes the execution errors but leaves formatting mediocre, and the DOCREWARD-augmented training produces documents with visibly better structure (clearer section breaks, better alignment, more professional use of space). These visual differences are what the quantitative metrics (DOCREWARD score, human ranking) capture, and they represent the practical payoff of the entire pipeline.

5. Experimental Analysis

Evaluation Methodology

Dataset. The primary evaluation benchmark is DOCPAIRBENCH, a human-annotated dataset of 1,443 comparison pairs constructed from a held-out subset of the curated documents not used in training (Section 4.1). Each pair consists of two documents sharing identical textual content but differing in structure and style, with binary labels indicating which document a human annotator preferred. The pairs are drawn from six document origins: four from different Textual Content to Document agents (using GPT-4o, OpenAI o1, Claude Sonnet 4, and GPT-5 as base LLMs), one from the Refinement for Better Structure and Style agent (GPT-5 refining synthetic documents), and one from the curated human-authored originals. The benchmark spans ten domains: Government, Education, Non-Profit, Medical, Science, Legal, Business, Academic, Technology, and Other.
Base model(s). Two sizes are trained: DOCREWARD-3B and DOCREWARD-7B, both built on the Qwen2.5-VL multimodal vision-language model family (Section 3.2). The choice of Qwen2.5-VL is motivated by its ability to process rendered document pages as visual input, which aligns with the task's inherently visual nature — professionalism in structure and style is perceived visually, not through structured markup. For the reinforcement learning experiments, the paper uses Qwen2.5-Coder-1.5B (open-source) and GPT-4o (closed-source) as the document generation agents being optimized.
Metrics. The primary metric on DOCPAIRBENCH is accuracy — the fraction of comparison pairs where the model's predicted preference (higher score = preferred) matches the human annotation. This is reported both overall and per-domain (Table 2). For the Best-of-N evaluation, the metric is win/lose/tie rate based on human annotators ranking documents selected by different reward models (Table 3). For the reinforcement learning experiments, four metrics are reported (Table 4): Success rate (fraction of generated Python code that executes without errors), ROUGE-L (measuring textual content preservation between original plain text and generated document), Score (the sigmoid-normalized DOCREWARD score reflecting formatting quality), and Rank (average human ranking among the six compared methods, lower is better). Inter-annotator reliability is measured using Cohen's Kappa, achieving 0.834 across three annotators (Appendix A.4).
Baselines. The paper compares DOCREWARD against five baselines on DOCPAIRBENCH (Table 2): Qwen2.5-VL-3B and Qwen2.5-VL-7B (the base vision-language models without specialized training, representing the untrained starting point); GPT-4o (Hurst et al., 2024); Claude Sonnet 4 (Anthropic, 2025); and GPT-5 (OpenAI, 2025). These baselines are evaluated in both pointwise (independent scoring of each document) and pairwise (direct comparison) settings. For the Best-of-N experiment (Table 3), baselines include Random selection and GPT-5-based selection. For reinforcement learning (Table 4), the baseline is the base generation model without any RL training, and the + rule condition (using only R_rule without DOCREWARD) serves as an intermediate baseline isolating the contribution of the formatting reward.
Generation budget / compute accounting. For Best-of-N, the budget is the number of candidate documents generated per textual content input, with N = 8 used in the reported experiments (Appendix A.6). For the reinforcement learning experiments, the compute budget is not explicitly quantified in terms of FLOPs or training steps — the paper reports training outcomes rather than compute-normalized comparisons. The reward model training itself uses a fixed configuration: batch size 256, 3 epochs, on 8 NVIDIA A100 GPUs (80GB), with a maximum input resolution of 300,000 pixels and context length of 16,000 tokens (Appendix A.2).
Cross-validation / statistical protocol. For DOCPAIRBENCH construction, three annotators independently ranked documents, and the inter-annotator agreement was measured via Cohen's Kappa (0.834) to validate annotation quality (Appendix A.4). The annotation guidelines (Figure 8 in Appendix A.4) specify explicit, objective criteria for structure and style evaluation, intended to minimize dependence on annotators' cultural and professional backgrounds. For the Best-of-N evaluation, human annotators ranked the outputs selected by the three compared reward models across 130 comparison pairs, with win/lose/tie rates computed from these pairwise rankings (Section 4.3). The reinforcement learning results include human rankings (the "Rank" metric in Table 4) as the ultimate quality signal, though the paper does not specify the number of annotators or inter-annotator agreement for this specific evaluation.

Main Quantitative Results

Intrinsic Evaluation: DOCPAIRBENCH Accuracy

The headline result appears in Table 2: DOCREWARD-7B achieves 82.3% overall accuracy in the pointwise setting (the primary evaluation mode), compared to GPT-5's 67.7% — a 14.6 percentage point advantage. DOCREWARD-3B achieves 80.6%, still outperforming GPT-5 by 12.9 points.

The pairwise setting (where models are shown both documents and asked to choose) produces similar relative rankings: DOCREWARD-7B at 82.3% versus GPT-5 at 68.9% — a 13.4 point gap. This suggests the models' rankings are consistent regardless of whether evaluation is pointwise or pairwise.

Domain-level breakdown (Table 2, pointwise setting). DOCREWARD-7B outperforms GPT-5 in 8 of the 10 domains, with particularly large gaps in:

Government: 89.3% vs. 69.7% (+19.6 points)
Education: 92.3% vs. 80.8% (+11.5 points)
Legal: 87.0% vs. 72.0% (+15.0 points)
Academic: 80.9% vs. 60.0% (+20.9 points) — the largest single-domain gap

GPT-5 edges ahead in only two domains:

Science: 49.0% vs. DOCREWARD-7B's 67.0% — a surprising reversal where both models struggle (near-chance for GPT-5, notably below DOCREWARD's average for the specialized model), suggesting scientific documents may have domain-specific formatting conventions that DOCREWARD has learned but GPT-5 has not fully absorbed
Technology: 64.6% vs. DOCREWARD-7B's 76.4%

The baseline Qwen2.5-VL models without DOCREWARD training perform substantially worse: Qwen2.5-VL-7B achieves only 46.0% overall in the pointwise setting — below chance, indicating that the base vision-language model, despite its multimodal pretraining, has not learned to evaluate document professionalism without the specialized training data and preference optimization. This 36.3-point gap between DOCREWARD-7B (82.3%) and its own base model (46.0%) demonstrates that the DOCPAIR training data and Bradley-Terry optimization, not the base model architecture, are responsible for the performance.

Model scale comparison. DOCREWARD-7B outperforms DOCREWARD-3B by only 1.7 percentage points (82.3% vs. 80.6%), suggesting that the task benefits more from specialized training data than from model capacity — the 3B model already captures most of the signal, with diminishing returns from the 3B → 7B scale increase.

Extrinsic Evaluation: Best-of-N Selection

Table 3 reports the results of using DOCREWARD as a re-ranking model to select the best candidate from N = 8 agent-generated documents. When human annotators compare the documents selected by three different reward models (Random, GPT-5, DOCREWARD), DOCREWARD achieves a win rate of 60.8% against the combined pool of the other two models, with a loss rate of only 16.9% and a tie rate of 22.3%.

In comparison:

GPT-5 achieves a win rate of 37.7%, loss rate of 40.0%, and tie rate of 22.3%
Random achieves a win rate of 24.6%, loss rate of 66.2%, and tie rate of 9.2%

The baseline random selection shows that the agent-generated candidates have meaningful quality differences: picking randomly yields a document preferred by humans only 24.6% of the time, while the "best" document among the candidates (as judged by humans comparing the selections) would win roughly 60.8% of the time if perfectly selected. DOCREWARD's 60.8% win rate means it is nearly matching the oracle upper bound of this evaluation paradigm, while GPT-5's 37.7% leaves substantial room for improvement.

Extrinsic Evaluation: Reinforcement Learning for Document Generation

Table 4 presents the results of using DOCREWARD as a reward signal in RL training of document generation agents. The evaluation includes four metrics measured on held-out test documents.

Qwen2.5-Coder-1.5B results:

Base model (no RL): Success rate 30.0%, ROUGE-L 20.61, DOCREWARD score 0.0663, human rank 4.58
+ rule (rule-based reward only): Success rate 98.0%, ROUGE-L 97.94, DOCREWARD score 0.1785, human rank 4.06
+ DocReward (rule + DOCREWARD reward): Success rate 100.0%, ROUGE-L 97.95, DOCREWARD score 0.3046, human rank 2.84

The rule-based reward alone dramatically improves execution success (30% → 98%) and content preservation (ROUGE-L 20.61 → 97.94), indicating that the base model's primary weakness is generating valid, content-accurate Python code. Adding DOCREWARD further improves success rate to 100% and nearly doubles the formatting quality score (0.1785 → 0.3046), while improving human ranking from 4.06 to 2.84 — a substantial jump in perceived quality.

GPT-4o results (training-free GRPO):

Base model (no RL): Success rate 52.0%, ROUGE-L 48.73, DOCREWARD score 0.2682, human rank 3.18
+ rule: Success rate 66.0%, ROUGE-L 62.15, DOCREWARD score 0.3189, human rank 2.70
+ DocReward: Success rate 78.0%, ROUGE-L 74.33, DOCREWARD score 0.4486, human rank 2.02

The GPT-4o base model starts from a stronger position (52% success vs. 30% for Qwen2.5), reflecting its superior code generation capabilities. The rule-based reward provides meaningful but modest improvements. Adding DOCREWARD yields substantial gains across all metrics: success rate improves 12 more points (66% → 78%), ROUGE-L gains 12 points (62.15 → 74.33), and the DOCREWARD score increases by 0.1297 (0.3189 → 0.4486). The human ranking improves from 2.70 to 2.02, placing the DOCREWARD-augmented GPT-4o model as the best-ranked system among all six variants.

Cross-model comparison. A notable pattern emerges: DOCREWARD-augmented GPT-4o achieves a DOCREWARD score of 0.4486 compared to DOCREWARD-augmented Qwen2.5-Coder's 0.3046 — a 47% relative improvement for the closed-source model. This suggests that DOCREWARD's reward signal is complementary to the base model's capabilities: a stronger base model (GPT-4o vs. Qwen2.5-Coder-1.5B) can better exploit the formatting guidance to produce more professional output. The human ranking confirms this (2.02 vs. 2.84), though the gap narrows when measured by human preference rather than automated score.

Robustness: Out-of-Domain Generalization

Table 5 reports evaluation on domains that were excluded from training — a direct test of whether DOCREWARD learns general principles of professional formatting or memorizes domain-specific conventions.

DOCREWARD-7B achieves 77.5% on out-of-domain (OOD) documents, compared to 82.3% in-domain — a drop of only 4.8 percentage points. This small degradation suggests the model has learned transferable formatting principles rather than domain-specific heuristics. In comparison, GPT-5 achieves 68.4% on OOD documents, statistically indistinguishable from its 67.7% in-domain performance — but DOCREWARD-7B still outperforms GPT-5 by 9.1 points in the OOD setting (77.5% vs. 68.4%). The other baselines show varying OOD degradation: Qwen2.5-VL-7B actually improves slightly on OOD (48.1% vs. 46.0%), while Claude Sonnet 4 drops by 5.1 points (49.1% vs. 54.2%).

DOCREWARD-3B shows a similarly small OOD gap: 80.6% → 76.3%, a 4.3-point drop, while still outperforming GPT-5's OOD performance by 7.9 points.

Robustness: Cross-Lingual Generalization

Table 6 evaluates models on documents in French, Spanish, and Russian — all languages not represented in the English-only DOCPAIR training data.

In the non-English aggregate (average across French, Spanish, and Russian), DOCREWARD-7B achieves 77.9%, compared to 82.3% on English — a drop of 4.4 percentage points. This is comparable to its OOD degradation and smaller than the cross-lingual degradation observed for GPT-4o (−7.4) and GPT-5 (−7.3). The relative ordering among languages varies: DOCREWARD-7B performs best on Spanish (88.8%) and worst on Russian (66.3%), while GPT-5 shows a similarly uneven pattern (Spanish 76.3%, Russian 47.5%).

The key finding from Table 6 is the ∆ column (performance gap between English and non-English). DOCREWARD-3B (−3.1) and DOCREWARD-7B (−4.4) show smaller degradation than GPT-4o (−7.4) and GPT-5 (−7.3). This suggests that DOCREWARD's learned formatting assessment relies more on visual cues (spacing, alignment, font hierarchy, white space distribution) that transcend language, and less on text-level comprehension that would be more language-dependent. A document with good margins, clear headings, and consistent spacing looks professional regardless of whether the text is in English or Russian, and DOCREWARD appears to capture this language-agnostic visual signal.

Visualization: Attention Map Analysis

Figure 6 provides qualitative evidence of what DOCREWARD attends to when scoring documents. The attention maps highlight specific structural and stylistic elements:

Figure 6a: Attention focuses on headings and numbering patterns, indicating the model has learned that clear hierarchical organization (sections labeled with numbers, headings visually distinct from body text) is a hallmark of professional structure.
Figure 6b: Attention gravitates to page headers and footers (e.g., document identifiers like "CS-66" and dates like "DEC. 2006"), bullet points, and page corners. The header/footer attention suggests the model recognizes that professional documents include consistent metadata across pages. The page corner attention implies the model evaluates margin uniformity and white space balance — uneven margins or cramped corners would signal unprofessional layout.
Figure 6c: Attention concentrates on table borders and corners, indicating sensitivity to structured data presentation. Well-drawn tables with clear borders and proper alignment are visual indicators of professional formatting.

These attention patterns are consistent with the annotation criteria defined in Figure 8 (layout and design, readability and typography, professional standards, visual elements) and provide a plausibility check that the model's decisions are based on interpretable formatting features rather than spurious correlations.

Case Study: Score Calibration

Figure 7 presents a qualitative case study with three versions of the same content scored by DOCREWARD:

Case (a), score 1.21: An imbalanced layout with ineffective white space allocation (insufficient space for "Last Name," excessive space for "First Name") and misaligned key fields (Faculty/Department, Country, Country Code not vertically aligned). The low score correctly reflects these structural problems.
Case (b), score 2.11: A table-like arrangement that addresses alignment but introduces new issues — the level-1 heading "The teaching staff member" uses a small font that fails to distinguish it from body text, and missing table borders make input fields hard to locate. The moderate score reflects partial improvement over case (a) with remaining deficiencies.
Case (c), score 5.34: A clear, well-structured layout with appropriately sized headings, vertical alignment of fields, and table borders for readability. The highest score reflects comprehensive structural and stylistic professionalism.

The scores (1.21, 2.11, 5.34) are not on an absolute scale — the raw DOCREWARD output is unnormalized before sigmoid — but their relative ordering matches the qualitative improvement visible in the rendered documents. The score ratio between the worst and best document is approximately 4.4×, indicating the model produces well-separated scores that discriminate between quality levels rather than clustering all documents in a narrow range.

Ablation Studies and Robustness Checks

Input modality (Appendix A.7, Table 8). When comparing image-only input versus image + OCR text with bounding boxes, image-only outperforms the combined input for both model sizes. On a subset of the test set, DOCREWARD-3B achieves 85.00% with image-only versus 80.30% with image + OCR; DOCREWARD-7B achieves 87.94% versus 84.41%. This finding is non-obvious: one might expect that providing explicit text positions (bounding boxes) would help the model evaluate alignment and spacing more precisely. The counterintuitive result suggests that OCR-derived representations lose subtle visual information (font rendering quality, color, spacing nuances, visual texture) that the model uses for assessment, and that the model's native visual processing is more effective than discretized positional features.
Model scale (Table 2). DOCREWARD-3B (80.6%) vs. DOCREWARD-7B (82.3%) — the 1.7-point gap is remarkably small given the 2.3× parameter increase, suggesting the task saturates at relatively modest model capacity when training data quality and quantity are sufficient.
Domain exclusion — training vs. evaluation split (Table 5). Domains held out from training show only a 4.8-point degradation for DOCREWARD-7B (82.3% → 77.5%), confirming generalization rather than memorization of domain-specific formatting conventions. However, the paper does not report which specific domains were held out or how many OOD test pairs were used — these details matter for assessing whether the OOD test covers genuinely distinct formatting conventions or domains that happen to share conventions with the training set.
Cross-lingual training data (Table 6). DOCREWARD was trained exclusively on English documents, yet achieves 77.9% on non-English documents. The paper does not ablate the effect of including non-English training data — an experiment that would clarify whether the 4.4-point cross-lingual gap could be closed by adding a modest amount of multilingual training pairs.

Critical Assessment

Claim 1 from the executive summary: "DOCREWARD outperforms GPT-5 by 14.6 percentage points."

This claim is supported and precise when stated as it appears in the paper: "in the same setting" (pointwise evaluation on DOCPAIRBENCH). However, the evaluation setting requires careful qualification. The comparison is between a specialized 7B model fine-tuned on 117K domain-specific preference pairs and a frontier model evaluated zero-shot — GPT-5 was not fine-tuned on document formatting preferences. This is a valid and practically relevant comparison (since users would use GPT-5 zero-shot if they lacked DOCREWARD), but it answers the question "does specialization beat zero-shot prompting?" rather than "does specialization beat comparable-scale training on the same data?" The paper does not report what would happen if GPT-5 were fine-tuned on DOCPAIR, which would be the stronger specialization-versus-scale test. This missing experiment is understandable — GPT-5 fine-tuning access was likely unavailable — but it means the 14.6-point gap cannot be entirely attributed to the training framework versus model architecture; some fraction of the gap likely reflects the fine-tuning process itself, independent of the dataset quality.

A second qualification: the "pointwise" setting in Table 2 uses DOCREWARD's raw pointwise scoring, but the paper does not clarify whether baselines are also evaluated pointwise (score each document independently and compare) or whether baselines might perform better with a pairwise prompt (show both documents and ask "which is better?"). The table includes both "Pairwise Setting" and "Pointwise Setting" rows, suggesting each baseline is evaluated under its best protocol. However, the pairwise results for GPT-5 (68.9%) are only 1.2 points higher than its pointwise results (67.7%) — the protocol choice does not dramatically change the outcome.

Claim 2 from the executive summary: "Reinforcement learning experiments demonstrate that DOCREWARD effectively guides agents toward generating documents with consistently higher structural and stylistic professionalism."

This claim is supported by Table 4 and Figure 5, but with significant gaps in experimental detail. The human ranking metric (the most important column in Table 4) is reported without any information about how many annotators ranked the documents, how many documents were ranked, what the ranking protocol was, or what inter-annotator agreement was achieved. For the DOCPAIRBENCH evaluation, the paper carefully documents annotation protocol (three annotators, Cohen's Kappa 0.834, detailed guidelines in Figure 8). For the RL evaluation — arguably the more impactful extrinsic result — this rigor is absent. The "Rank" numbers (2.84, 2.02, etc.) lack confidence intervals, making it impossible to assess whether the differences between methods are statistically significant. A difference of 2.84 vs. 2.02 (0.82 rank positions on a 6-point scale) could be substantively meaningful or could fall within annotator variance.

The figure of 130 comparison pairs for the Best-of-N evaluation (Appendix A.6) provides transparency about sample size, but no comparable detail exists for the RL ranking evaluation. The qualitative evidence in Figure 5 strengthens the claim — the visual differences between "Base model," "+ Rule," and "+ DocReward" documents are apparent — but qualitative evidence cannot substitute for rigorous quantitative evaluation with documented protocols.

A missing experiment: the paper does not report an ablation where DOCREWARD is used as the sole reward (without R_rule). This would clarify whether the formatting improvements come from DOCREWARD directly guiding formatting choices, or from a synergistic effect where DOCREWARD reinforces behaviors already incentivized by R_rule. The staged reward formulation (I_rule gating DOCREWARD) is sensible — you don't want to reward formatting when the code doesn't even execute — but it makes it impossible to isolate DOCREWARD's independent contribution.

Claim from the abstract: "DOCPAIR, a dataset of 117K paired documents covering 32 domains and 267 types."

This claim is supported by the data statistics in Table 1 and Figure 3, but the documentation of the dataset construction has a notable ambiguity. Section 3.1 states that the "Real vs. Synth." ranking heuristic (human-authored always wins) was "grounded in a human inspection on a randomly sampled set of 100 samples." The paper reports that this inspection confirmed synthetic documents were inferior. However, this heuristic means that all Real-vs-Synth pairs have the same label direction — the synthetic document is always the loser. If the training data is heavily imbalanced toward this case (which seems plausible given that many pairs likely involve the human-authored original against one or more synthetic variants), the model might learn a shortcut: always prefer documents that "look more like the training set's real documents" rather than learning general principles of professional formatting. The paper does not report the distribution of pair types in DOCPAIR (Real vs. Synth vs. Synth vs. Synth), making it difficult to assess this risk.

The oracle-anchored annotation method (for Synth vs. Synth pairs) was validated on 120 pairs with 92.5% alignment against human experts. This is a strong validation, but 120 pairs is a small sample relative to the scale of DOCPAIR (117K pairs). If the annotation errors are systematic rather than random — for instance, if GPT-5 consistently prefers certain formatting styles that diverge from human preferences in specific domains — then validation on 120 pairs drawn from a narrow subset of domains might overestimate the method's reliability. The paper does not break down the 92.5% figure by domain or document type.

What the experiments demonstrate versus what they leave open.

The experiments convincingly demonstrate that:

DOCREWARD distinguishes professionally formatted documents from unprofessionally formatted ones substantially better than zero-shot GPT-5 on the DOCPAIRBENCH benchmark (Table 2).
DOCREWARD generalizes to unseen domains and non-English languages with modest degradation (Tables 5, 6).
DOCREWARD can serve as an effective re-ranking model for Best-of-N selection (Table 3).
Adding DOCREWARD to a rule-based reward function improves RL training outcomes for document generation (Table 4).

The experiments do not demonstrate:

How DOCREWARD compares to a fine-tuned GPT-5 or fine-tuned Claude Sonnet 4 — the specialization-versus-scale claim is relative to zero-shot baselines, not to equally fine-tuned larger models.
The statistical reliability of the RL human ranking results (no confidence intervals, no sample size, no inter-annotator agreement scores).
How DOCREWARD's performance scales with training data size — is 117K pairs necessary, or would 10K pairs achieve similar performance? This ablation would be practically valuable for practitioners wanting to construct similar reward models in other domains.
Whether the Real-vs-Synth training pairs create a "realism bias" that impairs the model's ability to compare two synthetic documents — the DOCPAIRBENCH contains pairs from all six document origins, but the paper does not report accuracy broken down by pair type (Real vs. Synth, Synth vs. Synth).
How DOCREWARD handles edge cases: documents with extreme formatting (very long, image-heavy, multi-column layouts, embedded charts) that may deviate from the training distribution. The training pipeline explicitly filtered out documents larger than 1 MB dominated by images and documents exceeding 20 pages (Phase 1), creating a systematic exclusion whose impact on the model's robustness is unexplored.
Whether DOCREWARD's scores are well-calibrated across different document types — does a score of 5.0 mean the same level of professionalism for a government form as for an academic paper? The case study (Figure 7) provides anecdotal evidence but not systematic calibration analysis.

A missing ablation of high practical importance: the paper does not evaluate DOCREWARD on documents generated by document agents different from the ones used in DOCPAIR construction. The training data includes documents generated by GPT-4o, o1, Claude Sonnet 4, and GPT-5. The evaluation data (DOCPAIRBENCH) includes documents from the same four models plus refinement outputs. If a practitioner deploys DOCREWARD with a different document generation agent (e.g., Gemini, Llama-based agent, or a template-based system), the reward model might encounter formatting styles outside its training distribution — and its performance in this setting is unknown. A cross-agent generalization experiment would substantially strengthen the practical utility claim.

Summary. The experimental evidence supports the paper's central narrative — DOCREWARD is an effective specialized reward model for document formatting assessment that can improve agentic document generation — but the support is stronger for the intrinsic evaluation (does the model capture human preferences?) than for the extrinsic evaluation (does using the model actually improve generated documents?). The intrinsic results are methodical, well-documented, and convincing. The extrinsic results are promising but underspecified in critical dimensions (sample sizes, annotator protocols, statistical reliability). The paper's claims are appropriately scoped — it does not claim to have solved document generation, only to have built a useful reward model for one dimension of quality — and the experiments match this scope. However, several straightforward experiments (data scaling ablation, cross-agent generalization, calibration analysis by document type) would have substantially increased confidence in the practical deployability of the approach.

6. Limitations and Trade-offs

6.1 The Oracle-Anchored Ranking Strategy Assumes That Human-Authored Documents Are Always Superior to Synthetic Ones

The assumption. In Phase 3 of the DOCPAIR construction pipeline, the paper adopts a ranking heuristic for Real-vs-Synthetic pairs: "When comparing an original document with its agent-generated counterparts, the human-authored version is designated as the winner" (Section 3.1). This heuristic is grounded in a human inspection of 100 randomly sampled pairs, which "indicates that current state-of-the-art models (e.g., GPT-5, Claude Sonnet 4) produce documents with structural and stylistic quality inferior to high-quality human-authored documents curated by the rigorous filtering pipeline."

The consequence. This heuristic bakes a strong structural assumption into the training data: the model never sees a training example where a synthetic document is preferred over a human-authored one. Two failure modes follow.

First, the model may learn a "realism shortcut" rather than genuinely evaluating formatting quality. If human-authored documents share certain surface-level visual signatures (e.g., specific font families common in government documents, particular margin conventions, characteristic header styles) that synthetic documents lack regardless of their objective formatting quality, the model could learn to prefer documents that "look like training-set real documents" rather than documents that genuinely have better structure and style. The DOCPAIRBENCH evaluation partially addresses this — it includes pairs where synthetic documents are compared to each other — but it does not include pairs where a synthetic document genuinely outperforms a human-authored one, because the annotation protocol (Figure 8) always presents the human-authored document as the reference, and there is no mechanism for annotators to rank a synthetic document above it. The paper acknowledges the possibility: "Note that there may exist cases where human-authored documents are not the best ones" (Figure 8 annotation guidelines). Whether those cases exist in practice — and how often — is unknown.

Second, the heuristic may become less valid over time as document generation agents improve. The paper's validation was conducted with GPT-5 and Claude Sonnet 4, and it confirmed these models' synthetic outputs were inferior. But if a future generation agent (GPT-6, or a specialized document formatting model) produces synthetic documents that match or exceed human formatting quality, the Real-always-wins heuristic would introduce incorrect labels. The model would be trained to penalize better formatting in those cases. This makes DOCPAIR a snapshot of a particular moment in LLM capability, not a timeless training resource.

What evidence exists in the paper. The paper provides only the 100-sample validation (Section 3.1) to support the heuristic. It does not report the gap in quality between human and synthetic documents — how large is the margin? Are synthetic documents uniformly worse, or does the gap vary by domain and document type? There is no breakdown of synthetic document quality by agent type (GPT-4o vs. GPT-5 vs. Claude Sonnet 4), which would help assess whether more capable agents are approaching human-level formatting. The paper also does not analyze whether the human-authored documents in the evaluation set are actually always preferred — the annotation guidelines allow for synthetic documents to be ranked highest, but the paper does not report how often this occurred.

Mitigation status. The paper does not address this limitation. No experiment measures whether the model has learned a realism shortcut versus genuine formatting assessment. The out-of-domain evaluation (Table 5) provides indirect evidence against severe shortcut learning (the model generalizes to unseen domains with only a 4.8-point degradation), but generalization to unseen domains does not rule out a realism shortcut — the unseen domains may share the same surface-level signatures as the training domains, or the model may have learned a combination of genuine formatting principles and realism cues.

The paper suggests no future work to address this. A straightforward diagnostic would be to construct a small test set of human-authored documents with deliberate formatting errors (e.g., remove headers, collapse margins, mix font sizes) and verify that DOCREWARD penalizes these documents relative to well-formatted synthetic ones — a test of whether the model values formatting quality or merely "human-authoredness."

6.2 DOCREWARD Produces Only a Scalar Score, Not Actionable Feedback

The assumption. DOCREWARD is designed as a pointwise scoring model: given a rendered document, it outputs a single scalar number representing overall structural and stylistic professionalism. The model provides no natural language explanation, no identification of specific formatting problems, and no suggestions for improvement.

The consequence. A scalar score limits how DOCREWARD can be used in agentic workflows. In the Best-of-N selection paradigm (Section 4.3), the score is sufficient — the agent generates multiple candidates, and the one with the highest score is selected. But for the more ambitious goal of iterative improvement, a scalar score is impoverished feedback.

Consider a reinforcement learning setup where the agent needs to learn how to improve formatting. When the reward is a single number, the agent must discover through trial and error which formatting changes increase the score — thousands of rollouts may be needed to associate specific actions (increase font size, add borders, adjust margins) with reward improvements. If DOCREWARD instead produced diagnostic feedback ("the heading font is too small relative to body text; the margins are uneven on page 2; the table on page 3 lacks borders"), the agent could target specific improvements directly, potentially learning faster and with less compute.

The scalar-only limitation also affects human-in-the-loop workflows. A human reviewer using DOCREWARD to evaluate generated documents sees only a score (e.g., "4.23") with no explanation of what lowered the score or how to fix it. This makes the model less useful as a quality assurance tool compared to alternatives that provide justifications alongside scores.

What evidence exists in the paper. The paper explicitly acknowledges this limitation in Appendix A.1:

"A limitation is that DOCREWARD is designed as a scalar reward model, providing only a numerical score to represent the overall structural and stylistic quality of a document. It currently cannot generate natural language feedback explaining why a document is perceived as unprofessional."

The attention map analysis in Figure 6 provides some post-hoc interpretability — it shows which visual regions the model attended to when scoring — but attention maps are not feedback. They tell a researcher what the model looked at, not what the document generator should change.

Mitigation status. The paper identifies this as the primary focus of future work: "extending DOCREWARD from a scalar model to an interpretable reward model that can produce both quantitative scores and qualitative diagnostic rationales remains a key focus of our future work" (Appendix A.1). The current version offers no mitigation — users of DOCREWARD in its present form must accept a single number as the sole output. The RL experiments (Section 4.3) demonstrate that a scalar reward can drive improvement, but they do not measure the sample efficiency of learning from scalar feedback versus richer feedback — a comparison that would quantify how much this limitation matters in practice.

A partial mitigation exists implicitly: the agent's training still includes the rule-based reward R_rule (Equation 3), which provides explicit feedback on execution success and content preservation. But for the formatting-specific component, the signal is purely scalar.

6.3 The Paper Does Not Measure or Account for the Inference Cost of DOCREWARD in Downstream Workflows

The assumption. DOCREWARD is presented as a practical plug-in component for agentic document generation workflows. The paper reports accuracy improvements (Table 2, 3) and reinforcement learning gains (Table 4) without quantifying the computational cost of running DOCREWARD itself relative to the cost of the document generation agent.

The consequence. For a practitioner deciding whether to integrate DOCREWARD into their pipeline, the missing cost analysis creates uncertainty about the net benefit. In the Best-of-N setting with N = 8 (Appendix A.6), the document agent must generate 8 candidate documents, and DOCREWARD must score all 8 before selecting the best one. If DOCREWARD's inference cost is comparable to the generation agent's cost, this doubles the total compute per query (8 generations + 8 scorings). If DOCREWARD runs on separate hardware or has high latency, the wall-clock impact could be worse.

In the reinforcement learning setting, DOCREWARD is called for every rollout during training. The paper does not specify how many rollouts were used in the GRPO training, what the ratio of DOCREWARD inference cost to agent generation cost is, or whether DOCREWARD inference was a bottleneck. Without these numbers, it is impossible to assess whether the improvements in Table 4 (e.g., DOCREWARD score from 0.1785 to 0.3046 for Qwen2.5-Coder) justify the additional inference compute.

The cost question is particularly salient because DOCREWARD-7B (based on Qwen2.5-VL-7B) is a 7-billion-parameter vision-language model processing multi-page rendered documents. Vision transformers are computationally intensive — processing 300,000 pixels of document images through a 7B-parameter model likely costs more than running a text-only agent of comparable size. The paper's statement that DOCREWARD "serves as an effective and generalizable reward model" (Section 4.3) is an accuracy claim, not a cost-effectiveness claim.

What evidence exists in the paper. The paper includes no inference cost analysis. The training hardware is specified (8 NVIDIA A100 GPUs, Appendix A.2), but the inference requirements (GPU type, tokens per document, latency per document) are not reported. There is no FLOPs comparison between DOCREWARD scoring and the document generation agent. The Best-of-N evaluation (Table 3) reports win rates but not the total wall-clock time or compute cost to achieve those win rates.

Mitigation status. The paper does not address this limitation. There is no mention of inference cost as a consideration, no latency measurements, and no discussion of whether DOCREWARD could be distilled, quantized, or otherwise optimized for deployment. The availability of a 3B variant (DOCREWARD-3B) that achieves 80.6% accuracy versus 82.3% for the 7B version suggests a potential cost-accuracy tradeoff — the 3B model is roughly 2.3× smaller and likely faster, with only a 1.7-point accuracy sacrifice — but this tradeoff is not analyzed in inference cost terms.

For a practitioner, the practical question is: at what N in Best-of-N does the cost of scoring N documents with DOCREWARD become comparable to or exceed the cost of generating N documents with the agent? If the crossover point is N = 4 and the practitioner needs N = 8 for sufficient quality improvement, the cost-benefit calculus changes. The paper provides no data to answer this question.

6.4 The DOCPAIR Construction Pipeline Systematically Excludes Certain Document Types, Limiting Generalization Scope

The assumption. The training data for DOCREWARD is drawn from a specific distribution: documents that survived the Phase 1 quality filtering pipeline (Section 3.1). This filtering excludes several categories of documents that may appear in real-world agentic workflows.

The consequence. Three exclusion criteria create systematic coverage gaps:

First, documents exceeding 20 pages are removed during preprocessing. This means DOCREWARD has never been trained on long documents — research reports, technical manuals, legal contracts, book chapters, and other lengthy professional documents fall outside its training distribution. When presented with a 50-page report during inference, DOCREWARD may exhibit undefined behavior: the 16,000-token context window and 300,000-pixel input budget impose hard limits, but the paper does not specify whether the model truncates, downscales, or rejects such documents. A practitioner generating long-form documents (the primary use case for systems like Deep Research) cannot rely on DOCREWARD's evaluation for documents whose length exceeds the training maximum.

Second, files larger than 1 MB dominated by images are discarded. This excludes image-rich documents: illustrated reports, photo-heavy brochures, documents with embedded charts and diagrams, and visually complex layouts where images are central to professionalism. DOCREWARD is trained primarily on text-dominated documents, and its ability to evaluate the professionalism of image placement, image-text integration, and visual balance in image-heavy layouts is untested and likely limited. The paper's evaluation documents (DOCPAIRBENCH, Section 4.1) are drawn from the same filtered corpus, so the benchmark does not measure this gap.

Third, files smaller than 10 KB with trivial content are removed. This excludes short documents — single-page forms, certificates, short memos, cover pages. While these may seem less professionally demanding, they represent a non-trivial fraction of real-world documents. Evaluating the professionalism of a single-page certificate (where every formatting element — centering, font choice, border style, white space balance — is highly visible) may require different sensitivity than evaluating multi-page reports where individual formatting flaws are diluted by volume.

What evidence exists in the paper. The paper documents the filtering criteria explicitly (Section 3.1) but does not analyze the excluded population — what fraction of the original source documents fell into each exclusion category, what document types they represented, and whether their exclusion creates systematic biases. The average document length in DOCPAIR is 3.2 pages (Table 1), confirming that the training distribution is centered on short-to-medium documents. The paper's out-of-domain evaluation (Table 5) tests generalization to unseen domains but does not test generalization to unseen document lengths or image densities.

Mitigation status. The paper does not address this limitation. It does not acknowledge that length, image density, and document complexity are potential generalization axes beyond the domain and type categories explicitly measured. No experiment evaluates DOCREWARD on documents exceeding the training length maximum. No ablation tests whether performance degrades as document length increases (e.g., accuracy on 1-page, 5-page, 10-page, and 20-page documents separately).

For a practitioner deploying DOCREWARD in a workflow that generates long or image-rich documents, this is a meaningful gap: the model's accuracy on the primary evaluation benchmark (DOCPAIRBENCH, average length ~3.2 pages) may not transfer to the practitioner's use case, and the paper provides no basis for predicting how large the degradation would be.

6.5 The Reinforcement Learning Evaluation Lacks Statistical Rigor for the Human Ranking Metric

The assumption. The paper's most practically important claim is that DOCREWARD "serves as an effective reward model in reinforcement learning for agentic workflows" (Section 1), supported by the RL experiments in Section 4.3. The credibility of this claim depends on the reliability of the human evaluation, particularly the "Rank" metric in Table 4.

The consequence. The human ranking evaluation is underdocumented in critical dimensions, making it difficult to assess the strength of the RL results:

Sample size: The paper does not report how many documents human annotators ranked for the RL evaluation. The Best-of-N evaluation specifies 130 comparison pairs (Appendix A.6). The RL evaluation provides no comparable figure. If only a small number of documents were ranked (e.g., 20–30), the average ranks in Table 4 could be heavily influenced by a few edge cases.
Number of annotators and agreement: The DOCPAIRBENCH construction involved three annotators with Cohen's Kappa of 0.834 (Appendix A.4). The RL evaluation does not report how many annotators performed the ranking or what inter-annotator agreement was achieved. If only one annotator ranked the documents, the results reflect individual preference rather than consensus. If multiple annotators ranked them but agreement was low, the average ranks may mask substantial disagreement about which documents were better.
Statistical significance: The "Rank" column in Table 4 shows differences like 4.58 vs. 4.06 vs. 2.84 for Qwen2.5-Coder variants. These are average ranks on a 6-point scale (six model variants compared). The paper provides no confidence intervals, no standard deviations, and no significance tests. A difference of 0.52 rank positions (4.58 vs. 4.06) could be meaningful or could be noise — there is no way to tell from the reported data.
Ranking protocol: The paper states that "human annotators rank documents produced by six model variants based on the professionalism of structure and style" (Section 4.3), but does not specify whether annotators saw all six documents simultaneously, whether ranking was done in pairwise comparisons, whether documents were anonymized, or whether the order of presentation was randomized. Multi-item ranking is cognitively demanding and susceptible to order effects — without protocol details, the reliability of the rankings is uncertain.

What evidence exists in the paper. The RL evaluation is reported in Table 4 and visualized qualitatively in Figure 5. The paper provides exact numbers for Success rate, ROUGE-L, and DOCREWARD Score (automated metrics), but the most important metric — human ranking — is reported as point estimates without any measure of uncertainty. The qualitative examples in Figure 5 provide supporting visual evidence but cannot substitute for rigorous quantitative evaluation.

The paper demonstrates strong methodological rigor for the DOCPAIRBENCH evaluation (three annotators, detailed guidelines in Figure 8, Cohen's Kappa reported in Table 7) but does not extend this rigor to the RL evaluation. This asymmetry is concerning because the RL results are the paper's primary demonstration of practical utility — the claim that DOCREWARD actually improves document generation, not just evaluates it.

Mitigation status. The paper does not address this limitation. It does not acknowledge the lack of statistical detail for the RL human evaluation, nor does it suggest that additional validation is needed. The inclusion of automated metrics (Success rate, ROUGE-L, DOCREWARD Score) provides some triangulation — all three improve when DOCREWARD is added, which is internally consistent with the human ranking improvement — but these metrics are not independent: the DOCREWARD Score is the model's own evaluation, and improvement in it is expected by construction. The ROUGE-L and Success rate improvements show that DOCREWARD-augmented training also improves content preservation and execution reliability, which is evidence that the reward combination does not harm these dimensions. However, the central claim — that humans prefer the formatting of DOCREWARD-trained documents — rests on the human ranking evidence, and that evidence is underspecified.

6.6 The Paper Tests Only a Single Family of Vision-Language Models as the Base Architecture

The assumption. DOCREWARD is built exclusively on the Qwen2.5-VL model architecture (both 3B and 7B variants). The paper's claims about DOCREWARD as an effective reward model are claims about this specific architecture fine-tuned on this specific dataset.

The consequence. A practitioner who wants to deploy DOCREWARD but cannot use Qwen2.5-VL — due to licensing restrictions, infrastructure constraints (the model must run on specific hardware or integrate with a particular inference stack), or organizational policies requiring specific model families — has no guidance on whether the approach transfers. The paper does not demonstrate that the textual-quality-agnostic framework and DOCPAIR dataset can produce an effective reward model when applied to a different base architecture (e.g., LLaVA, InternVL, GPT-4V fine-tuned, or Claude).

More fundamentally, the paper does not disentangle the contribution of the training framework (DOCPAIR data + Bradley-Terry optimization) from the contribution of the base model's visual capabilities. Qwen2.5-VL may possess specific visual processing strengths (or weaknesses) that interact with the DOCPAIR training in unknown ways. If the base model has strong document layout understanding from its pretraining, the DOCPAIR fine-tuning may be primarily teaching it preference ordering rather than visual perception — in which case, starting from a model with weaker layout understanding might produce substantially worse results. Conversely, if Qwen2.5-VL's document understanding is merely average, the DOCPAIR framework might produce even stronger results when applied to a more capable base model. The paper provides no data to distinguish these possibilities.

The ablation in Appendix A.7 (input modality comparison) shows that image-only input outperforms image + OCR for Qwen2.5-VL, but this finding may not generalize — a different base model with a different visual encoder might benefit more from explicit positional features. Architecture-specific findings like this cannot be assumed to transfer without cross-architecture validation.

What evidence exists in the paper. All experiments use Qwen2.5-VL as the base architecture. The paper does not train or evaluate DOCREWARD on any other vision-language model. The baseline comparisons (Table 2) include other model families (GPT-4o, Claude Sonnet 4, GPT-5) but only in zero-shot evaluation mode — these models are not fine-tuned on DOCPAIR to serve as architecture-varied DOCREWARD variants.

Mitigation status. The paper does not address this limitation. It does not discuss architectural sensitivity, does not claim that the approach is architecture-agnostic, and does not suggest cross-architecture validation as future work. The choice of Qwen2.5-VL is motivated in Appendix A.2 only by a reference to the model's technical report, not by comparative analysis showing it is the best or most representative architecture for document reward modeling.

For practitioners, this limitation means that the paper's approach has been validated only for a narrow slice of the possible design space. Adapting DOCREWARD to a different base model may require non-trivial adjustments to training hyperparameters, input preprocessing, or model architecture (regression head design) that the paper does not explore. The 82.3% accuracy figure should be understood as the performance of Qwen2.5-VL-7B + DOCPAIR training, not as the expected performance of the DOCPAIR training framework applied to an arbitrary vision-language model.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a new model architecture or a novel training algorithm — it introduces a diagnostic insight and a data construction methodology that together make a previously intractable problem solvable. The shift it causes is in how the field should think about evaluating and optimizing visual quality in document generation: not as a holistic "is this document good?" judgment, but as a carefully disentangled problem where content quality and formatting quality must be assessed independently, and where the key to independence is the construction of training pairs that hold content constant while varying formatting.

The magnitude of this shift is methodological rather than paradigmatic. The paper does not propose that all reward models should be trained this way, or that the textual-quality-agnostic constraint is universally applicable. Rather, it demonstrates that for a specific, practically important subproblem — evaluating whether a document looks professional — the standard approach of training a model on naturally occurring document pairs (which confound content and formatting quality) is fundamentally limited, and that a carefully constructed synthetic dataset can break this confounding. This is a reframing of the problem: from "we need a better model" to "we need better training data that isolates the signal of interest."

The paper resolves a latent contradiction in the document AI and agentic generation literature. Prior work on document generation focused overwhelmingly on content correctness — can the agent produce text that is factually accurate, logically coherent, and appropriately structured in terms of information organization? The visual presentation was either ignored (the output was evaluated as plain text) or treated as a trivial post-processing step (apply a template, choose default fonts). Yet anyone who has received an automatically generated report knows that the difference between a wall of text and a professionally formatted document is substantial for readability and credibility. The contradiction was: how can the field claim to generate "professional documents" when it cannot evaluate, and therefore cannot optimize, the dimension of professionalism that readers actually see? This paper names that gap explicitly and provides a concrete solution.

The landscape change is most visible in what research directions become attractive versus unattractive:

More attractive after this work:

Specialized reward models for narrow perceptual quality dimensions. The paper demonstrates that a 7B model trained on 117K domain-specific preference pairs can substantially outperform a frontier model likely 1–2 orders of magnitude larger on the specific task of document formatting assessment. This suggests a broader research program: identify quality dimensions that are confounded in natural data (code documentation layout vs. code correctness, slide deck design vs. content quality, data visualization aesthetics vs. data accuracy), construct disentangled training pairs by holding the confounding dimension constant, and train specialized small reward models. The economic argument — smaller, cheaper models providing better signals than general-purpose giants — is compelling if the paper's 14.6-point gap generalizes to other domains.
Data-centric approaches to multimodal reward modeling. The paper's primary contribution is its data construction pipeline (DOCPAIR), not its model architecture or training objective (both of which are standard). This reinforces a broader trend — that in the era of capable pretrained vision-language models, the bottleneck for specialized evaluation tasks is not model capacity but training data that isolates the target signal. The paper provides a template for how to construct such data: curate high-quality reference examples, use agents to generate variants that differ only along the target dimension, and use oracle anchoring (reference-based comparison) to obtain reliable preference labels at scale.
Plug-in reward models as modular components in agentic workflows. The paper demonstrates two integration modes — Best-of-N re-ranking (no agent modification needed) and RL training (agent optimization guided by reward) — and shows that both improve document quality. This establishes a pattern: specialized evaluators can be developed independently of the generation agents they support, and they can be integrated at different levels of the agent pipeline depending on access (black-box API vs. trainable model). This modularity makes the research investment in reward model development more justifiable — the same reward model can serve multiple generation agents and multiple integration modes.

Less attractive after this work:

Relying on zero-shot GPT-5 or similar frontier models as document formatting judges. The paper provides concrete evidence (Table 2, Table 3) that general-purpose multimodal models, even state-of-the-art ones, underperform a modestly sized specialized model on this task by a wide margin. A team building a document generation system that previously used GPT-5 API calls to evaluate formatting quality now has strong empirical motivation to invest in a specialized reward model instead. The 67.7% accuracy (versus DOCREWARD's 82.3%) and the 37.7% Best-of-N win rate (versus DOCREWARD's 60.8%) are not borderline results — they represent a qualitative difference between a judge that is unreliable one-third of the time and one that is reliable more than four-fifths of the time.
Treating document structure and style as an afterthought in generation research. The paper demonstrates that formatting quality can be systematically optimized through RL with a learned reward signal, producing documents that human annotators demonstrably prefer (Table 4 human rankings). This makes it harder to justify generation systems that focus exclusively on textual content while ignoring visual presentation — the tools to evaluate and optimize presentation now exist, and the paper shows they make a measurable difference. Research that evaluates document generation solely on ROUGE scores or content accuracy metrics now faces the obvious criticism that it is ignoring a dimension of quality that readers care about and that can be improved.
Manual document template design as the primary approach to professional formatting. Before DOCREWARD, the default approach to ensuring professional document output was to constrain generation to predefined templates — the agent fills in content slots, and the template handles formatting. While templates remain useful (especially for highly standardized documents like government forms), the paper's RL results suggest that agents can learn general formatting principles from reward signals, potentially producing more flexible and context-appropriate formatting than rigid templates allow. Template-based approaches also require per-domain template design, whereas DOCREWARD generalizes across 32 domains with a single model.

Follow-Up Research This Work Enables

Diagnosing whether DOCREWARD has learned a "realism shortcut" versus genuine formatting assessment. The paper's training data construction relies on the heuristic that human-authored documents always win against synthetic documents in Real-vs-Synthetic pairs. This risks teaching the model to prefer documents that "look like training-set real documents" rather than documents with objectively better formatting. A targeted experiment would construct a test set of human-authored documents with deliberately introduced formatting errors (collapsed margins, inconsistent fonts, missing headers, misaligned elements) paired against carefully formatted synthetic documents that correct those errors. If DOCREWARD consistently prefers the error-containing human documents over the corrected synthetic ones, the realism shortcut hypothesis is confirmed — and the training pipeline would need modification (e.g., mixing in pairs where synthetic documents win when they are objectively better). If DOCREWARD correctly prefers the better-formatted synthetic documents, the heuristic is validated for more than just the 100-sample spot check reported in the paper. This experiment would directly test the paper's most significant unvalidated assumption and provide guidance for future DOCPAIR-style dataset construction.

Measuring the sample efficiency of scalar reward versus diagnostic feedback for RL-based formatting improvement. The paper acknowledges (Appendix A.1) that DOCREWARD's scalar-only output is a limitation — it provides no natural language explanation of formatting problems. The obvious follow-up is to compare how quickly a document generation agent learns to improve formatting when trained with DOCREWARD's scalar score versus a reward model that provides per-element diagnostic feedback ("heading font is too small," "margins are uneven on page 2," "table lacks borders"). The experiment would fix the total number of RL training rollouts and measure the DOCREWARD score achieved by agents trained under each feedback condition. If the diagnostic feedback enables significantly faster learning (e.g., achieving the same formatting quality in 50% fewer rollouts), the case for building an interpretable DOCREWARD variant becomes strong. If learning rates are similar, the scalar output is sufficient, and the practical priority shifts to improving the scalar model's accuracy rather than adding interpretability. This experiment would quantify how much the scalar-only limitation actually matters — the paper currently identifies it as a limitation but provides no evidence about its practical impact.

Cross-architecture validation of the DOCPAIR training framework. The paper trains DOCREWARD exclusively on Qwen2.5-VL. To determine whether the training framework generalizes across architectures, a follow-up study would train DOCPAIR-based reward models on at least two additional vision-language model families — for example, LLaVA-NeXT and InternVL2, at comparable parameter scales (7B–8B). The key measurement is whether the resulting models achieve similar accuracy on DOCPAIRBENCH (within, say, ±5 points of DOCREWARD-7B's 82.3%), or whether performance is strongly architecture-dependent. If all architectures converge to similar accuracy, the framework is robust and practitioners can choose their base model based on deployment constraints (licensing, latency, hardware) rather than worrying about architecture-specific compatibility. If some architectures substantially underperform, the paper's findings are architecture-bound rather than framework-bound, and follow-up work would need to diagnose what architectural properties (visual encoder resolution, pretraining data composition, attention mechanism) matter for document formatting assessment.

Data scaling analysis: how many training pairs are necessary for DOCREWARD-level performance? The paper trains DOCREWARD on 117K document pairs without ablating the effect of dataset size. A practical follow-up would train DOCREWARD variants on random subsets of DOCPAIR at sizes of 1K, 5K, 20K, 50K, and 117K pairs, measuring accuracy on DOCPAIRBENCH at each size. The resulting scaling curve would answer two questions. First, what is the marginal value of additional training pairs? — if accuracy saturates at 20K pairs, the DOCPAIR construction effort was larger than necessary, and practitioners in other domains can target smaller datasets. Second, what is the minimum viable dataset size for a useful reward model? — if 5K pairs already achieve, say, 75% accuracy (still above GPT-5's 67.7%), then the approach is feasible for niche domains where collecting 117K pairs is impractical. The paper currently presents the 117K figure as a data point but provides no information about whether it reflects necessity or abundance.

Evaluation on out-of-distribution document lengths and image densities. The DOCPAIR training pipeline excludes documents exceeding 20 pages and image-heavy documents (files >1 MB dominated by images, Section 3.1). DOCPAIRBENCH, drawn from the same filtered distribution, does not test generalization along these axes. A stress-test experiment would construct a held-out evaluation set stratified by document length (1 page, 3 pages, 10 pages, 20 pages, 40 pages) and image density (text-only, 25% images, 50% images, image-dominated), with human preference labels for each pair. Evaluating DOCREWARD on this set would reveal whether its performance degrades systematically as documents deviate from the training distribution's center (average 3.2 pages, text-dominated). If accuracy on 40-page documents drops to near-chance levels, the model's practical utility for long-form document generation (the primary use case for Deep Research-style agents) is severely limited despite strong benchmark results. If accuracy remains stable across lengths and image densities, the filtering criteria in DOCPAIR construction can be relaxed in future iterations without fear of degrading model quality.

Combining DOCREWARD with content-quality reward models for holistic document optimization. The paper deliberately isolates structure/style from content quality. In practice, a document generation system needs to optimize both dimensions simultaneously. A natural follow-up would train or adapt a separate content-quality reward model (e.g., a model that evaluates factual accuracy, logical coherence, and writing quality) and combine it with DOCREWARD in a multi-objective RL setup for document generation. The key experimental question is whether optimizing these two dimensions jointly produces trade-offs (e.g., the agent learns to sacrifice content accuracy for better formatting, or vice versa) or synergies (improving formatting also improves perceived content quality, or improving content organization makes formatting easier). The combination could use a weighted sum reward R_total = w_content * R_content + w_formatting * R_formatting and sweep the weight w_formatting to map out the Pareto frontier of content-quality versus formatting-quality trade-offs. This experiment would transform DOCREWARD from a standalone tool into a component of a more complete document quality optimization system, directly addressing the paper's stated motivation of improving agentic document generation workflows.

Practical Applications and Downstream Use Cases

Quality filtering for agentic deep research and report generation systems. Systems like OpenAI's Deep Research, open-source alternatives like OpenManus, and various technical documentation generators produce multi-page reports from user queries. These systems typically generate a single output with no quality filtering on visual presentation. Integrating DOCREWARD as a Best-of-N re-ranker — generate 8 candidate reports, score each with DOCREWARD, return the highest-scoring one — would directly improve the visual professionalism of user-facing outputs. The paper's Best-of-N results (Table 3) demonstrate that DOCREWARD-selected documents achieve a 60.8% win rate against GPT-5-selected documents (37.7%) and random selection (24.6%), indicating that the re-ranking step substantially improves formatting quality without modifying the underlying generation agent. The implementation cost is modest: the agent must generate multiple candidates (already feasible with non-zero temperature sampling), and DOCREWARD must score each candidate (the 7B model can run on a single GPU). For systems where user-facing document quality directly impacts perceived product quality and user trust, the integration is low-risk and demonstrably beneficial.

Training data filtering for document generation model fine-tuning. When fine-tuning a document generation model on synthetic or human-authored examples, the quality of the training data determines the quality of the resulting model. A dataset of 100K generated documents likely contains substantial variation in formatting quality — some examples use professional layouts while others are amateurish. Using DOCREWARD to score all training examples and filter to the top-scoring quartile (or to remove the bottom-scoring decile) before fine-tuning would improve the average formatting quality of the training data without requiring human review. The paper's accuracy numbers (82.3% on DOCPAIRBENCH) provide confidence that DOCREWARD's scoring aligns well enough with human preferences to serve as a reliable filter. This is analogous to how perplexity-based filtering or reward-model-based filtering is used for text data — extended to the visual domain for document generation. The 3B model variant (80.6% accuracy) would likely suffice for coarse filtering, offering low inference cost for processing large datasets.

Automated formatting quality assurance in document-intensive industries. Legal, financial, government, and academic organizations produce large volumes of documents where formatting professionalism is not merely aesthetic but often a regulatory or professional requirement — contracts with inconsistent formatting may be questioned, grant proposals with amateurish layout may be penalized, government forms with misaligned fields may be rejected. DOCREWARD can serve as an automated first-pass quality check: score every outgoing document, flag any document scoring below a threshold (determined by calibration on a sample of organization-specific documents) for human review. The paper's cross-domain generalization (Table 5, only 4.8-point degradation on unseen domains) and cross-lingual robustness (Table 6, only 4.4-point degradation on non-English documents) suggest that DOCREWARD can be deployed across diverse document types and languages without retraining, though domain-specific threshold calibration would be advisable. The scalar output is a limitation for this use case — a flagged document receives only a low score, not an explanation of what is wrong — but even an unexplained flag is valuable if it directs human attention to documents most likely to have formatting issues. The cost savings come from reducing the human review burden: instead of reviewing all documents, reviewers focus on the small fraction that DOCREWARD flags as potentially unprofessional. The paper's accuracy figures suggest this triage would be effective: with 82.3% accuracy on preference pairs, a well-calibrated threshold should catch most genuinely unprofessional documents while producing an acceptable false-positive rate.