Adapting Large Language Models for Document-Level Machine Translation

URL: https://arxiv.org/pdf/2401.06468v3.pdf

🎯 Pitch

Fine-tuning a 7B LLM can beat GPT-4 on document-level translation, but a shocking failure mode emerges: up to 98% of outputs are in the wrong language due to error propagation during decoding, where a single mistake poisons the entire document.

1. Executive Summary

This study empirically analyzes how to adapt moderately-sized large language models (7B parameters) for document-level machine translation (DOCMT) through task-specific fine-tuning. Using three LLM backbones — LLAMA2-7B, BLOOM-7B, and VICUNA-7B — fine-tuned via both Parameter-Efficient Fine-Tuning (PEFT) (specifically LoRA, which updates only ~8M parameters) and Fully Fine-Tuning (FFT) across 18 translation tasks on the IWSLT2017 benchmark, the paper investigates a two-stage training strategy (monolingual document fine-tuning followed by parallel document fine-tuning) and four prompt designs, finding that Prompt 4 — which interleaves source-target sentence pairs with a natural language instruction — yields the best overall performance, and that LoRA generally outperforms FFT while FFT demonstrates greater data efficiency, reaching full-dataset performance with only ~1% of the training data versus ~10% for LoRA. The specialized models can surpass GPT-4-TURBO on certain translation tasks (B-7B-LORA achieves 29.9 sBLEU on X→En tasks), but suffer from severe off-target translation due to error propagation during autoregressive decoding with reused context (off-target rates reaching up to 98.3% for V-7B-FFT on Korean→English), establishing that the gains from fine-tuning LLMs for document-level translation are substantial but critically dependent on mitigating decoding-time error propagation, particularly when previous translations serve as context for subsequent sentences.

2. Context and Motivation

The Core Problem: Adapting LLMs for Document-Level Translation Is Poorly Understood

The fundamental question this paper tackles is: can moderately-sized large language models (LLMs) be effectively adapted for document-level machine translation (DOCMT) through task-specific fine-tuning, and if so, under what conditions do they succeed or fail?

This matters because the translation landscape has been dramatically reshaped by two parallel developments. On one side, specialized encoder-decoder translation models — exemplified by NLLB (Costa-jussà et al., 2022) — have achieved state-of-the-art performance through supervised training on massive parallel corpora, but they require dedicated architecture design (multi-encoder setups, specialized attention mechanisms, translation caches) to handle document-level context. On the other side, general-purpose LLMs like GPT-3.5-TURBO and GPT-4-TURBO have demonstrated remarkable few-shot translation capabilities without any task-specific training, but they are prohibitively large (hundreds of billions of parameters), expensive to deploy, and underperform on low-resource languages (Robinson et al., 2023; Jiao et al., 2023; Hendy et al., 2023).

The gap this paper identifies is precisely the middle ground: moderately-sized LLMs (7B parameters) that are small enough to fine-tune and deploy practically, yet potentially powerful enough to capture document-level discourse phenomena when properly adapted. Prior to this work, there was no systematic study of how to adapt such models for DOCMT — which fine-tuning methods work, which backbone architectures suit the task, how prompts affect training dynamics, and what failure modes emerge.

Why Document-Level Translation Demands Special Attention

Document-level MT differs fundamentally from sentence-level MT in ways that make it both more important and more challenging. At the sentence level, each source sentence is translated independently, with no access to surrounding context. This creates well-documented problems that degrade translation quality in real-world applications:

• Pronoun resolution: Languages differ in how they mark gender, number, and formality on pronouns. A sentence-level translator seeing "The doctor entered the room. She sat down." cannot know from the first sentence alone that the doctor is female, leading to incorrect pronoun translation in languages with grammatical gender (Müller et al., 2018; Voita et al., 2018).
• Lexical cohesion: Documents maintain consistent terminology across sentences. A sentence-level model might translate the same source term differently in different sentences, creating confusing inconsistency for the reader (Voita et al., 2019).
• Discourse structure: Ellipsis (omitted information recoverable from context), deixis (words requiring contextual reference like "this" or "here"), and discourse connectives all depend on cross-sentence relationships that sentence-level models cannot access.

The practical stakes are high: any translation system deployed for real documents — whether TED talks (as in the IWSLT2017 benchmark this paper uses), legal contracts, scientific papers, or news articles — must handle these discourse phenomena to produce coherent, professional output. The IWSLT2017 dataset, with approximately 1.9K sentence-aligned parallel documents across nine language pairs, captures exactly this domain: spoken presentations where ideas flow across sentence boundaries and pronoun reference, terminology consistency, and discourse coherence are essential.

Where Prior Approaches Fall Short

The paper identifies specific limitations in existing work along several axes:

Specialized DOCMT models require dedicated architectures. The traditional approach to DOCMT involves building purpose-specific model architectures: concatenating previous sentences as context (Tiedemann and Scherrer, 2017), using separate encoders for context and current sentence (Bawden et al., 2018; Zhang et al., 2018), modifying attention patterns to attend across sentence boundaries (Miculicich et al., 2018; Maruf et al., 2019), or maintaining translation caches that store prior translations (Maruf and Haffari, 2018; Feng et al., 2022). While effective, these methods require explicit architectural decisions — each new approach modifies the model in a different way — and they are typically applied to encoder-decoder frameworks like Transformer, not to the decoder-only architectures that dominate modern LLMs. The paper's re-implemented DOCMT baselines (DOC2DOC-MT5, MR-DOC2SEN-MT5, DOCFLAT-MT5, IADA-MT5) represent strong instantiations of these approaches using MT5 as the backbone, achieving µsBLEU scores in the 19–22 range for X→En translation (Table 2). These provide the performance floor that LLM-based methods must beat.

GPT-scale LLMs address DOCMT through prompting, but are impractical for deployment. The most directly relevant prior work is Wang et al. (2023b), who study GPT-3.5-TURBO's document-level translation capabilities through careful prompt engineering. They show that large LLMs can achieve competitive DOCMT performance simply through prompting — GPT-4-TURBO achieves 31.7 µsBLEU on X→En in this paper's evaluation (Table 2), substantially outperforming all specialized models. However, this approach has severe practical limitations: GPT-4-TURBO is a massive proprietary model accessible only through paid API calls, making it unsuitable for many deployment scenarios (offline use, low-latency requirements, cost-sensitive applications, data privacy constraints). Moreover, the study by Wang et al. (2023b) focuses exclusively on inference-time prompting, leaving open the question of whether fine-tuning smaller open LLMs could bridge the gap.

Sentence-level LLM fine-tuning exists, but document-level adaptation is unexplored. A growing body of work explores fine-tuning LLMs for translation, but it remains confined to the sentence level. Xu et al. (2023) proposed a paradigm shift — using a two-stage training strategy (monolingual fine-tuning then parallel sentence fine-tuning) to boost LLM translation performance, demonstrating that fine-tuned LLMs can compete with supervised NMT models on sentence-level benchmarks. Similarly, Zhu et al. (2023), Yang et al. (2023), and Zhang et al. (2023) explore multilingual translation with fine-tuned LLMs. None of these efforts extend to document-level translation, where the context window must span multiple sentences and the training objective must account for discourse coherence. The paper positions itself to fill exactly this gap: taking the two-stage training paradigm from Xu et al. (2023) and extending it to the document level.

The interaction between prompt design and fine-tuning is unknown for DOCMT. Prompt engineering is a central concern in LLM research — optimal prompts can dramatically improve performance or reveal latent capabilities (Kojima et al., 2022; Wei et al., 2022b). For translation, Wang et al. (2023b) investigated how prompting strategies affect GPT-3.5-TURBO and GPT-4-TURBO at inference time. But the effect of prompt design during fine-tuning — when the model is trained to internalize the prompt structure — is a different question entirely. A prompt that works well for few-shot inference with a frozen model may be suboptimal when baked into the training data. The paper's preliminary study in Section 4 directly investigates this question, comparing four prompt variations (Figure 1) that differ in context structure (separated context blocks vs. interleaved sentence pairs) and the presence of natural language instructions. This investigation is motivated by the hypothesis that document-level translation, with its need to represent multi-sentence context coherently, may be particularly sensitive to how that context is formatted in the prompt.

How This Paper Positions Itself

The paper frames its contribution as a systematic empirical analysis rather than a novel method proposal. It positions itself at the intersection of two active research directions — document-level MT and LLM fine-tuning — each of which has been studied independently, but whose combination raises questions that neither field has answered:

From the DOCMT perspective: The paper asks whether decoder-only LLMs can replace the specialized encoder-decoder architectures that have dominated document-level translation research. The answer, as revealed in the main results (Table 2), is nuanced: LLM-based DOCMT models can match or exceed conventional DOCMT models on translation into English (L-7B-LORA achieves 23.8 µsBLEU vs. 21.5 for DOC2DOC-MT5-1.2B), but underperform on translation out of English (17.2 vs. 19.2). This suggests that LLMs capture discourse phenomena differently — perhaps more effectively for some language directions — and that the choice between specialized and general-purpose architectures is not a simple win for either approach.

From the LLM fine-tuning perspective: The paper investigates whether instruction-tuned base models (VICUNA-7B, fine-tuned from LLAMA2-7B) provide advantages over raw base models (LLAMA2-7B) for DOCMT fine-tuning. This is a non-obvious question: instruction-tuned models have already been optimized for following prompts and generating coherent output, which seems aligned with the DOCMT task. Yet the results in Table 2 show that VICUNA-7B variants generally underperform LLAMA2-7B variants — the additional instruction training does not translate to better translation performance and may even interfere with learning the specific prompt structure of the translation task. This finding challenges the assumption that instruction-tuned models are always better starting points for downstream fine-tuning.

The two-stage training strategy (Section 3.1) is a direct extension of Xu et al. (2023) but adapted for documents: Stage 1 fine-tunes all model parameters on monolingual documents in target languages (1B tokens total, pruned from CulturaX using the technique of Marion et al., 2023), addressing the English-centric bias of existing LLMs. Stage 2 fine-tunes on parallel documents with three preceding sentence pairs as context (following Wang et al., 2023a). The paper explicitly investigates whether this two-stage approach is optimal by comparing it to a one-stage strategy (direct fine-tuning on parallel documents) and a three-stage strategy (adding an intermediate parallel sentence fine-tuning stage). The finding that two-stage training dominates (Section 6, Table 6) suggests that the monolingual stage improves target-language representations while the parallel document stage teaches translation with context — and that adding a sentence-level intermediate stage provides no additional benefit, likely because the two-stage approach already captures the necessary sentence-to-document progression.

The discovery of off-target translation as a failure mode represents the paper's most practically significant finding. As shown in Table 3, off-target rates reach catastrophic levels for certain language pairs: 98.3% for V-7B-FFT on Korean→English, 93.1% for L-7B-LORA on Chinese→English. The paper traces this to error propagation during autoregressive decoding with context reuse: when translating a document sentence-by-sentence, the model uses its own previous translations as context (the REUSE strategy in Section 3.4). If an early sentence is mistranslated (or produced in the wrong language), that erroneous context corrupts all subsequent translations. This is a uniquely document-level problem — sentence-level translation has no such dependency chain. The paper demonstrates the causal link by introducing an alternative REGEN strategy where all context translations are regenerated from scratch, which dramatically reduces off-target rates (e.g., L-7B-LORA on Arabic→English improves from 3.9 sBLEU to 17.5, Table 4) but at $4\times$ inference cost. This finding reframes the DOCMT challenge: the primary obstacle is not the model's inability to produce good translations (when given clean context, it does), but rather the fragility of the autoregressive document-level decoding pipeline.

The relationship between model backbones is carefully chosen for diagnostic insight: LLAMA2-7B (English-centric pre-training), BLOOM-7B (multilingual pre-training on 46 languages), and VICUNA-7B (instruction-tuned from LLAMA2-7B). This triplet allows the paper to disentangle the effects of pre-training language coverage from instruction tuning. The consistent finding that BLOOM-7B maintains the lowest off-target rates (only 2.8% on average for X→En with LoRA, Table 3) while sometimes underperforming on standard metrics suggests that multilingual pre-training provides robustness against language confusion during autoregressive decoding — a benefit that is invisible when looking only at evaluation metrics on clean, independent test sentences, but becomes critical in the chained document-level decoding setting.

The Practical Motivation

Beyond the academic framing, the paper is motivated by a clear practical scenario: organizations that want to deploy document-level translation systems but cannot rely on massive, proprietary, API-based models like GPT-4. This could include:

• Offline or air-gapped environments where API calls are impossible.
• Cost-sensitive applications processing millions of documents.
• Privacy-sensitive domains (medical records, legal documents) that cannot be sent to external services.
• Low-resource language pairs where general-purpose LLMs underperform and specialized models must be built.

For these scenarios, a 7B-parameter model that can be fine-tuned on available parallel documents — possibly with as little as ~2K examples (1% of IWSLT2017) for full fine-tuning, as shown in Figure 4 — represents a viable alternative to both the architectural complexity of specialized DOCMT models and the scale requirements of GPT-class LLMs. The paper's scaling law analysis directly addresses the practical question: "How much parallel document data do I need to collect?"

Reconciling Conflicting Signals in the Literature

The paper enters a field with competing claims. On one hand, recent work like Gunasekar et al. (2023), Luo et al. (2023), and Azerbayev et al. (2023) suggests that "smaller, specialized models can outperform larger, general-purpose models in specific tasks" — a finding the paper explicitly cites in its introduction. On the other hand, the translation literature has shown that only very large LLMs (GPT-3.5-TURBO and above) can match supervised NMT systems, and even then they struggle on low-resource languages. The paper's results partially reconcile these claims: specialized 7B models can surpass GPT-4 on some tasks (B-7B-LORA on Arabic→English: 30.9 sBLEU vs. GPT-4-TURBO's 31.7 µsBLEU average, but exceeding it on specific pairs as shown in Figure 2), but they can also fail catastrophically due to off-target translation — a failure mode that GPT-4, with its massive scale and broad language coverage, largely avoids. The resolution is not a simple "smaller specialized models are better" but rather "smaller specialized models can be better when the decoding pipeline is robust to error propagation, and worse when it is not."

This framing — success conditional on mitigating a specific, identifiable failure mode — is more actionable than blanket comparisons. It tells future researchers that improving LLM-based DOCMT requires not just better training recipes but also better decoding strategies (like REGEN or, ideally, methods with lower overhead) that break the error propagation chain.

3. Technical Approach

3.1 Reader Orientation

This paper builds a specialized document-level machine translation (DOCMT) system by taking a generic 7B-parameter language model and training it in two stages — first on monolingual documents to strengthen its target-language representations, then on parallel documents formatted with a specific prompt structure — so that it can translate entire documents sentence by sentence while using its own previous translations as context. The core problem is that off-the-shelf LLMs either require prohibitive scale (GPT-4) to handle document-level discourse phenomena, or produce catastrophic off-target translations when smaller models are applied to the chained, context-dependent decoding that document translation requires; the solution involves carefully choosing the fine-tuning method, prompt design, and decoding strategy to balance translation quality against error propagation.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components, arranged in a pipeline:

1. Base LLM backbone — one of LLAMA2-7B (English-centric pre-training), BLOOM-7B (multilingual pre-training on 46 languages), or VICUNA-7B (instruction-tuned from LLAMA2-7B). This provides the raw generative capability that will be shaped by fine-tuning. The choice of backbone determines the starting language coverage and affects off-target translation robustness.

2. Monolingual fine-tuning stage — all model parameters are updated on 100M tokens per target language (10 languages, 1B tokens total) drawn from the CulturaX corpus and pruned using the technique of Marion et al. (2023). This addresses the English-centric bias of the base LLMs and strengthens representations for the target languages.

3. Parallel document fine-tuning stage — the model is trained on document-level parallel corpora from IWSLT2017, where each training example consists of the current source sentence, its translation, and the three preceding sentence pairs as context, all formatted according to a specific prompt template (Prompt 4, Figure 1d). This stage uses either full fine-tuning (all 7B parameters) or LoRA (only ~8M parameters, 0.1% of total). The training objective is standard autoregressive next-token prediction on the target-language tokens only.

4. Prompt formatting layer — the prompt template (Figure 1d) that structures how context and instructions are presented to the model. It interleaves source and target sentences as parallel pairs, followed by a natural language instruction ("Given the provided parallel sentence pairs, translate the following <src_lang> sentence to <tgt_lang>:"), followed by the current source sentence and the target sentence prefix. This is fixed during both training and inference.

5. Autoregressive document decoder — at inference time, documents are translated in their original order using beam search (beam size 5). The model's own previous translations are fed back as context for subsequent sentences, creating a chain where each translation depends on the quality of all prior translations. This is the REUSE strategy; the alternative REGEN strategy regenerates all context translations from scratch, breaking the dependency chain but at $4\times$ cost.

Information flows as follows during inference: the document's first sentence (no context) enters the prompt formatter → the fine-tuned LLM generates a translation via beam search → this translation is stored → for sentence 2, the stored translation of sentence 1 is inserted into the prompt as context → the LLM generates sentence 2's translation conditioned on this context → the chain continues through the full document. At each step, the prompt includes exactly three preceding source-target sentence pairs (or fewer for early sentences), formatted as interleaved parallel pairs with the instruction.

3.3 Roadmap for the Deep Dive

• First, the two-stage training strategy (Section 3.1) — why monolingual fine-tuning matters, what data is used, and how parallel document fine-tuning is structured. This is the foundation: everything else depends on how the model is trained.
• Second, the datasets (Section 3.2) — what IWSLT2017 and the monolingual corpus contain, their sizes, and why they were chosen. Understanding the data is essential for interpreting the scaling law and transfer results.
• Third, the model backbones and baselines (Section 3.3) — the three LLM starting points, the comparison systems (NLLB, GPT-3.5/4-TURBO, re-implemented DOCMT models), and what each baseline controls for.
• Fourth, the prompt design space (Section 4, analyzed as a preliminary study) — the four prompt variations in Figure 1, what research questions they address, and why Prompt 4 emerges as optimal. This is where the format of training data is decided.
• Fifth, the fine-tuning mechanics — LoRA vs. FFT, hyperparameter choices, and what "parameter-efficient" means concretely for a 7B model.
• Sixth, the decoding strategies — REUSE vs. REGEN, beam search configuration, and how inference-time choices interact with training-time design to produce (or prevent) off-target translation.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical analysis paper whose core idea is that adapting moderately-sized LLMs for document-level translation requires careful coordination of training strategy, prompt design, and decoding procedure, and that the dominant failure mode — off-target translation — is an artifact of error propagation during autoregressive context reuse, not a fundamental limitation of the models' translation capability.

Two-Stage Training Strategy

The paper adopts a two-stage training pipeline, directly extending the sentence-level paradigm of Xu et al. (2023) to the document level. The intuition is that fine-tuning an LLM for DOCMT requires solving two sub-problems sequentially: first, ensuring the model has strong representations of the target languages (since most LLMs are English-centric), and second, teaching the model to use document-level context during translation.

Stage 1: Monolingual document fine-tuning. All parameters of the LLM are updated on monolingual text in the target languages. The training data consists of 100M tokens per language for each of the ten languages involved in the translation tasks (Arabic, German, French, Italian, Japanese, Korean, Dutch, Romanian, Chinese, plus English), totaling 1B tokens. These tokens are drawn from the CulturaX corpus (Nguyen et al., 2023), a large-scale multilingual dataset, and are pruned using the data pruning technique of Marion et al. (2023). The pruning step is motivated by computational constraints and the need to avoid catastrophic forgetting: fine-tuning on too much monolingual data could overwrite the model's general language understanding. The learning rate is set to $5 \times 10^{-5}$, batch size 256, with a linear learning rate schedule including a warm-up phase comprising 10% of total training steps. The objective is standard autoregressive language modeling: predict the next token given previous tokens, with loss computed only on the target language tokens. This stage applies to all three backbones (LLAMA2-7B, BLOOM-7B, VICUNA-7B) identically.

The design choice to fine-tune all parameters rather than use parameter-efficient methods at this stage reflects the authors' judgment that the English-centric bias requires substantial model-wide adjustment, not just adaptation of a small subset of weights. The 100M-token-per-language budget (with pruning for quality) balances coverage against training cost and catastrophic forgetting risk.

Stage 2: Parallel document fine-tuning. The model is fine-tuned on document-level parallel corpora from IWSLT2017. Each training example is formatted as: three preceding source-target sentence pairs (the context), followed by a natural language instruction, followed by the current source sentence and its target translation. Concretely, using Prompt 4 (Figure 1d):

[<src_lang>]: <src1> [<tgt_lang>]: <tgt1>
[<src_lang>]: <src2> [<tgt_lang>]: <tgt2>
[<src_lang>]: <src3> [<tgt_lang>]: <tgt3>
Given the provided parallel sentence pairs, translate the following
<src_lang> sentence to <tgt_lang>:
[<src_lang>]: <src4> [<tgt_lang>]: <tgt4>

where <src_lang> and <tgt_lang> are language identifiers (e.g., "English" and "German"), <src1>–<src4> and <tgt1>–<tgt4> are the source and target sentences. During training, all target sentences <tgt*> are present, and the model is trained via standard next-token prediction on the target language tokens (including the context target sentences and the current target sentence). The context window of three preceding sentence pairs follows Wang et al. (2023a). Sentences earlier in the document than the three most recent are not included — this is a fixed-length context window, not full-document context.

During Stage 2, two fine-tuning variants are explored:

• Full Fine-Tuning (FFT): All 7B parameters are updated. Learning rate $5 \times 10^{-5}$, batch size 64, linear learning rate schedule with 10% warm-up. Models are fine-tuned for up to 3 epochs with early stopping based on validation loss.

• Parameter-Efficient Fine-Tuning via LoRA (Low-Rank Adaptation, Hu et al., 2022): Only low-rank adaptation matrices are trained, with rank set to 16. This affects approximately 8M parameters — 0.1% of the total 7B. Learning rate $5 \times 10^{-5}$, batch size 64, same schedule and early stopping as FFT. LoRA works by inserting trainable low-rank matrices $A$ and $B$ into the weight updates for attention layers, such that the effective weight matrix becomes $W + AB$ where $W$ is frozen, $A \in \mathbb{R}^{d \times r}$, $B \in \mathbb{R}^{r \times d}$, and $r = 16$ is the rank. During training, gradients flow only through $A$ and $B$; the original weights $W$ remain unchanged.

The paper also experiments with variations of this two-stage approach in the analysis (Section 6, Table 6):

• One-stage: Skip Stage 1 entirely; directly fine-tune the base LLM on parallel documents. This tests whether monolingual pre-fine-tuning is necessary.

• Three-stage: Add an intermediate stage between Stages 1 and 2, where the model is fine-tuned on parallel sentences (not documents). This tests whether a sentence-level intermediate step helps the model bridge from monolingual representation learning to document-level translation.

The finding that two-stage is optimal (Table 6) suggests that the monolingual stage provides necessary target-language grounding, while the sentence-level intermediate stage adds no benefit — the model can learn to handle documents directly after the monolingual warm-up, without needing a sentence-level stepping stone.

Why this training strategy: The two-stage design separates language capability building (Stage 1) from translation skill acquisition (Stage 2). The monolingual stage addresses the well-documented problem that LLMs pre-trained on English-centric corpora underperform on multilingual benchmarks (Li et al., 2023; Chen et al., 2023). By fine-tuning on target-language monolingual text first, the model strengthens its representations for those languages before being asked to produce translations in them. This is particularly important for the decoder-only LLM architecture, where the model must generate target-language text autoregressively — weak target-language representations would manifest as disfluent output or language confusion (the off-target problem). The parallel document stage then teaches the model to use context — the three preceding sentence pairs — to produce coherent, context-aware translations. The fixed three-sentence context window is a practical compromise: full-document context would be ideal but computationally expensive and potentially introduces noise from distant, irrelevant sentences; three sentences captures local discourse phenomena (pronoun reference, lexical cohesion) that span typical adjacent-sentence distances in TED talk transcripts.

Datasets

Parallel documents: IWSLT2017. The primary training and evaluation data comes from IWSLT2017 (Cettolo et al., 2017), a collection of TED talk transcripts translated between English and nine other languages: Arabic (Ar), German (De), French (Fr), Italian (It), Japanese (Ja), Korean (Ko), Dutch (Nl), Romanian (Ro), and Chinese (Zh). This creates 18 translation tasks (9 En→X, 9 X→En). The dataset contains approximately 1.9K sentence-aligned parallel documents per language pair, with about 240K sentences total per pair. The train/validation/test splits are provided by the dataset: training sets range from 206K–237K sentences (1705–1920 documents), validation sets contain approximately 2.4K–2.8K sentences (19 documents), and test sets contain approximately 1.1K–1.5K sentences (10–12 documents). The full statistics are in Table 9 (Appendix A).

The choice of IWSLT2017 is deliberate: TED talks represent a domain where document-level context matters — speakers develop arguments across sentences, use pronouns that require cross-sentence resolution, and maintain consistent terminology. The dataset is medium-sized (~200K training sentences per language pair), making it feasible for full fine-tuning of 7B models while being large enough to study scaling behavior. The nine language pairs span diverse language families (Semitic, Germanic, Romance, Japonic, Koreanic, Sinitic) and varying degrees of relatedness to English, enabling analysis of how language properties interact with the DOCMT approach.

Monolingual documents: CulturaX. For Stage 1, monolingual documents are gathered from CulturaX (Nguyen et al., 2023), a large-scale cleaned multilingual corpus covering 167 languages. The authors select 100M tokens for each of the ten languages (English + nine target languages), totaling 1B tokens. The data pruning technique of Marion et al. (2023) is applied to select high-quality, diverse subsets rather than using random sampling. The pruning criterion is based on the observation that not all training data contributes equally to model quality; Marion et al.'s method identifies and retains examples that are most informative for language modeling, measured through reference model perplexity and other signals. The 100M-token-per-language budget is a computational compromise: it is large enough to meaningfully shift the model's language distribution (given that the base models were trained on hundreds of billions of tokens, 100M is a small but non-trivial fraction) while keeping the total Stage 1 training cost manageable.

Model Backbones and Baselines

LLM backbones (the starting points for fine-tuning):

• LLAMA2-7B (Touvron et al., 2023b): A 7-billion-parameter decoder-only transformer pre-trained predominantly on English text. This represents the "English-centric base model" condition, testing how well an English-focused LLM can be adapted to multilingual document translation through fine-tuning.

• BLOOM-7B (Scao et al., 2022): A 7-billion-parameter decoder-only transformer pre-trained on multilingual text spanning 46 languages. This represents the "multilingual base model" condition, testing whether multilingual pre-training provides advantages for DOCMT (particularly for avoiding off-target translation by already having strong target-language representations).

• VICUNA-7B (v1.5): An instruction-tuned variant of LLAMA2-7B, fine-tuned to follow user instructions in a conversational format. This represents the "instruction-tuned base model" condition, testing whether the instruction-following capabilities acquired during instruction tuning transfer positively to the structured prompt format of DOCMT, or whether they interfere with task-specific learning.

These three backbones form a $2 \times 2$ design (English vs. multilingual pre-training, base vs. instruction-tuned) with the BLOOM comparison being English-centric (LLAMA2) vs. multilingual (BLOOM), and the VICUNA comparison being base (LLAMA2) vs. instruction-tuned (VICUNA), though note that VICUNA is built on LLAMA2, not BLOOM, so it is not a fully crossed design.

Fine-tuned variants (the models the paper produces):

The paper produces six models from these backbones: L-7B-LORA (LLAMA2 + LoRA), L-7B-FFT (LLAMA2 + full fine-tuning), B-7B-LORA (BLOOM + LoRA), B-7B-FFT (BLOOM + full fine-tuning), V-7B-LORA (VICUNA + LoRA), V-7B-FFT (VICUNA + full fine-tuning). All undergo the same two-stage training procedure. The naming convention is {Backbone}-{Size}-{FineTuningMethod}.

Baseline categories (the comparison points):

State-of-the-art sentence-level MT (SENMT) models:

• NLLB (No Language Left Behind; Costa-jussà et al., 2022): A supervised encoder-decoder translation model available in three sizes — 600M, 1.3B, and 3.3B parameters. NLLB represents the best available sentence-level specialized translation systems. Its presence in the baseline set controls for the document-level aspect: if the LLM-based DOCMT models cannot outperform a sentence-level model, the document-level context is providing no benefit (or the model is failing to use it).
• Google Translate: A commercial production translation system. This provides a real-world deployment baseline.

State-of-the-art LLMs for DOCMT:

• GPT-3.5-TURBO and GPT-4-TURBO: The largest available general-purpose LLMs, evaluated using Prompt 4 (Figure 1d) in a zero-shot/few-shot setting (no task-specific fine-tuning). These represent the "scale solves everything" baseline, testing how far moderately-sized fine-tuned models can approach or surpass massive general-purpose models.

Re-implemented DOCMT baselines: The paper implements several established DOCMT architectures using MT5 (Xue et al., 2021) as the backbone, available in three sizes (300M, 580M, 1.2B):

• DOC2DOC-MT5 (Tiedemann and Scherrer, 2017): The simplest DOCMT approach, which concatenates the three preceding source sentences with the current source sentence (separated by a special separator token) and feeds this concatenated string to the encoder. The decoder generates the target translation. This is the "concatenation baseline" — no special architecture, just extended input.

• MR-DOC2SEN-MT5 (Sun et al., 2022): A multi-resolution approach that trains the model to translate both at the sentence level and document level, using document-level context during training but allowing sentence-level decoding. This tests whether multi-task training with sentence-level objectives improves document translation.

• MR-DOC2DOC-MT5 (Sun et al., 2022): The fully document-level variant of the multi-resolution approach. Results are only available for dBLEU (Table 2), as the approach is specifically designed for document-level metrics.

• DOCFLAT-MT5 (Wu et al., 2023): A method that "flattens" the document structure by reordering sentences to bring contextually related sentences closer together before concatenation. This tests whether smart context selection beats naive concatenation.

• IADA-MT5 (Wu et al., 2024c): An importance-aware data augmentation approach that identifies and augments training examples where document context is most critical. This tests whether data-centric improvements transfer to the DOCMT setting.

These baselines establish the performance range of specialized DOCMT models using encoder-decoder architectures (MT5). They provide the comparison point for the central question: can decoder-only LLMs, fine-tuned for DOCMT, match or exceed purpose-built encoder-decoder DOCMT systems?

Baseline evaluation protocol: All LLM-based models (both the fine-tuned variants and the GPT baselines) use Prompt 4 (Figure 1d) during inference. The re-implemented DOCMT baselines use beam search with the same beam size (5) where applicable. GPT-3.5-TURBO and GPT-4-TURBO are accessed via API with model signatures gpt-3.5-turbo-1106 and gpt-4-1106-preview.

Prompt Design Space (Preliminary Study, Section 4)

Before committing to a prompt format for the main experiments, the paper conducts a preliminary study comparing four prompt variations (Figure 1). The study uses three LoRA-fine-tuned models (L-7B-LORA, B-7B-LORA, V-7B-LORA) on four English-centric translation tasks involving German and Chinese. This design tests two independent research questions:

Research question 1: How does context structure impact translation quality? This is tested by comparing Prompt 1 (separate context blocks for source and target) vs. Prompt 2 (interleaved source-target sentence pairs):

• Prompt 1 (Figure 1a): Source context sentences are grouped together under a [<src_lang> Context] header, and target context sentences are grouped under a [<tgt_lang> Context] header. The current sentence pair follows under [<src_lang> Sentence] and [<tgt_lang> Sentence] headers. This separates source context from target context, requiring the model to align them implicitly.

• Prompt 2 (Figure 1b): Each context sentence pair is presented as [<src_lang>]: <src_i> [<tgt_lang>]: <tgt_i> on its own line, with pairs interleaved. The current pair follows the same format. This aligns each source sentence directly with its translation, making the cross-lingual correspondence explicit.

The finding (Table 1) is that Prompt 2 consistently outperforms Prompt 1 across all three models: B-7B-LORA increases from 19.3 to 20.6 µsBLEU, V-7B-LORA from 19.0 to 20.4, and L-7B-LORA from 15.5 to 19.0. The interleaved format, by placing source and target sentences adjacent to each other, makes the translation relationship more salient to the model. This matters for decoder-only LLMs because they process text left-to-right; an interleaved format means that when the model encounters a source sentence, its translation immediately follows, providing a natural "translation demonstration" pattern that the model can learn from and replicate.

Research question 2: How do natural language instructions influence translation quality? This is tested by comparing Prompt 1 (no instruction, only structure) vs. Prompt 3 (same structure as Prompt 1 plus an explicit instruction):

• Prompt 3 (Figure 1c): Same separate-context structure as Prompt 1, but with the instruction "Given the provided parallel context, translate the following <src_lang> sentence to <tgt_lang>:" inserted before the current sentence pair.

The results are mixed (Table 1): L-7B-LORA and B-7B-LORA benefit from the instruction (15.5→15.8 and 19.3→19.8 µsBLEU respectively), while V-7B-LORA performs worse with the instruction (19.0→18.3). The paper interprets this as evidence that "natural language instructions are less effective when using instruction-tuned language models as model backbones" — VICUNA-7B has already been optimized to follow instructions, so adding an instruction during fine-tuning may interfere with its existing instruction-following behavior or cause it to over-specialize to the specific instruction phrasing. Base models (LLAMA2, BLOOM), which have not been instruction-tuned, benefit from the explicit task specification.

Prompt 4 (Figure 1d) — the combined optimum: This prompt combines the interleaved structure of Prompt 2 with a natural language instruction (similar to Prompt 3 but adapted to the interleaved format): "Given the provided parallel sentence pairs, translate the following <src_lang> sentence to <tgt_lang>:" The results (Table 1) show that Prompt 4 achieves the best overall performance for all three models: L-7B-LORA 20.2, B-7B-LORA 23.1, V-7B-LORA 22.4 µsBLEU. The paper concludes that there is a "positive compound effect of context structure and instructions" — the interleaved format provides explicit cross-lingual alignment, and the instruction clarifies the task, and their combination is better than either alone.

Why Prompt 4 for the main experiments: Based on this preliminary study, all subsequent experiments (Section 5 onward) use Prompt 4 for both fine-tuning and inference. The prompt is fixed throughout — there is no prompt tuning or dynamic prompt selection at inference time. This standardization is important because it means all performance comparisons (between backbones, fine-tuning methods, and baselines) control for prompt format. The disadvantage is that the optimal prompt might differ across language pairs or backbones — a prompt that works best on average for German and Chinese might be suboptimal for Arabic or Japanese — but the preliminary study design (only testing on German and Chinese) does not explore this.

Fine-Tuning Mechanics: LoRA Implementation Details

For parameter-efficient fine-tuning, the paper uses LoRA (Hu et al., 2022) with rank 16. LoRA works by injecting trainable low-rank decomposition matrices into the weight updates of the transformer's attention layers. For a pre-trained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA constrains its update to a low-rank decomposition:

$W = W_0 + \Delta W = W_0 + BA$

where $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$, and the rank $r \ll \min(d, k)$. In this paper, $r = 16$. During training, $W_0$ is frozen (receives no gradient updates), while $A$ and $B$ contain the trainable parameters.

What it computes: For each attention layer, the forward pass computes $h = W_0 x + BA x$. The first term is the frozen pre-trained computation; the second term adds a low-rank correction learned during fine-tuning. The matrix $A$ projects the input into a low-dimensional space (dimension $r = 16$), and $B$ projects back to the output dimension. This factorization means that instead of learning $d \times k$ parameters for the update, the model learns only $r \times (d + k)$ parameters per layer.

Why this form: The low-rank constraint is motivated by the observation that the weight updates during fine-tuning of large pre-trained models often have low intrinsic rank — the model only needs to adjust in a few directions in weight space, not in the full high-dimensional space. By forcing the update through a low-rank bottleneck, LoRA drastically reduces the number of trainable parameters (0.1% of total, ~8M vs. 7B) while maintaining most of the expressive power of full fine-tuning. This makes training feasible on consumer-grade hardware (the paper's 7B models with LoRA can be fine-tuned on a single GPU). Additionally, LoRA acts as a regularizer: by restricting the update's rank, it prevents the model from overfitting to the fine-tuning data, which is particularly important when the dataset is relatively small (~200K sentence pairs per language pair). The paper's finding that LoRA often outperforms FFT (Table 2) supports this regularization interpretation — full fine-tuning on a medium-sized dataset can lead to overfitting, while LoRA's constrained updates provide implicit regularization.

Hyperparameters: Learning rate $5 \times 10^{-5}$, batch size 64, linear schedule with 10% warm-up, up to 3 epochs with early stopping based on validation loss. The LoRA adaptation is applied only to attention layers (not feed-forward layers), following standard practice. The rank of 16 was chosen based on preliminary experiments (not detailed in the paper) balancing parameter efficiency against model capacity.

For comparison, the re-implemented DOC2DOC-MT5 baselines use the same LoRA hyperparameters except for learning rate ($5 \times 10^{-4}$ for MT5 models vs. $5 \times 10^{-5}$ for LLMs) and training duration (up to 10 epochs for MT5 vs. up to 3 for LLMs). The longer training for MT5 reflects the fact that these models are trained from task-specific pre-training, not from a general LLM checkpoint, so they require more steps to converge.

Decoding Strategies and the Off-Target Translation Problem

At inference time, documents are translated using beam search with a beam size of 5. The critical design decision is how context translations — the model's own previous outputs that serve as context for subsequent sentences — are generated and fed back into the model. The paper identifies and compares two strategies:

REUSE (the default strategy, Section 3.4): During document translation, sentences are processed in their original order. For the first sentence, there is no context (the prompt contains only the instruction and the current source sentence). The model generates a translation via beam search. For sentence $i$ (where $i > 1$), the model's previously generated translations for sentences $i-3$, $i-2$, and $i-1$ (or fewer if $i \leq 3$) are inserted into the prompt as the target-language context. The model then generates the translation for sentence $i$ conditioned on this reused context.

REGEN (the alternative, introduced in Section 6): Instead of reusing previously generated translations directly, the REGEN strategy regenerates all context translations from scratch for each sentence. For sentence $i$, this means translating sentences $i-3$, $i-2$, and $i-1$ independently (or with their own preceding context) to produce fresh translations, which then serve as context for sentence $i$. This breaks the dependency chain: if sentence 1 was translated poorly, that poor translation does not pollute sentence 2's context, because sentence 2's context is generated independently.

The cost tradeoff: REGEN requires translating each context sentence separately for each target sentence, leading to approximately $c \times N$ total generations for a document of $N$ sentences with context window $c = 3$. This is roughly $4\times$ the cost of REUSE, which generates each sentence exactly once and reuses the outputs $c$ additional times. The paper's experiment on Arabic→English (Table 4) demonstrates the performance impact: L-7B-LORA improves from 3.9 sBLEU (REUSE) to 17.5 sBLEU (REGEN), a massive gain that confirms the error propagation hypothesis. The 17.5 sBLEU with REGEN is much closer to the model's capability when context is clean, suggesting that the model itself is capable — the failure is in the decoding pipeline, not in the learned translation function.

The error propagation mechanism: In the REUSE strategy, errors compound because each translation depends on the quality of up to three previous translations. If the model produces an off-target translation for an early sentence (e.g., generating Chinese text when the target language is English), that erroneous Chinese text is then presented as "context" for the next sentence. The model, seeing Chinese context when it expects English context, may become confused about the target language and continue producing Chinese (or produce a mix of languages, or produce nonsensical output). This creates a cascade: one early error can corrupt the entire document. The high off-target rates in Table 3 (e.g., 98.3% for V-7B-FFT on Korean→English) represent near-total document corruption — once the chain breaks, recovery is nearly impossible because every subsequent sentence receives corrupted context.

Why beam search with beam size 5? Beam search with a moderate beam width balances translation quality against computational cost. Larger beams would explore more candidate translations and potentially find better outputs, but at linearly increasing cost. The beam size of 5 is standard in MT research and provides a reasonable tradeoff. However, for the off-target translation problem, beam search may actually be counterproductive: if the model has a tendency to produce off-target output, beam search might amplify this by selecting hypotheses that score highly under the model's own probability distribution (which might favor fluent but wrong-language output over disfluent but correct-language output). The paper does not explore whether greedy decoding (beam size 1) reduces off-target rates.

Language identification for off-target measurement: The paper uses the fastText library (Bojanowski et al., 2017) to automatically identify the language of generated translations and compute off-target rates. FastText is a lightweight, efficient text classifier that uses subword n-gram features to predict the language of a given text. For each generated translation, fastText predicts a language label; if the predicted label does not match the intended target language, the translation is counted as off-target. The off-target rate is the proportion of translations (across all test sentences for a language pair) that are classified as off-target. This automated measurement enables the paper to quantify the problem at scale without manual inspection, though it may miss subtle cases (e.g., a translation that is in the correct language but heavily code-switched, or one that is misclassified by fastText).

Evaluation Setup

Metrics. Three evaluation metrics are reported:

• sBLEU (sentence-level BLEU; Papineni et al., 2002): Computes n-gram overlap between the generated translation and the reference translation for each sentence independently. The SacreBLEU signature is nrefs:1|case:mixed|eff:no|tok:[13a|ja-mecab-0.996-IPA|ko-mecab-0.996/ko-0.9.2-KO|zh]|smooth:exp|version:2.3.1. The language-specific tokenizers (Japanese: MeCab, Korean: MeCab, Chinese: character-level) handle the script differences appropriately.

• dBLEU (document-level BLEU; Liu et al., 2020): Computes BLEU at the document level, concatenating all sentences in a document before computing n-gram statistics. Documents are treated as single long sequences. This captures cross-sentence n-gram overlap that sentence-level BLEU misses (e.g., consistent terminology use across sentences).

• COMET (Rei et al., 2020): A neural evaluation metric that uses a pre-trained model (Unbabel/wmt22-comet-da) to predict a quality score for each translation given the source and reference. COMET captures semantic similarity beyond n-gram overlap and has been shown to correlate better with human judgments than BLEU (Freitag et al., 2022).

The average metrics µsBLEU, µdBLEU, and µCOMET aggregate across all translation tasks (nine language pairs for En→X or X→En) using arithmetic mean.

Inference protocol: Documents are translated in their natural order, starting with the first sentence (which has no preceding context). For sentence $i$, the prompt is constructed using the three preceding sentence pairs: source sentences $i-3, i-2, i-1$ and their translations (from the model's own output in REUSE mode, or reference translations during evaluation preprocessing — though the paper uses model-generated context during actual inference to simulate real deployment). The model generates the translation for sentence $i$ using beam search with beam size 5. The process repeats until all sentences in the document are translated.

Cross-validation for strategy comparison: When comparing training strategies (one-stage, two-stage, three-stage) and when analyzing scaling laws, the paper uses the provided train/validation/test splits of IWSLT2017. The validation set (approximately 19 documents per language pair) is used for early stopping during training. The test set (approximately 10–12 documents per language pair) is used for final evaluation. No cross-validation across folds is mentioned for the main results; the computational cost of fine-tuning 7B models multiple times would be prohibitive. This means the reported test-set results are from single training runs, and the stability of these results across different random seeds is not assessed (a limitation the paper acknowledges in Section 8 under "Instability in Training").

Summary of Design Choices and Their Justifications

• Two-stage training over one-stage: The monolingual stage addresses the English-centric bias of base LLMs, which is essential for document-level translation where the model must produce coherent target-language text across multiple sentences. Direct fine-tuning on parallel documents (one-stage) does not provide the necessary target-language grounding (Table 6: one-stage achieves only 1.3–30.2 sBLEU vs. 2.5–38.9 for two-stage on representative language pairs).

• Prompt 4 (interleaved pairs + instruction) over other formats: Interleaved source-target pairs provide explicit cross-lingual alignment signals that decoder-only LLMs can exploit during autoregressive generation. The natural language instruction clarifies the task, particularly for base models (LLAMA2, BLOOM) that have not been instruction-tuned. The combined format yields the best average performance (Table 1).

• LoRA over FFT for most settings: LoRA's low-rank constraint provides implicit regularization that prevents overfitting to the medium-sized IWSLT2017 dataset (~200K examples per language pair). FFT's advantage appears primarily in data-scarce regimes (Figure 4: FFT reaches full-dataset performance with ~1% of data) where the extra capacity helps, while LoRA's regularization matters more when data is plentiful.

• Three-sentence context window over full document: The fixed window of three preceding sentence pairs balances context relevance against computational cost. Longer windows would capture more distant discourse phenomena but increase computational cost and potentially introduce noise from irrelevant distant sentences. The choice follows Wang et al. (2023a).

• BLOOM-7B for multilingual robustness: Among the LLM backbones, BLOOM-7B uniquely maintains low off-target rates across all language pairs (Table 3: average 2.8% for X→En with LoRA) due to its multilingual pre-training. This makes it the preferred backbone when robustness to language confusion is critical, even if its raw translation quality sometimes lags behind LLAMA2-based models on certain language pairs.

• FastText for automated off-target detection: Using an efficient, off-the-shelf language identifier enables systematic measurement of the off-target problem across all language pairs and models without expensive manual inspection. The approach scales to the full test set, providing reliable statistics rather than anecdotal observations.

4. Key Insights and Innovations

Innovation 1: Off-Target Translation as the Primary Diagnostic Concept for LLM-Based DOCMT

The paper's most conceptually distinctive contribution is identifying and systematically characterizing off-target translation as the dominant failure mode of fine-tuned LLMs for document-level translation, and tracing its root cause to error propagation during autoregressive context reuse rather than to any deficiency in the models' learned translation capability per se.

Before this work, the literature on LLM-based translation focused almost exclusively on aggregate quality metrics (BLEU, COMET) that can obscure catastrophic but pattern-specific failures. A model with 98.3% off-target rate on Korean→English (V-7B-FFT, Table 3) and a model with 0% off-target rate on Dutch→English are both summarized by a single average µsBLEU score — the average hides the bimodal distribution. The paper's diagnostic move is to separate the question "Can this model translate?" from "Does the decoding pipeline corrupt the model's translations?" — and to demonstrate that the answer to the first is often "yes" while the answer to the second is "frequently, catastrophically."

Contrast with prior work: Wang et al. (2023b) studied GPT-3.5-TURBO for DOCMT via prompting and reported generally strong performance without identifying off-target translation as a systematic concern—likely because GPT-3.5-TURBO's massive scale and broad pre-training confer robustness against language confusion that smaller fine-tuned models lack. The sentence-level LLM fine-tuning work by Xu et al. (2023) and others circumvented the problem entirely because sentence-level translation has no autoregressive context chain: each sentence is an independent generation with no dependency on previous model outputs. The off-target problem is therefore uniquely a document-level, decoder-only phenomenon that only becomes visible when (a) the model generates multiple sentences sequentially, (b) its own previous outputs serve as conditioning context, and (c) the model is small enough (7B) that its language representations are fragile under distribution shift.

The diagnostic power of the REUSE vs. REGEN comparison: The paper's most elegant demonstration comes from Table 4, where L-7B-LORA on Arabic→English jumps from 3.9 sBLEU (REUSE, the standard approach) to 17.5 sBLEU (REGEN, where context translations are regenerated independently). The REGEN score of 17.5 is competitive with conventional DOCMT baselines like DOC2DOC-MT5-300M (19.4 on Ar→En, Table 13), suggesting that the fine-tuned model knows how to translate — it just cannot maintain language consistency when forced to condition on its own potentially erroneous previous outputs. This reframes the DOCMT challenge: it is less a problem of teaching models to use context than a problem of making the context chain robust to errors. The 4× cost of REGEN is presented as the price of robustness, setting up a clear optimization target for future work.

Significance beyond performance: This finding is more important as a reconceptualization than as a metric improvement. It tells the field that comparing LLM-based DOCMT systems purely on BLEU/COMET averages (as Table 2 does) fundamentally misses the main obstacle to deployment. The real metric of interest is not average quality but robustness of the decoding chain — a property that current evaluation protocols do not measure because test sets present sentences independently. The paper provides the measurement methodology (fastText-based language identification, off-target rate computation) and demonstrates that the problem varies dramatically across language pairs (from 1.6% to 98.3%) and model backbones (BLOOM-7B is far more robust than LLAMA2-7B or VICUNA-7B). This is a fundamental diagnostic contribution that should change how future DOCMT systems are evaluated.

Innovation 2: Multilingual Pre-Training as Implicit Robustness Regularization for Document-Level Decoding

The paper's three-backbone design (LLAMA2-7B, BLOOM-7B, VICUNA-7B) generates a finding that is conceptually sharper than a simple "which backbone is best" comparison: multilingual pre-training provides a form of implicit regularization against language confusion during autoregressive decoding, and this benefit is largely invisible when measuring translation quality on clean, independent test sentences but becomes decisive in the chained document-level setting.

The evidence is clearest in the off-target rate comparison (Tables 3, 16, 17). BLOOM-7B-LORA achieves only 2.8% average off-target rate on X→En translation, compared to 29.2% for L-7B-LORA and 32.3% for V-7B-LORA. This robustness advantage exists despite BLOOM-7B-LORA's mixed performance on standard quality metrics (Table 2: 29.9 µsBLEU on X→En, competitive but not uniformly dominant). The key insight is that quality and robustness are distinct axes: a model can produce excellent translations when it stays on-target, but if it goes off-target 30% of the time (as LLAMA2-based models do), its effective quality is far lower than what aggregate metrics suggest.

What makes this novel relative to prior work: The standard narrative in multilingual NLP is that multilingual pre-training improves performance on low-resource languages by enabling cross-lingual transfer (as in NLLB, Costa-jussà et al., 2022, and mT5, Xue et al., 2021). This paper identifies a different, previously undocumented benefit: multilingual pre-training stabilizes the model's language identity signal during generation, making it less likely to drift into producing text in the wrong language when conditioning on noisy or self-generated context. This is not about cross-lingual transfer of translation knowledge — it is about the robustness of the language generation process itself. The mechanism is likely that multilingual pre-training gives the model sharper, more distinct representations of different languages, so that when it is prompted with English source text and English-target-language instruction, it remains anchored in the target language even when its own previous outputs introduce distribution shift.

Why this matters for model selection: The paper's finding challenges the implicit assumption that better pre-training for a task (here, LLAMA2-7B/VICUNA-7B, which are English-centric and instruction-tuned respectively) translates to better downstream performance after fine-tuning. For document-level translation specifically, the quality-robustness tradeoff means that BLOOM-7B may be the preferred backbone despite sometimes lower peak quality, because its translations are reliably in the correct language whereas LLAMA2-based models will occasionally produce untranslated or wrongly-translated output — a failure mode that is far more damaging in deployed systems than a few missing n-gram matches. This is a fundamental empirical finding that should influence backbone selection in future DOCMT work.

Innovation 3: The Prompt Design Space for Fine-Tuning Is Fundamentally Different from the Prompt Design Space for Inference

The preliminary study in Section 4 tests prompt variations during fine-tuning, not at inference time, and the results reveal a principle that the LLM prompting literature has largely overlooked: the optimal prompt for teaching a model a task via supervised fine-tuning is not necessarily the optimal prompt for eliciting that task from a frozen model at inference time.

The paper's four-prompt comparison (Figure 1, Table 1) systematically varies context structure (separated blocks vs. interleaved pairs) and instruction presence (with vs. without natural language instructions) across three fine-tuned models. The headline finding — that Prompt 4 (interleaved pairs + instruction) is best — is less interesting than the interaction between prompt features and model type: natural language instructions help base models (L-7B-LORA: 15.5→15.8 with instruction; B-7B-LORA: 19.3→19.8) but hurt instruction-tuned models (V-7B-LORA: 19.0→18.3). This interaction would not be discoverable by studying inference-time prompting alone, where all three models would be evaluated with frozen weights and the prompt would simply be prepended to the input.

The mechanism likely involves training-time overfitting to prompt format: VICUNA-7B has already been fine-tuned to follow a wide variety of instruction formats during its instruction-tuning phase. When further fine-tuned on DOCMT data with a specific instruction phrasing, the model may overfit to that phrasing, losing the generalization that makes instruction-tuned models flexible. Alternatively, the instruction-tuning may have already taught VICUNA-7B to recognize and respond to task specifications, so adding an explicit instruction during DOCMT fine-tuning is redundant and the model learns to ignore it — meaning the prompt's instruction tokens occupy context window space without providing useful signal. The base models, lacking any instruction-following prior, genuinely learn from the instruction and benefit from its presence.

Prior work and the gap this fills: The LLM prompting literature (Kojima et al., 2022; Wei et al., 2022b; and many others) has extensively studied how prompt design affects inference-time performance. Wang et al. (2023b) specifically studied GPT-3.5-TURBO's DOCMT performance under different prompts at inference time. But the question of how prompt design affects fine-tuning dynamics — how the prompt becomes baked into the model's weights during supervised training — has received far less attention. This paper's finding that instruction-tuned backbones may actually perform worse with instructions during fine-tuning is a counterintuitive empirical result that should cause practitioners to rethink the default "instruction-tuned models are better starting points" assumption. It is a fundamental refinement of best practices for LLM fine-tuning, not specific to translation.

Scope of the finding: The preliminary study tests only four prompts on two language pairs (German and Chinese) with three LoRA models. The paper does not explore whether different prompts would be optimal for different language pairs (the optimal prompt might depend on how well the source and target languages are represented in the base model's pre-training) or whether the prompt design conclusions generalize to other document-level tasks. The space of possible prompt designs is combinatorially large, and the study's four variations barely scratch the surface. However, the qualitative finding — that prompt features interact with model type during fine-tuning in ways that differ from inference-time prompting — is robust and important regardless of whether Prompt 4 is truly optimal.

Innovation 4: Data Efficiency Asymmetry Between Full Fine-Tuning and Parameter-Efficient Fine-Tuning

The scaling law analysis in Figure 4 reveals a finding that challenges common intuitions about when to use full fine-tuning vs. parameter-efficient methods: FFT is dramatically more data-efficient than LoRA at small dataset sizes, but this advantage disappears as data scales, creating a crossover pattern that has practical implications for resource allocation in low-resource translation settings.

The concrete numbers: on English→German translation, FFT models reach near-full-dataset performance with only ~1% of the training data (~2K sentence pairs), while LoRA models require ~10% (~20K sentence pairs) to achieve comparable COMET scores. At the full dataset size (~200K sentence pairs), LoRA and FFT converge to similar or LoRA-superior performance (Table 2: L-7B-LORA achieves 17.2 µsBLEU on En→X vs. 13.7 for L-7B-FFT; B-7B-LORA achieves 17.7 vs. 12.0 for FFT). This asymmetry contradicts the naive expectation that FFT, with its larger capacity (updating all 7B parameters vs. 8M), would always overfit more severely on small data.

Why this is conceptually interesting: The standard narrative around parameter-efficient fine-tuning emphasizes its computational efficiency (fewer trainable parameters = lower memory and faster training) and its regularization effect (fewer degrees of freedom prevent overfitting). The data-efficiency story is usually the opposite: because LoRA has fewer parameters, it should need less data to converge, not more. The paper's finding suggests a different dynamic: FFT's ability to adjust all parameters enables rapid adaptation to the task distribution when data is extremely scarce, essentially performing a form of few-shot learning where the model can make large, coordinated weight changes to align with the translation format. LoRA's low-rank constraint, while preventing overfitting at scale, also limits the magnitude of possible adaptation — with only 8M parameters, the model can only shift its behavior so far from the pre-trained distribution, which is insufficient when training data is minimal.

Tie to evidence and robustness: The pattern is consistent across all three backbones (L-7B, B-7B, V-7B in Figure 4) and across additional language pairs tested in Appendix H (English→Romanian, English→Chinese, Figure 7), suggesting it is not an artifact of a particular backbone or language pair. The FFT advantage is most pronounced at the 1% data mark and narrows steadily as data increases, with LoRA sometimes overtaking FFT at the 100% mark (e.g., L-7B-LORA at ~77 COMET vs. L-7B-FFT at ~77 COMET on English→German at 100% data, Figure 4). This is an incremental empirical contribution — it does not propose a new method or theory — but it has significant practical implications: for low-resource language pairs where only a few thousand parallel documents exist, FFT is clearly the better choice despite its higher computational cost, while for medium-to-large datasets, LoRA's regularization and efficiency advantages make it preferable.

Connection to the instability limitation: The paper acknowledges training instability as a limitation (Section 8: "there are noticeable inconsistencies in performance... too significant to attribute solely to the randomness inherent in training"). The data-efficiency curves in Figure 4 are not perfectly monotonic — some models show performance dips at intermediate data percentages — which may reflect this instability. The FFT advantage at 1% data might be partially confounded by the fact that FFT training runs are less stable (more prone to non-convergence or divergence), and the reported results may be from the best of several runs rather than a stable expected value. The paper does not report multiple random seeds, so the reliability of the 1%-data FFT advantage is unquantified. This is a limitation of the finding itself, not a reason to dismiss it — the pattern is consistent enough across models and language pairs to be credible, but the precise data-efficiency crossover points should be treated as approximate.

Innovation 5: Instruction-Tuned Backbones Underperform Base Backbones for Task-Specific Fine-Tuning — A Counterintuitive Finding with Implications Beyond Translation

The consistent underperformance of VICUNA-7B relative to LLAMA2-7B across most evaluation settings (Table 2: V-7B-LORA at 15.8 µsBLEU on En→X vs. L-7B-LORA at 17.2; V-7B-FFT at 14.3 vs. L-7B-FFT at 13.7 — the gap is less consistent for FFT in this direction, but clearly present for LoRA) and the negative zero-shot cross-lingual transfer results (Table 8: V-7B-LORA shows average COMET decrease of −8.9 when transferring from English-German fine-tuning to English→other languages, while L-7B-LORA shows +29.4 increase) establish that instruction tuning is not a universally beneficial pre-training step for downstream task-specific fine-tuning, and for document-level translation specifically, it can be actively harmful.

What makes this intellectually distinctive: The dominant narrative in the LLM community since the rise of models like Alpaca, Vicuña, and FLAN has been that instruction-tuned models are better starting points for downstream tasks because they are already optimized for following task specifications and generating coherent, on-topic output. This paper provides one of the first clear counterexamples in the translation domain. The finding is not merely that VICUNA-7B fails to outperform LLAMA2-7B — it is that VICUNA-7B loses capabilities during fine-tuning that the base model retains or acquires. The zero-shot transfer results are particularly striking: an English-German fine-tuned LLAMA2-7B model improves its COMET on English→Arabic by 36.3 points compared to the unfine-tuned backbone, while the equivalent VICUNA-7B model degrades by 12.6 points (Table 19). The instruction-tuned model has become less capable on languages it was not fine-tuned on, while the base model has generalized positively.

Mechanism hypothesis: The paper speculates that instruction-tuned models may overfit to the specific prompt format and task distribution seen during fine-tuning, losing the broad instruction-following flexibility that their instruction tuning was designed to provide. During instruction tuning, VICUNA-7B learned to attend to instruction text and adjust its behavior accordingly across many tasks. When fine-tuned on DOCMT with a single fixed instruction format, this instruction-attending mechanism may become overspecialized — the model learns to expect exactly Prompt 4's format and fails when the format changes (as in zero-shot transfer, where the language pair changes but the prompt structure is identical). The base model, having never been trained to attend to instructions specifically, treats the prompt as part of the conditioning context and learns to map from source-context-instruction to target translation more robustly.

Scope and limitations of this innovation: The finding is specifically about fine-tuning for a single, narrowly-defined task with a fixed prompt format. It does not imply that instruction-tuned models are worse for all downstream uses — for general-purpose assistants, multi-task models, or scenarios where the prompt varies significantly at inference time, instruction-tuned backbones may still be preferable. The paper's zero-shot transfer task approximates a multi-language scenario where the model is fine-tuned on one language pair and tested on others, which is a specific form of generalization. The finding's significance is that it identifies a boundary condition on the "instruction-tuned is better" assumption: when the downstream task involves substantial further supervised training on a fixed-format dataset, the instruction-tuning prior can interfere rather than help. This is a fundamental empirical correction to prevailing practice, with implications for any domain where fine-tuning on task-specific data is the primary adaptation strategy (code generation, mathematical reasoning, domain-specific summarization, etc.).

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the IWSLT2017 translation benchmark (Cettolo et al., 2017), which consists of TED talk transcripts translated between English and nine other languages: Arabic, German, French, Italian, Japanese, Korean, Dutch, Romanian, and Chinese. This creates 18 translation tasks (9 En→X, 9 X→En). The training sets range from approximately 206K to 237K sentence-aligned parallel sentences per language pair (1,705–1,920 documents), with validation sets of approximately 2,400–2,800 sentences (19 documents) and test sets of approximately 1,100–1,500 sentences (10–12 documents). Full statistics are provided in Appendix A, Table 9.

• Base model(s). Three 7B-parameter decoder-only LLMs serve as backbones: LLAMA2-7B (Touvron et al., 2023b), pre-trained predominantly on English text; BLOOM-7B (Scao et al., 2022), pre-trained on multilingual text spanning 46 languages; and VICUNA-7B v1.5, an instruction-tuned variant of LLAMA2-7B. The three backbones are chosen to disentangle the effects of English-centric vs. multilingual pre-training (LLAMA2 vs. BLOOM) and base vs. instruction-tuned initialization (LLAMA2 vs. VICUNA). Each backbone undergoes the same two-stage fine-tuning procedure: Stage 1 updates all parameters on 100M monolingual tokens per target language (1B total) from CulturaX (Nguyen et al., 2023) pruned using the technique of Marion et al. (2023), and Stage 2 fine-tunes on IWSLT2017 parallel documents using either full fine-tuning (FFT, all 7B parameters) or LoRA (rank 16, ~8M trainable parameters, 0.1% of total).

• Metrics. Three translation quality metrics are reported: sBLEU (sentence-level BLEU, Papineni et al., 2002) computed with the SacreBLEU signature nrefs:1|case:mixed|eff:no|tok:[13a|ja-mecab-0.996-IPA|ko-mecab-0.996/ko-0.9.2-KO|zh]|smooth:exp|version:2.3.1, which applies language-specific tokenization for Japanese (MeCab), Korean (MeCab), and Chinese (character-level); dBLEU (document-level BLEU, Liu et al., 2020), which concatenates all sentences in a document before computing n-gram statistics, capturing cross-sentence consistency that sentence-level BLEU misses; and COMET (Rei et al., 2020), a neural evaluation metric using the Unbabel/wmt22-comet-da model, which predicts quality scores from source, reference, and hypothesis and has been shown to correlate better with human judgments than BLEU (Freitag et al., 2022). Aggregate metrics µsBLEU, µdBLEU, and µCOMET are arithmetic means across the relevant language pairs (nine for En→X, nine for X→En). Off-target translation is measured using the fastText library (Bojanowski et al., 2017) to automatically identify the language of generated translations; the off-target rate is the proportion of test sentences classified as not being in the intended target language.

• Baselines. Four categories of comparison systems are evaluated: (1) State-of-the-art sentence-level MT models: NLLB (Costa-jussà et al., 2022) in three sizes (600M, 1.3B, 3.3B parameters) and the commercial Google Translate system, representing the best available sentence-level translation without document context. (2) State-of-the-art LLMs for DOCMT: GPT-3.5-TURBO and GPT-4-TURBO evaluated zero-shot using Prompt 4 (Figure 1d), representing the upper bound of what massive general-purpose models can achieve without task-specific fine-tuning. (3) LLM backbones without fine-tuning: LLAMA2-7B, BLOOM-7B, and VICUNA-7B evaluated zero-shot with Prompt 4, establishing the pre-fine-tuning performance floor. (4) Re-implemented specialized DOCMT models: DOC2DOC-MT5 (Tiedemann and Scherrer, 2017) — the concatenation baseline that prepends three preceding source sentences; MR-DOC2SEN-MT5 and MR-DOC2DOC-MT5 (Sun et al., 2022) — multi-resolution approaches combining sentence-level and document-level objectives; DOCFLAT-MT5 (Wu et al., 2023) — document flattening with smart context reordering; and IADA-MT5 (Wu et al., 2024c) — importance-aware data augmentation. All re-implemented baselines use MT5 (Xue et al., 2021) as backbone and are available in 300M, 580M, and 1.2B parameter sizes, establishing the performance range of purpose-built encoder-decoder DOCMT architectures.

• Generation budget / compute accounting. The paper measures computational cost implicitly through the number of model parameters and the number of trainable parameters (Table 2), but does not standardize a "generation budget" in the manner of test-time compute scaling papers because all models are evaluated using the same beam search configuration (beam size 5) on the same test sets. The primary cost distinction is between fine-tuning methods: LoRA trains ~8M parameters while FFT trains all 7B. For the REGEN vs. REUSE decoding comparison (Table 4), the cost differential is explicitly stated: REGEN requires approximately 4× the inference cost of REUSE because context sentences are regenerated for each target sentence rather than reused. No FLOPs-matched comparison between LLM-based models and specialized baselines is performed — the comparison in Table 2 is at fixed model scale and training data, not fixed compute.

• Cross-validation / statistical protocol. The paper uses the standard IWSLT2017 train/validation/test splits without cross-validation. The validation set (approximately 19 documents per language pair) is used for early stopping during fine-tuning; the test set (10–12 documents per language pair) is used for final evaluation. All reported test-set results are from single training runs — the paper does not report multiple random seeds or confidence intervals. For the prompt preliminary study (Section 4), results are aggregated across four English-centric translation tasks (involving German and Chinese) and three models, with the best prompt chosen based on average µsBLEU. For the scaling law analysis (Section 6, Figure 4), models are trained on subsets of the full training data at percentages ranging from 1% to 100%, with COMET evaluated on the English→German, English→Romanian, and English→Chinese test sets. The paper acknowledges training instability as a limitation (Section 8): "there are noticeable inconsistencies in performance... too significant to attribute solely to the randomness inherent in training."

Main Quantitative Results

Overall Performance (Table 2): Fine-Tuned 7B LLMs Are Competitive with Specialized DOCMT Models on X→En but Lag on En→X

The headline aggregate result from Table 2 is that LLM-based DOCMT models fine-tuned with LoRA achieve competitive or superior performance to conventional DOCMT models on translation from other languages to English, but underperform on translation from English to other languages:

• X→En (translation into English): B-7B-LORA achieves 29.9 µsBLEU, substantially outperforming all re-implemented DOCMT baselines (the best conventional model, DOCFLAT-MT5-1.2B, achieves 22.2 µsBLEU). L-7B-LORA achieves 23.8 µsBLEU, still ahead of all DOCMT baselines. The LLM-based models approach but do not match GPT-4-TURBO (31.7 µsBLEU). On µdBLEU, B-7B-LORA achieves 33.6 vs. 24.3 for DOCFLAT-MT5-1.2B. However, the COMET scores tell a different story: conventional DOCMT models maintain an advantage, with IADA-MT5-1.2B achieving 80.4 µCOMET vs. 81.4 for B-7B-LORA and 73.7 for L-7B-LORA.

• En→X (translation out of English): The pattern reverses. DOCFLAT-MT5-1.2B achieves 19.2 µsBLEU, ahead of B-7B-LORA at 17.7 and L-7B-LORA at 17.2. Conventional DOCMT models dominate across all three metrics, with IADA-MT5-1.2B achieving 80.7 µCOMET vs. 70.8 for L-7B-LORA and 68.5 for B-7B-LORA. The gap is larger for COMET than BLEU, suggesting that LLM-based models may produce translations that are lexically similar (good n-gram overlap) but semantically or stylistically deficient (lower COMET) when translating into non-English languages.

The asymmetry — LLMs performing better into English than out of English — is consistent with the English-centric pre-training of LLAMA2-7B and VICUNA-7B, and suggests that Stage 1 monolingual fine-tuning (100M tokens per language) partially but incompletely addresses the target-language generation deficit.

Fine-Tuning Method Comparison (Table 2): LoRA Generally Outperforms FFT, but with Language-Direction Asymmetry

Across all six model variants in Table 2, LoRA-fine-tuned models achieve higher or comparable µsBLEU to their FFT counterparts:

• L-7B-LORA vs. L-7B-FFT: 17.2 vs. 13.7 on En→X; 23.8 vs. 22.4 on X→En. LoRA advantage is consistent.
• B-7B-LORA vs. B-7B-FFT: 17.7 vs. 12.0 on En→X; 29.9 vs. 22.3 on X→En. LoRA advantage is large and consistent.
• V-7B-LORA vs. V-7B-FFT: 15.8 vs. 14.3 on En→X; 21.6 vs. 21.8 on X→En. LoRA advantage for En→X; essentially tied for X→En.

The one exception — V-7B-FFT slightly outperforming V-7B-LORA on X→En (21.8 vs. 21.6 µsBLEU) — is attributed to overfitting: "Extensive fine-tuning with a large corpus often leads to rapid overfitting, whereas LORA, which updates only a few parameters, helps prevent this issue." The COMET scores reinforce this interpretation: FFT models consistently achieve lower COMET than their LoRA counterparts (e.g., L-7B-FFT achieves 67.4 µCOMET on En→X vs. 70.8 for L-7B-LORA; B-7B-FFT achieves 59.6 vs. 68.5), suggesting that FFT overfitting degrades semantic quality even when n-gram overlap metrics remain partially intact.

Breakdown Results by Language Pair (Figure 2, Tables 10–15): Selective Excellence Masked by Catastrophic Failures

The aggregate metrics in Table 2 conceal extreme per-language-pair variance that becomes visible in the breakdown results (Figure 2 for X→En with LoRA models). The key pattern is bimodal performance:

• Success cases (LLM-based models competitive with or exceeding GPT-4-TURBO): On Dutch→English, L-7B-LORA achieves 39.2 sBLEU vs. GPT-4-TURBO's 36.6 (Table 13), and B-7B-LORA achieves 35.0. On French→English, L-7B-LORA achieves 40.3 sBLEU (vs. GPT-4-TURBO's 41.4). On German→English, L-7B-LORA achieves 33.1 (vs. GPT-4-TURBO's 31.2). On Romanian→English, L-7B-LORA achieves 39.0 (vs. GPT-4-TURBO's 36.5). These are substantial achievements: a 7B fine-tuned model matching a ~100× larger general-purpose model on specific language pairs.

• Failure cases (near-zero performance): On Arabic→English, L-7B-LORA achieves only 3.9 sBLEU (Table 13). On Chinese→English, L-7B-LORA achieves 0.1 sBLEU. On Japanese→English, L-7B-LORA achieves 8.3 sBLEU. On Korean→English, L-7B-LORA achieves 5.0 sBLEU. These failures are not subtle degradations — they represent essentially complete collapse of translation capability for specific language pairs.

The explanation for this bimodality is provided in Table 3 and Tables 16–17: the failing language pairs suffer from catastrophic off-target translation rates. For L-7B-LORA, the off-target rate on Chinese→English is 93.1%, on Arabic→English is 87.9%, on Korean→English is 44.2%, and on Japanese→English is 25.5% (Table 17). When the model produces text in the wrong language, BLEU scores collapse to near zero because there is no n-gram overlap with the English reference. The success cases have low off-target rates: Dutch→English 1.9%, French→English 4.9%, German→English 2.0%, Romanian→English 1.8%.

Backbone Comparison (Tables 2, 3, 8, 19): BLOOM-7B Provides Robustness; VICUNA-7B Provides No Benefit and Harms Transfer

The three-backbone design reveals clear differentiation:

• BLOOM-7B for multilingual robustness: B-7B-LORA achieves the lowest off-target rates across all language pairs (Table 3: average 2.8% on X→En, compared to 29.2% for L-7B-LORA and 32.3% for V-7B-LORA). On the specific failing pairs for LLAMA2-based models, B-7B-LORA maintains low off-target rates: Chinese→English 1.6%, Arabic→English 2.9%, Japanese→English 4.0%, Korean→English 8.4%. This robustness translates directly to competitive BLEU scores on these pairs (Table 13: B-7B-LORA achieves 30.9 sBLEU on Ar→En, 25.6 on Zh→En, 13.9 on Ja→En, 15.5 on Ko→En). B-7B-LORA is the only model that achieves strong performance across all nine X→En language pairs, making it the most reliable backbone despite not always achieving the highest peak scores.

• VICUNA-7B underperforms LLAMA2-7B: Despite being instruction-tuned from LLAMA2-7B, VICUNA-7B variants consistently achieve lower or equal performance to their LLAMA2-7B counterparts (Table 2: V-7B-LORA at 15.8 µsBLEU on En→X vs. L-7B-LORA at 17.2; V-7B-FFT at 14.3 vs. L-7B-FFT at 13.7 — the FFT comparison is close but LoRA shows clear separation). The off-target rates are similar between VICUNA and LLAMA2 models (Table 3: 32.3% vs. 29.2% average), suggesting that instruction tuning does not mitigate the language confusion problem. The paper notes: "further fine-tuning with instruction-tuned models does not always improve task-specific performance," a finding that contradicts the common assumption that instruction-tuned models are universally better starting points.

• Zero-shot cross-lingual transfer reveals a stark difference (Tables 8, 19): When English-German fine-tuned models are evaluated on English→other-language test sets, LLAMA2-based models show positive transfer: L-7B-LORA improves COMET by an average of +29.4 across all tested languages (Table 8), with gains of +36.3 on Arabic, +38.8 on German (the fine-tuned pair itself), +37.2 on French, +29.5 on Chinese. BLOOM-based models also show positive transfer: B-7B-LORA achieves +20.3 average gain. In contrast, VICUNA-based models show negative transfer: V-7B-LORA decreases COMET by an average of −8.9, with losses of −12.6 on Arabic, −34.1 on Chinese. This means fine-tuning VICUNA-7B on English-German DOCMT actually damages its ability to translate into other languages — the model overfits to the fine-tuning language pair in a way that the base model does not.

State-of-the-Art LLM Comparison (Table 2): GPT-4-TURBO Dominates Aggregates; Fine-Tuned 7B Models Compete on Specific Pairs

GPT-4-TURBO achieves 27.0 µsBLEU on En→X and 31.7 on X→En (Table 2), substantially ahead of all fine-tuned 7B models on aggregate. GPT-3.5-TURBO achieves 26.3 and 30.7 respectively. The gap is larger for En→X (best fine-tuned: 20.2 for L-7B-LORA vs. 27.0 for GPT-4-TURBO) than for X→En (best fine-tuned: 29.9 for B-7B-LORA vs. 31.7 for GPT-4-TURBO). On COMET, GPT-4-TURBO achieves 86.3 and 86.0, while the best fine-tuned models achieve 70.8 and 81.4 — a substantial gap suggesting that GPT-4's translations are semantically more accurate even when lexical overlap (BLEU) is closer.

However, the breakdown results in Figure 2 and Tables 13–15 reveal specific language pairs where fine-tuned 7B models exceed GPT-4-TURBO: B-7B-LORA on Arabic→English (30.9 vs. GPT-4-TURBO's average of 31.7 — the paper does not provide GPT-4-TURBO's per-language-pair breakdown, but the aggregate includes this pair), and L-7B-LORA on Dutch→English (39.2 vs. 36.6) and Romanian→English (39.0 vs. 36.5) as shown in Table 13. These are cases where a 7B specialized model genuinely outperforms a massive general-purpose model, validating the "smaller, specialized models can outperform larger, general-purpose models" thesis that the paper cites in its introduction.

Re-Implemented DOCMT Baselines (Table 2): Specialized Architectures Provide Modest Gains Over Simple Concatenation

The five re-implemented DOCMT baselines using MT5 backbone establish the performance range of specialized encoder-decoder document-level systems:

• DOC2DOC-MT5 (concatenation): 19.2 µsBLEU on En→X at 1.2B parameters; 21.5 on X→En.
• MR-DOC2SEN-MT5 (multi-resolution, sentence-level decoding): 18.8 on En→X, 22.0 on X→En at 1.2B.
• MR-DOC2DOC-MT5 (multi-resolution, document-level decoding): only dBLEU reported — 22.5 on En→X, 24.0 on X→En.
• DOCFLAT-MT5 (document flattening): 19.2 on En→X, 22.2 on X→En at 1.2B.
• IADA-MT5 (importance-aware data augmentation): 19.3 on En→X, 22.1 on X→En at 1.2B.

The gains from sophisticated architectures over simple concatenation are modest: IADA-MT5 improves over DOC2DOC-MT5 by only 1.3 µsBLEU on En→X (19.3 vs. 18.4 at 1.2B) and 0.6 on X→En (22.1 vs. 21.5). This suggests that for the IWSLT2017 benchmark, the dominant factor is not the architectural mechanism for handling context but rather the base model's translation capability — the MT5 backbone itself provides most of the performance. The specialized architectural innovations provide small, incremental improvements that are dwarfed by the gap between MT5-based models and the best LLM-based models (B-7B-LORA at 29.9 on X→En).

NLLB Baselines (Table 2): Sentence-Level Models Are Surprisingly Competitive with Document-Level Systems

The NLLB models, which translate sentences independently without any document context, establish a strong baseline: NLLB-3.3B achieves 26.8 µsBLEU on En→X and 25.8 on X→En. This is substantially higher than all DOCMT baselines (best: IADA-MT5-1.2B at 19.3 on En→X, 22.1 on X→En) and competitive with GPT-4-TURBO (27.0, 31.7). The sentence-level NLLB models outperform document-level specialized models by a wide margin, raising the question of whether document-level context is actually beneficial for the IWSLT2017 benchmark, or whether the stronger base pre-training of NLLB (on massively multilingual data) simply outweighs any benefit from context.

This finding is not discussed in the paper's analysis but is visible in Table 2. It suggests that the IWSLT2017 test sets may not contain sufficient discourse phenomena (pronoun disambiguation, lexical cohesion, etc.) to demonstrate the value of document context, or that the evaluation metrics (particularly sentence-level BLEU) are insensitive to document-level improvements. The paper's later discourse phenomena evaluation (Table 5) partially addresses this by using targeted contrastive test sets, but the aggregate metric comparison remains dominated by sentence-level translation quality.

Error Analysis (Figure 3): Fine-Tuned LLMs Produce Fewer Translation Errors than Google Translate and Conventional DOCMT Models

Using GPT-4-TURBO to classify translation errors according to a subset of the Multidimensional Quality Metrics (MQM) framework (Burchardt, 2013), the paper compares four systems — L-7B-LORA, L-7B-FFT, DOC2DOC-MT5-1.2B, and Google Translate — on translations from English to German, Romanian, and Chinese (Figure 3). Despite achieving similar or lower sBLEU/dBLEU/COMET scores than Google Translate and DOC2DOC-MT5-1.2B (Table 2), the LLM-based models produce fewer errors across most categories:

• Mistranslation: L-7B-LORA produces 1,395 errors (total across the three language pairs), L-7B-FFT produces 1,356, compared to 1,712 for Google Translate and 2,002 for DOC2DOC-MT5-1.2B.
• Overtranslation: L-7B-LORA produces 717 errors vs. 715 for Google Translate and 836 for DOC2DOC-MT5-1.2B.
• Cohesion (context-dependent): L-7B-LORA produces 709 errors vs. 804 for Google Translate and 933 for DOC2DOC-MT5-1.2B.
• Coherence (context-dependent): L-7B-LORA produces 454 errors vs. 592 for Google Translate and 717 for DOC2DOC-MT5-1.2B.

The paper highlights that "our LLM-based DOCMT models (L-7B-LORA and L-7B-FFT) exhibit fewer context-independent and context-dependent errors," interpreting this as evidence that "fine-tuning LLMs holds promise for enhancing DOCMT performance" even when standard metrics fail to capture the improvement. The implication is that current evaluation metrics (sBLEU, dBLEU, COMET) may be insufficiently sensitive to document-level translation quality, and that LLM-based models may be producing better translations than the metrics suggest. This is a nuanced claim — the models achieve lower metric scores but produce fewer errors as judged by GPT-4 — that complicates the straightforward "LLMs underperform on En→X" narrative from Table 2.

Discourse Phenomena Evaluation (Table 5): LLAMA2-Based Models Excel at Context-Dependent Pronoun Resolution

Using contrastive test sets specifically designed to evaluate discourse phenomena — the English-German set by Müller et al. (2018) and the English-French set by Lopes et al. (2020) — the paper measures accuracy in selecting the correct pronoun from multiple translation options when the correct choice depends on cross-sentence context:

• English-German: L-7B-LORA achieves 83.1% accuracy, V-7B-LORA achieves 84.9%, and V-7B-FFT achieves 84.4%, all substantially outperforming DOC2DOC-MT5-1.2B at 77.0%. B-7B-LORA underperforms at 75.5%, and B-7B-FFT at 68.3%.
• English-French: L-7B-LORA achieves 95.1%, V-7B-LORA achieves 94.8%, both outperforming DOC2DOC-MT5-1.2B at 89.9%. B-7B-LORA achieves 91.9%.

The paper attributes BLOOM-7B's poor performance to "the lack of German text in BLOOM pre-training" (as documented by Scao et al., 2022), suggesting that contextual understanding is largely acquired during pre-training rather than during task-specific fine-tuning. The LLAMA2 and VICUNA models, despite being English-centric, apparently retain enough German and French exposure from pre-training to handle discourse-level pronoun resolution, while BLOOM-7B's multilingual pre-training did not include sufficient representation of these specific languages. The generative accuracy results (Appendix G, Table 18) show a different pattern: L-7B-LORA achieves 64.4% on English-German vs. 39.8% for DOC2DOC-MT5-1.2B, but only 29.9% on English-French vs. 26.2%, suggesting that the evaluation methodology (discriminative accuracy vs. generative accuracy) matters substantially and that the LLM-based models' advantage is more pronounced in the discriminative setting.

Training Strategy Comparison (Table 6): Two-Stage Training Is Optimal; Adding a Parallel Sentence Stage Provides No Benefit

The paper compares three training strategies on four representative language pairs using LLAMA2-7B with FFT:

• One-stage (direct fine-tuning on parallel documents): Achieves 30.2 sBLEU on Dutch→English and 28.3 on Romanian→English (strong pairs), but only 1.3 on Arabic→English and 0.1 on Chinese→English (weak pairs).
• Two-stage (monolingual, then parallel documents): Achieves 38.9 on Dutch→English, 38.2 on Romanian→English, 2.5 on Arabic→English, 0.1 on Chinese→English. The gains on strong pairs are substantial (+8.7 and +9.9 sBLEU); the gains on weak pairs are minimal.
• Three-stage (monolingual, then parallel sentences, then parallel documents): Achieves 39.1 on Dutch→English, 38.4 on Romanian→English, 2.3 on Arabic→English, 0.3 on Chinese→English — essentially identical to two-stage.

The conclusion is that "both the one-stage and three-stage training strategies are sub-optimal for both high-performing languages (Dutch and Romanian) and low-performing languages (Arabic and Chinese)." The monolingual stage provides necessary target-language grounding that one-stage lacks, while the intermediate sentence-level stage provides no additional benefit — the model can transition directly from monolingual fine-tuning to document-level translation without needing a sentence-level stepping stone. The failure of the low-performing languages to improve substantially even with two-stage training (Arabic→English: 2.5 sBLEU; Chinese→English: 0.1 sBLEU) indicates that the off-target translation problem is not solved by better training strategies — these language pairs have high off-target rates that training strategy alone cannot fix.

Ablation Studies and Robustness Checks

Prompt design (Table 1): Four prompt variations are compared using three LoRA models on four English-centric translation tasks involving German and Chinese. Prompt 4 (interleaved source-target pairs with natural language instruction) achieves the best overall performance: 20.2 µsBLEU for L-7B-LORA, 23.1 for B-7B-LORA, 22.4 for V-7B-LORA. The interleaved structure (Prompt 2) consistently outperforms the separated-context structure (Prompt 1) across all three models, confirming that explicit cross-lingual alignment in the prompt format aids decoder-only LLMs. Natural language instructions benefit base models (L-7B-LORA: 15.5→15.8; B-7B-LORA: 19.3→19.8) but hurt the instruction-tuned model (V-7B-LORA: 19.0→18.3), suggesting that instruction-tuned backbone models may overfit or conflict with additional instruction during fine-tuning.

Decoding strategy (Table 4): The alternative REGEN strategy, which regenerates all context translations from scratch instead of reusing previous model outputs, dramatically reduces off-target translation and improves quality. On Arabic→English, L-7B-LORA + REGEN achieves 17.5 sBLEU and 69.7 COMET, compared to 3.9 sBLEU and 53.9 COMET with the default REUSE strategy. L-7B-FFT shows similar improvement: 15.9 sBLEU vs. 2.5. The cost is 4× inference computation. This ablation confirms that error propagation during autoregressive context reuse, not deficient translation capability, is the root cause of the off-target problem.

Training strategy stages (Table 6): As described in the main results, two-stage training dominates one-stage (direct parallel document fine-tuning) with gains of 8.7–9.9 sBLEU on strong language pairs (Dutch, Romanian), while three-stage (adding an intermediate parallel sentence stage) provides no additional benefit over two-stage. The monolingual stage is essential; the sentence-level stage is unnecessary.

Backbone pre-training language coverage (Tables 3, 16, 17): BLOOM-7B's multilingual pre-training provides robustness against off-target translation that LLAMA2-7B's English-centric pre-training lacks. B-7B-LORA achieves an average off-target rate of 2.8% on X→En (Table 3) and 11.2% on En→X (Table 16), compared to 29.2% and 6.2% for L-7B-LORA. The robustness advantage is most pronounced on the languages where LLAMA2-based models fail catastrophically: Chinese→English (B-7B-LORA 1.6% vs. L-7B-LORA 93.1%), Arabic→English (2.9% vs. 87.9%), Korean→English (8.4% vs. 44.2%).

Instruction-tuned vs. base backbone (Tables 2, 8, 19): VICUNA-7B (instruction-tuned from LLAMA2-7B) provides no consistent benefit over LLAMA2-7B and is substantially worse for zero-shot cross-lingual transfer. V-7B-LORA achieves lower µsBLEU than L-7B-LORA on both En→X (15.8 vs. 17.2) and X→En (21.6 vs. 23.8) in Table 2. On zero-shot transfer (Table 8), V-7B-LORA shows negative transfer (−8.9 average COMET change) while L-7B-LORA shows positive transfer (+29.4). The negative result — that instruction tuning harms rather than helps — is the paper's most important ablation for backbone selection.

Parameter-efficient vs. full fine-tuning (Figures 4, 7): The scaling law analysis in Figure 4 (English→German) and Figure 7 (English→Romanian, English→Chinese) reveals that FFT reaches near-full-dataset COMET performance with approximately 1% of training data (~2K examples), while LoRA requires approximately 10% (~20K examples). At 100% data, LoRA matches or exceeds FFT for most models (e.g., L-7B-LORA at ~77 COMET vs. L-7B-FFT at ~77 COMET on English→German). The crossover pattern is consistent across all three backbones and additional language pairs in Appendix H. This ablation establishes the data-efficiency tradeoff: FFT is preferable in extreme low-resource settings; LoRA is preferable with moderate data or when computational constraints apply.

Monolingual data pruning (Section 3.2): The use of the data pruning technique of Marion et al. (2023) to select 100M tokens per language from CulturaX is motivated by computational constraints and catastrophic forgetting concerns, but the paper does not provide an ablation comparing pruned vs. random monolingual data. The contribution of the pruning step to final performance is therefore unknown.

Context window size (Section 3.1): The choice of three preceding sentence pairs as context follows Wang et al. (2023a) and is not ablated. The paper does not explore whether longer context windows (5, 7, full document) would improve discourse phenomena handling or exacerbate off-target propagation.

LoRA rank (Appendix B): The LoRA rank is fixed at 16 without ablation. The paper does not explore whether higher ranks (32, 64) would narrow or eliminate the gap with FFT in low-data regimes, or whether lower ranks (4, 8) would provide better regularization at scale.

Beam search configuration (Section 3.4): Beam size is fixed at 5 without ablation. The paper does not explore whether greedy decoding (beam size 1) reduces off-target rates by avoiding beam search's tendency to select high-probability but wrong-language hypotheses, or whether larger beams improve quality for robust models like B-7B-LORA.

Evaluation on recent test sets (Table 7): On WMT2023 English-German and German-English test sets, LLM-based DOCMT models substantially outperform conventional DOCMT models. L-7B-LORA achieves 28.9 dBLEU on En→De vs. 21.2 for IADA-MT5-1.2B (the best conventional model), and 35.5 dBLEU on De→En vs. 22.0. COMET scores show similar trends: L-7B-FFT achieves 77.0 COMET on En→De vs. 75.6 for MR-DOC2DOC-MT5, and 84.0 on De→En vs. 76.5. This reversal from the IWSLT2017 results (where conventional models outperformed on En→X) suggests that LLM-based models generalize better to out-of-domain text, while conventional models may overfit to the IWSLT2017 domain. The WMT2023 test sets are described as mitigating "data leakage risks" because they are newer than the training data.

GPT-4-based error analysis prompt (Appendix D, Figure 6): The error analysis uses a detailed MQM-based prompt instructing GPT-4-TURBO to identify specific error types with justifications, output in JSON format. The prompt is provided in full (Figure 6), making the analysis reproducible. However, the paper does not validate GPT-4-TURBO's error classifications against human judgments, relying instead on Kocmi and Federmann (2023)'s finding that "GPT-4 can identify error spans and achieve state-of-the-art MT evaluation accuracy" as external validation of the approach.

Critical Assessment

Claim 1: "Specialized models can sometimes surpass GPT-4 in translation performance"

The evidence for this claim is present but narrowly scoped. Table 2 shows that B-7B-LORA achieves 29.9 µsBLEU on X→En vs. GPT-4-TURBO's 31.7 — a gap, not a surpass. However, the breakdown results in Tables 13–15 reveal specific language pairs where the claim holds: L-7B-LORA on Dutch→English (39.2 vs. GPT-4-TURBO's 36.6 sBLEU), on Romanian→English (39.0 vs. 36.5), and on German→English (33.1 vs. 31.2). B-7B-LORA on Arabic→English (30.9 vs. GPT-4-TURBO's average of 31.7 — the per-pair GPT-4 score for Arabic is not separately reported, making this comparison ambiguous).

What weakens this claim: The "surpass" is limited to X→En direction only and to specific language pairs where the 7B model has low off-target rates. On En→X, no fine-tuned model approaches GPT-4-TURBO (best: L-7B-LORA at 20.2 µsBLEU vs. GPT-4-TURBO's 27.0). On COMET — the metric known to correlate better with human judgment — GPT-4-TURBO dominates across both directions (86.3 and 86.0 vs. best fine-tuned at 81.4 and 70.8). The "surpass" is detectable with BLEU but not with COMET, which suggests the fine-tuned models may be producing translations that match reference n-grams better but are semantically or stylistically inferior. The claim should be qualified: specialized models can match or slightly exceed GPT-4 on BLEU for translation into English from specific European languages, but not on semantic quality (COMET) or translation out of English.

Claim 2: "Off-target translation is the primary failure mode due to error propagation in decoding"

This claim is strongly supported. Table 3 quantifies off-target rates reaching 98.3% for V-7B-FFT on Korean→English, with a clear correspondence between off-target rates and BLEU collapse (Tables 13 and 17). Table 4 demonstrates the causal mechanism: switching from REUSE to REGEN decoding on Arabic→English improves L-7B-LORA from 3.9 to 17.5 sBLEU — a 4.5× improvement that confirms error propagation as the root cause. The pattern generalizes across multiple models (L-7B-FFT also improves dramatically: 2.5→15.9 sBLEU).

What weakens this claim: The REGEN experiment is conducted on only one language pair (Arabic→English). The paper does not demonstrate that REGEN rescues performance on the other failing pairs (Chinese→English, Japanese→English, Korean→English), though the mechanism (error propagation) plausibly generalizes. The 4× cost of REGEN is mentioned but not compared to alternative mitigation strategies (e.g., ensembling multiple models, post-hoc language detection and regeneration of off-target sentences, or mixed REUSE/REGEN schemes). The claim that off-target translation is "the primary failure mode" is well-supported for LLAMA2 and VICUNA models, but the paper does not fully disentangle whether off-target translation causes poor BLEU scores or is merely correlated with them — there could be a third factor (e.g., poor source-language understanding for certain languages) that produces both off-target output and poor-quality in-target output simultaneously.

Claim 3: "PEFT (LoRA) outperforms FFT overall, but FFT shows greater data efficiency"

The overall LoRA advantage is visible in Table 2: L-7B-LORA (17.2 µsBLEU on En→X, 23.8 on X→En) vs. L-7B-FFT (13.7, 22.4); B-7B-LORA (17.7, 29.9) vs. B-7B-FFT (12.0, 22.3). However, the gap is not uniform — V-7B-FFT slightly outperforms V-7B-LORA on X→En (21.8 vs. 21.6). The data-efficiency claim is supported by Figures 4 and 7, which show FFT reaching full-dataset performance with ~1% data across three language pairs and three backbones, while LoRA requires ~10%.

What weakens this claim: The "1% data" finding is based on a single training run per data percentage per model. With the paper's acknowledged training instability ("there are noticeable inconsistencies in performance... too significant to attribute solely to the randomness inherent in training" — Section 8), the scaling curves in Figures 4 and 7 should be interpreted as illustrative rather than precise. The non-monotonic behavior in some curves (e.g., V-7B-FFT in Figure 4) is consistent with training instability. Without multiple random seeds and confidence intervals, the crossover points (1% vs. 10%) are approximate. Additionally, the scaling law comparison is conducted only on En→X translation for three language pairs (German, Romanian, Chinese) — it is unknown whether the same data-efficiency pattern holds for X→En or for the other six language pairs in IWSLT2017.

Claim 4: "Base LLMs perform better than instruction-tuned LLMs for task-specific supervised fine-tuning"

Supported by Table 2 (V-7B-LORA underperforms L-7B-LORA on both directions) and emphatically supported by Table 8 (zero-shot cross-lingual transfer: V-7B-LORA shows −8.9 average COMET decrease vs. L-7B-LORA's +29.4 increase). The transfer result is the stronger evidence because it demonstrates a qualitative difference in behavior: instruction-tuned models lose existing capability when fine-tuned on a single language pair, while base models gain capability that transfers to other languages.

What weakens this claim: The comparison is between exactly two models: LLAMA2-7B and VICUNA-7B. VICUNA-7B is a single instruction-tuned variant of LLAMA2-7B; the finding may be specific to this particular instruction-tuning recipe (ShareGPT conversations) rather than general to instruction-tuned models. Other instruction-tuned variants of LLAMA2-7B (e.g., models fine-tuned with Super-NaturalInstructions, FLAN, or Self-Instruct) might show different behavior. The paper does not ablate whether the negative transfer is due to instruction tuning itself or to catastrophic forgetting during instruction tuning that damaged multilingual representations. The finding is important but should be framed as "VICUNA-7B, a specific instruction-tuned model, underperforms its base counterpart" rather than the general claim that instruction-tuned models are worse for task-specific fine-tuning.

Claim 5: "LLM-based DOCMT models generalize better on out-of-domain text" (from WMT2023 evaluation, Table 7)

This is a striking reversal from the IWSLT2017 results. On IWSLT2017 En→De, conventional DOCMT models dominated (Table 2: DOCFLAT-MT5-1.2B at 19.2 µsBLEU vs. L-7B-LORA at 17.2). On WMT2023 En→De, the pattern flips: L-7B-LORA achieves 28.9 dBLEU vs. 21.2 for IADA-MT5-1.2B; L-7B-FFT achieves 29.0 vs. 21.2. The COMET results similarly favor LLM-based models (77.0 for L-7B-FFT vs. 75.6 for MR-DOC2DOC-MT5 on En→De).

What weakens this claim: The WMT2023 evaluation uses a different metric (dBLEU only in Table 7, compared to µsBLEU in Table 2) and a different dataset (WMT2023 vs. IWSLT2017). The paper segments documents using spaCy and discards documents where source and target have different numbers of sentences — this filtering may introduce bias by removing difficult cases. The WMT2023 test sets are only for English-German in both directions — the generalization claim is supported for exactly one language pair. Without WMT2023 results for other language pairs or other test sets, the claim that LLM-based models "generalize better on out-of-domain text" is supported only for English-German. It could be that LLM-based models are specifically better at German (which has relatively high resource in LLAMA2 pre-training) and would not show the same advantage for lower-resource WMT languages.

Unaddressed Concerns

• Single benchmark, single domain (TED talks): All main experiments use IWSLT2017, a specific domain (spoken presentations) with specific characteristics (informal register, first-person narrative, relatively short documents). The paper acknowledges this implicitly by evaluating on WMT2023 as an out-of-domain test, but only for one language pair. The generalizability of findings to other domains (news, legal, medical, literary) is unknown.

• Small test sets: The IWSLT2017 test sets contain only 10–12 documents per language pair (1,100–1,500 sentences). The WMT2023 test set size is not reported. With such small test sets, per-language-pair results have high variance, and a few bad documents can dramatically affect averages. The paper does not report confidence intervals or statistical significance tests.

• Training instability unaddressed: The paper acknowledges training instability (Section 8) but does not incorporate it into the experimental design. Training multiple seeds and reporting variance would have substantially strengthened the reliability of the scaling law results and the fine-tuning method comparisons. As it stands, the reported numbers may be best-of-several-runs rather than expected values.

• Missing ablation: monolingual data pruning: The paper uses data pruning (Marion et al., 2023) for Stage 1 but does not compare against random sampling of monolingual data. The contribution of the pruning step to the effectiveness of Stage 1 is unknown.

• Missing comparison: larger fine-tuned LLMs: The paper studies only 7B models, arguing that "moderately-sized LLMs often outperform larger ones after task-specific fine-tuning" (abstract). But no 13B or 70B fine-tuned model is evaluated to test this claim. The comparison between 7B fine-tuned models and unfine-tuned GPT-4 conflates scale with fine-tuning — a 13B LLAMA2 fine-tuned on the same data might outperform the 7B version, challenging the "moderately-sized is sufficient" narrative.

• COMET vs. BLEU discrepancy: The paper reports both BLEU and COMET but does not deeply analyze cases where they diverge. For example, B-7B-LORA achieves 29.9 µsBLEU on X→En but only 81.4 µCOMET, while GPT-4-TURBO achieves 31.7 µsBLEU and 86.0 µCOMET. The 4.6-point COMET gap is substantially larger than the 1.8-point BLEU gap, suggesting that BLEU understates GPT-4-TURBO's quality advantage. The paper's conclusion that fine-tuned models "can sometimes surpass GPT-4 in translation performance" relies primarily on BLEU, which may be the less reliable metric.

• REGEN cost not quantified in context of deployment: The paper presents REGEN as a solution to off-target translation at 4× cost, but does not compare this cost to alternatives that might achieve similar robustness (e.g., using BLOOM-7B instead of LLAMA2-7B, which achieves low off-target rates with REUSE; or using post-hoc language detection to identify and regenerate off-target sentences only). A cost-benefit analysis across these alternatives would strengthen the practical recommendation.

6. Limitations and Trade-offs

The Off-Target Translation Problem Is Diagnosed but Not Solved

The assumption or constraint. The paper demonstrates conclusively that off-target translation — the model producing text in the wrong language — is the dominant failure mode, with error propagation during autoregressive context reuse as the root cause (Section 6, Tables 3–4). However, the proposed fix (REGEN decoding) carries a 4× inference cost that the paper presents as prohibitive, and no low-cost alternative is developed or evaluated. The paper explicitly acknowledges this gap only implicitly, by framing REGEN as a diagnostic tool rather than a practical solution: "These findings highlight the main reason for translation failures in LLM-based DOCMT models, offering insights for future research."

The consequence. A practitioner deploying LLM-based DOCMT faces an unresolved tradeoff: use REUSE and accept catastrophic failure on certain language pairs (off-target rates up to 98.3% for V-7B-FFT on Korean→English, Table 3), or use REGEN and accept 4× inference cost that may make deployment economically infeasible at scale. The paper provides no middle ground — no mixed strategy (e.g., language-detection-based selective regeneration of off-target sentences only), no architectural modification to the model or prompt that reduces language confusion without full regeneration, and no characterization of which language pairs or document types trigger the failure cascade. For any production system where documents contain hundreds of sentences and cost-per-token matters, neither REUSE nor REGEN is satisfactory.

What evidence exists in the paper. Table 4 provides the only REGEN experiment, on a single language pair (Arabic→English) with two models (L-7B-LORA and L-7B-FFT). The improvement is dramatic — 3.9→17.5 sBLEU — but the experiment is not replicated for the other failing pairs (Chinese→English, Japanese→English, Korean→English) or for any En→X direction. The paper quantifies the off-target problem comprehensively across all language pairs and models (Tables 3, 16, 17), establishing the scope of the failure, but the REGEN ablation covers only one pair.

Mitigation status. Not addressed. The paper identifies BLOOM-7B as a backbone that largely avoids off-target translation through its multilingual pre-training (B-7B-LORA achieves only 2.8% average off-target rate on X→En, Table 3), but this is a model selection workaround rather than a solution — it means abandoning the LLAMA2 and VICUNA backbones entirely for deployment, losing any advantages they might have on metrics like discourse phenomena (Table 5, where BLOOM-based models underperform on pronoun resolution). The paper suggests future work on the problem but proposes no concrete method beyond REGEN.

The Difficulty Estimation for Off-Target Risk Is Post-Hoc and Unavailable at Deployment

The assumption or constraint. The paper's central diagnostic — that off-target translation, not poor translation capability, is the primary failure mode — relies on measuring off-target rates using the fastText language identification tool on model outputs after translation is complete (Section 6, Tables 3, 16, 17). At deployment time, there is no mechanism to predict in advance whether a given language pair, document, or sentence will trigger off-target failure, nor to detect it during the autoregressive decoding chain early enough to intervene before the error propagates through the entire document.

The consequence. A deployed system would either (a) always use REUSE and silently produce wrong-language output on certain language pairs with no way to detect or recover, or (b) always use REGEN at 4× cost regardless of whether the document actually needs it, or (c) use an untested heuristic to decide. None of these options is satisfactory. The paper provides no signal that could be computed at inference time — e.g., a confidence score, an entropy-based measure, or a per-token language probability — that would indicate that the model is about to produce off-target output. Without such a signal, the system is blind to its own failures.

What evidence exists in the paper. The off-target rate tables (Tables 3, 16, 17) are computed on the entire test set after generation. The paper does not report any per-sentence or per-document indicator that correlates with off-target risk. The REGEN experiment (Table 4) demonstrates that the problem is fixable with oracle knowledge of which decoding strategy to use, but provides no method for making that decision without ground-truth labels. The paper does not analyze whether off-target translation clusters in specific documents (e.g., documents with certain discourse structures, certain source-language properties) or occurs uniformly.

Mitigation status. Not addressed. The paper does not propose or evaluate any runtime detection mechanism for off-target translation. This is a significant gap between diagnosis and remedy: identifying the problem is necessary but not sufficient for deployment.

The Scaling Law and Data Efficiency Claims Come from Single Training Runs Without Stability Quantification

The assumption or constraint. All reported results, including the scaling law curves (Figures 4 and 7, Appendix H) and the main comparison table (Table 2), are from single training runs. The paper explicitly acknowledges training instability as a limitation in Section 8:

"The process of supervised fine-tuning for LLMs shows instability in our observations. As detailed in Figure 4, there are noticeable inconsistencies in performance. These variations are too significant to attribute solely to the randomness inherent in training. In some cases, the fine-tuning of LLMs fails to reach convergence."

However, the paper does not incorporate this instability into its experimental design — no multiple random seeds, no confidence intervals, no characterization of which results are reliable and which are noise.

The consequence. The paper's most actionable quantitative findings — that FFT reaches full-dataset performance with ~1% of training data vs. ~10% for LoRA, and that LoRA generally outperforms FFT — are reported as point estimates from single runs that the authors themselves describe as unstable. A practitioner deciding between FFT and LoRA based on these numbers cannot know whether the observed difference (e.g., L-7B-LORA at 77 COMET vs. L-7B-FFT at 77 COMET at 100% data in Figure 4) is a stable effect or a draw from a high-variance distribution where the two methods are indistinguishable. The non-monotonic behavior in some scaling curves (e.g., V-7B-FFT in Figure 4, where performance dips at intermediate data percentages) is consistent with training instability rather than a genuine data-efficiency pattern, making the 1%-data claim less reliable than it appears.

What evidence exists in the paper. The instability is visible in the scaling curves themselves — they are not smooth, and several models show performance decreases when more data is added, which is unexpected for well-behaved learning curves. The paper provides no quantification: no variance across seeds, no report of how many training runs failed to converge (and were discarded), no description of the selection criteria for which runs to report. The authors' own acknowledgment in Section 8 confirms that they observed the problem and chose not to address it systematically, citing resource constraints.

Mitigation status. The paper acknowledges the limitation in Section 8 but defers investigation to future work: "Our limited resources restrict us from investigating these failures in depth or devising potential remedies." The acknowledgment is honest but does not retroactively make the single-run results reliable.

The Evaluation Is Confined to a Single Domain and a Single Metric Paradigm, with Limited Ecological Validity for DOCMT

The assumption or constraint. All main experiments use the IWSLT2017 benchmark, which consists of TED talk transcripts — a specific domain (prepared spoken presentations, informal register, first-person narrative, relatively short documents averaging approximately 120 sentences each). The paper adds a WMT2023 evaluation for English-German only (Table 7) as an out-of-domain test, but this covers one language pair out of nine. The paper does not evaluate on other document-level MT benchmarks (e.g., news commentary, legal, literary, or technical domains) or on languages beyond the IWSLT2017 set.

The consequence. The finding that "LLM-based DOCMT models generalize better on out-of-domain text" (Section 6, supported by Table 7) is supported for exactly one language pair (English-German) on one out-of-domain test set (WMT2023). It is unknown whether this advantage holds for other language pairs, particularly those where LLM-based models exhibit high off-target rates (Arabic, Chinese, Japanese, Korean). More broadly, the TED talk domain has specific discourse properties — high density of first- and second-person pronouns ("I", "you", "we"), relatively simple sentence structure, and strong local cohesion because speakers develop ideas linearly — that may make it unusually favorable for a fixed three-sentence context window. In domains with longer-distance discourse dependencies (legal contracts, scientific papers with cross-section references) or more diverse registers, the three-sentence window may be insufficient, and the LLM-based models' handling of longer contexts (which they are architecturally capable of, unlike the fixed-window DOCMT baselines) is untested.

What evidence exists in the paper. The IWSLT2017 results are comprehensive across nine language pairs and three metrics. The WMT2023 results (Table 7) cover one language pair, English-German, using dBLEU and COMET. The paper does not analyze why the LLM-based models outperform on WMT2023 when they underperform on IWSLT2017 En→De — is it better generalization, data leakage in the conventional models' training, or some property of the WMT2023 domain? The discourse phenomena evaluation (Table 5) uses targeted contrastive test sets for English-German and English-French, which is domain-agnostic but tests only a specific discourse phenomenon (pronoun resolution).

Mitigation status. Partially addressed by the WMT2023 evaluation, but this covers only 2 of 18 translation tasks. The paper does not claim domain-generality, and its conclusions are appropriately scoped to the IWSLT2017 setting, but the practical question — "will this work for my domain?" — is unanswered.

The 14× Larger Baseline (GPT-4) Is Not Task-Adapted, and the Comparison Confounds Scale with Adaptation

The assumption or constraint. The paper's headline claim that "specialized models can sometimes surpass GPT-4 in translation performance" (abstract, Section 1) compares fine-tuned 7B models against GPT-3.5-TURBO and GPT-4-TURBO evaluated zero-shot — without any task-specific fine-tuning, few-shot examples, or prompt optimization beyond using Prompt 4. GPT-4-TURBO's performance (31.7 µsBLEU on X→En, Table 2) is achieved entirely through its pre-training and instruction-tuning, with no access to IWSLT2017 training data or even a few in-domain exemplars. The fine-tuned 7B models, in contrast, are trained on the full IWSLT2017 training set (~200K sentence pairs) and specifically optimized for the prompt format and translation direction.

The consequence. The comparison does not isolate the effect of model scale from the effect of task-specific adaptation. It is possible — and indeed likely, given GPT-4's known few-shot capabilities — that providing GPT-4-TURBO with even a handful of IWSLT2017-style in-context examples, or fine-tuning a GPT-4-level model (were the weights accessible), would substantially widen the performance gap. The claim that "specialized models can surpass GPT-4" is therefore more precise when stated as: "a fine-tuned 7B model on IWSLT2017 can occasionally exceed a zero-shot GPT-4 on BLEU for specific language pairs." The practical relevance of this comparison depends on whether the deployment scenario allows fine-tuning on in-domain data — if it does, the comparison to zero-shot GPT-4 is fair; if the goal is a general translator without task-specific data, the 7B fine-tuned model is not available either (since it requires IWSLT2017 training data).

Additionally, GPT-4-TURBO is not given any test-time compute budget (e.g., majority voting, best-of-N, or self-consistency decoding) that could further improve its performance — it is evaluated with a single generation, analogous to the fine-tuned models' beam-search output. The paper does not discuss whether GPT-4's performance could be improved with chain-of-thought or multi-step refinement prompting, which have been shown to help on translation tasks (Lu et al., 2023).

What evidence exists in the paper. Table 2 includes GPT-3.5-TURBO and GPT-4-TURBO as baselines, evaluated zero-shot with Prompt 4. No few-shot, fine-tuned, or compute-scaled GPT variants are tested. The paper does not discuss this as a limitation of the comparison.

Mitigation status. Not addressed. The paper presents the GPT-4 comparison as a strength — evidence that small specialized models can compete with massive general ones — without discussing the confound between scale and adaptation. The finding is not invalid, but its interpretation requires care that the paper does not provide.

The Two-Stage Training Strategy Adds Computational Cost That Is Not Amortized in the Headline Performance Numbers

The assumption or constraint. All fine-tuned models undergo Stage 1 (monolingual fine-tuning on 1B tokens across 10 languages) before Stage 2 (parallel document fine-tuning on IWSLT2017). The Stage 1 cost — full-parameter fine-tuning of a 7B model on 1B tokens — is substantial and is incurred before any translation-specific training. The paper compares the performance of these two-stage models against conventional DOCMT models (trained from MT5 on IWSLT2017 only) and sentence-level NLLB models (pre-trained on massive multilingual data, then fine-tuned on parallel sentences), but does not account for the Stage 1 cost in any FLOPs or GPU-hour comparison.

The consequence. A fair cost-effectiveness comparison would ask: given a fixed total compute budget for building a DOCMT system (including all pre-training, continued training, and fine-tuning), how does the two-stage LLM approach compare to training a specialized encoder-decoder model from scratch, or to fine-tuning a pre-trained multilingual model like NLLB? The paper's Table 2 shows that NLLB-3.3B achieves 26.8 µsBLEU on En→X and 25.8 on X→En — substantially outperforming all fine-tuned 7B models on En→X (best: 20.2) and competitive with them on X→En (best: 29.9 vs. 25.8). NLLB's pre-training is massive but amortized across hundreds of languages; the 7B LLMs' Stage 1 is specific to the 10 target languages of this study. Without cost accounting, a practitioner cannot determine whether the LLM-based approach is economically rational or merely interesting.

What evidence exists in the paper. The paper reports the number of trainable parameters (Table 2) and provides hyperparameter details (Appendix B), but does not compute total FLOPs, GPU-hours, or wall-clock time for either training stage. The ablation in Table 6 compares one-stage vs. two-stage training, showing that Stage 1 is necessary for performance, but does not quantify its cost relative to the gain.

Mitigation status. Not addressed. The paper acknowledges resource constraints in Section 8 ("Constraints on Model Scale") as limiting the study to 7B models, but does not discuss the cost of the training strategy itself as a limitation of the method's practicality. The finding that FFT reaches full-dataset performance with ~1% of IWSLT2017 data (Figure 4) partially mitigates this concern — if only 2K parallel sentences are needed for Stage 2, the Stage 1 cost dominates and the question of whether Stage 1 is worth its cost becomes central, yet it is not analyzed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around LLM-based document-level machine translation from a simple "can LLMs do DOCMT?" question — which prior work answered with a tentative but uninformative "sometimes, for GPT-4" (Wang et al., 2023b) — toward a structured diagnostic framework that separates translation capability from decoding robustness, identifies specific failure modes, and provides actionable guidance on backbone selection, fine-tuning strategy, and prompt design. The magnitude of the contribution is not a paradigm shift (the methods — PEFT, two-stage training, prompt engineering — are all established techniques) but rather a systematic empirical reframing: the paper reorganizes the DOCMT problem around the off-target translation bottleneck, showing that many apparent failures of fine-tuned LLMs are not failures of learned translation skill but failures of the autoregressive context-reuse decoding pipeline.

This reframing has several concrete consequences for how researchers and practitioners should think about the problem:

The quality-robustness distinction becomes a first-class evaluation criterion. Before this work, DOCMT systems were compared almost exclusively on aggregate quality metrics — sBLEU, dBLEU, COMET — averaged across test sets. Table 2 in this paper shows that such averaging obscures catastrophic bimodality: a model with 98.3% off-target rate on Korean→English (V-7B-FFT, Table 3) and a model with 1.6% off-target rate on Chinese→English (B-7B-LORA) can be summarized by the same µsBLEU. The paper establishes off-target rate measurement (via fastText language identification) as a cheap, automated complement to quality metrics, and demonstrates that multilingual pre-training (BLOOM-7B) provides robustness that quality metrics alone fail to capture. The implication is that future DOCMT evaluations must report robustness metrics alongside quality metrics — a practice that the sentence-level MT community has not needed because sentence-level translation has no autoregressive context chain to degrade.

The decoder-only LLM architecture's mismatch with document-level decoding is identified as the core technical challenge. Encoder-decoder DOCMT systems (DOC2DOC-MT5, DOCFLAT-MT5, etc.) process the entire source context in a single encoder pass, then generate the target sentence in a single decoder pass — the translation of sentence i does not causally depend on the quality of the translation of sentence i-1 beyond what the model learned during training. Decoder-only LLMs, by contrast, must generate target sentences autoregressively and feed their own previous outputs back as conditioning context, creating a feedback loop where errors compound. This architectural difference was known before this work (it follows directly from the decoder-only design), but its practical severity — causing near-total collapse on language pairs with 93-98% off-target rates — was not documented. The paper's REGEN experiment (Table 4) demonstrates that breaking this feedback loop restores most of the model's translation capability, confirming that the architecture, not the training, is the bottleneck. This finding redirects research attention from better fine-tuning recipes (which cannot fix a decoding-time problem) toward better decoding strategies or architectural modifications that provide the robustness of encoder-decoder models while retaining the flexibility of decoder-only LLMs.

The instruction-tuning-as-universal-benefit narrative receives a concrete counterexample. The consistent underperformance of VICUNA-7B relative to LLAMA2-7B on DOCMT — particularly the negative zero-shot cross-lingual transfer (Table 8: V-7B-LORA shows −8.9 average COMET decrease when transferring from English-German fine-tuning to other languages, while L-7B-LORA shows +29.4 increase) — challenges the prevailing assumption that instruction-tuned models are always better starting points for downstream fine-tuning. This finding matters beyond translation because it identifies a boundary condition: when the downstream task involves extensive further supervised training on a fixed-format dataset, the instruction-tuning prior can interfere with learning rather than accelerate it. The paper does not explain the mechanism (catastrophic forgetting of multilingual capabilities during instruction tuning? overfitting of instruction-following mechanisms to the specific fine-tuning prompt?), but the empirical result is clear enough to justify caution when selecting backbones for task-specific fine-tuning in any domain.

The data-efficiency crossover between FFT and LoRA refines fine-tuning best practices. The finding that FFT reaches full-dataset performance with ~1% of IWSLT2017 training data while LoRA requires ~10% (Figure 4) contradicts the intuition that parameter-efficient methods are more data-efficient (fewer parameters = less data needed) and instead suggests that FFT's full-parameter access enables rapid few-shot-style adaptation that LoRA's low-rank constraint limits. This provides concrete guidance for low-resource language pairs: if only 2K parallel document sentences exist, use FFT; if 20K+ exist, LoRA's regularization advantage and computational efficiency make it the better choice. The paper also shows that this crossover varies by backbone (the gap between FFT and LoRA at 1% data is more pronounced for some models than others, Figure 4), suggesting that the interaction between pre-training and fine-tuning capacity is nuanced and model-specific.

The paper provides a reconciliation template for contradictory findings on LLM translation quality. Prior work has produced conflicting results: some studies find that LLMs match or exceed supervised NMT (Hendy et al., 2023, for high-resource languages with GPT-4), while others find substantial gaps (Robinson et al., 2023, for low-resource languages). The paper's breakdown results (Figure 2, Tables 13–15) show that the same fine-tuned model can simultaneously surpass GPT-4 on Dutch→English (39.2 vs. 36.6 sBLEU) and collapse to near-zero on Chinese→English (0.1 sBLEU) — the apparent contradiction is resolved by recognizing that performance is bimodal across language pairs, and studies that happen to sample different subsets of languages will reach opposite conclusions. The paper provides the diagnostic tool (off-target rate measurement) and the explanatory mechanism (error propagation) that allow future work to predict which language pairs will succeed and which will fail, rather than reporting averages that conflate the two regimes.

Follow-Up Research This Work Enables

Adaptive or selective REGEN: dynamic decoding strategies that trade cost for robustness only when needed. The paper demonstrates that REGEN (regenerating all context translations from scratch) restores translation quality on failing language pairs at 4× inference cost (Table 4), but provides no method for deploying this strategically. A natural extension is a selective REGEN strategy: use a lightweight language-detection classifier (e.g., fastText, as the paper already uses for measurement) to monitor each generated translation as it is produced. If the classifier detects off-target output with high confidence, regenerate that specific sentence's context translations (or the sentence itself) using REGEN, while keeping REUSE for the majority of sentences that stay on-target. This could recover most of REGEN's robustness benefit at a fraction of its cost, since off-target translation, while catastrophic when it occurs, affects only a subset of sentences even within failing language pairs (Table 3 reports document-level rates, not per-sentence rates). The experiment would measure: (a) off-target rate and BLEU/COMET under selective REGEN vs. full REGEN vs. REUSE, (b) the fraction of sentences that trigger regeneration, and (c) the resulting total inference cost relative to full REGEN and REUSE. A strong result would show that selective REGEN achieves >90% of full REGEN's quality improvement at <50% of its additional cost.

Teacher-forcing during training with scheduled sampling to make the model robust to its own errors. The paper identifies error propagation during autoregressive context reuse as the root cause of off-target translation, but the solution (REGEN) operates entirely at inference time. A complementary approach is to modify the training procedure so that the model learns to be robust to noisy context. During parallel document fine-tuning (Stage 2), the model always sees gold target-language context (the reference translations from the training data), never its own potentially erroneous output. This creates a train-test mismatch: the model learns to condition on clean context but must condition on self-generated noisy context at inference time. Scheduled sampling (Bengio et al., 2015) would address this by occasionally replacing reference context sentences with model-generated context during training, with the probability of using model-generated context increasing over the course of training. By the end of training, the model has learned to recover from (or at least not be derailed by) imperfect context. The experiment would compare off-target rates and BLEU/COMET for models trained with vs. without scheduled sampling, ideally measuring performance under both REUSE and REGEN to disentangle whether scheduled sampling improves robustness (helps REUSE) or merely improves overall translation quality (helps both). The paper's finding that BLOOM-7B is naturally more robust (Table 3) suggests that scheduled sampling might be particularly impactful for LLAMA2-based models, which have the largest gap between REUSE and REGEN performance.

Long-context fine-tuning: replacing the fixed three-sentence window with full-document context to test whether LLMs can exploit their architectural context capacity for DOCMT. The paper uses a fixed context window of three preceding sentence pairs (following Wang et al., 2023a), which is a legacy constraint from encoder-decoder DOCMT systems with limited context capacity. Decoder-only LLMs, however, can attend to substantially longer contexts — LLAMA2-7B has a 4096-token context window, and subsequent models (LLAMA-3, Mistral) support 8K–32K tokens. Full-document context would allow the model to access discourse phenomena spanning longer distances than three sentences (e.g., topic shifts, section-level coherence, coreference chains that span paragraphs). This is particularly relevant for domains beyond IWSLT2017's short TED talks — news articles, legal documents, and academic papers have discourse structures that operate at paragraph and section scales. The experiment would fine-tune and evaluate LLM-based DOCMT models with full-document context (truncated to the model's maximum context length) vs. the three-sentence window, measuring both standard metrics and discourse-phenomena accuracy (using the contrastive test sets from Müller et al., 2018, and Lopes et al., 2020, as in Table 5). The key question is whether full-document context improves discourse handling (as the LLM's architecture would predict) or whether it instead exacerbates off-target propagation (as the paper's error-propagation analysis would warn). The negative result — full-document context increases off-target rates by providing the model with more opportunities to condition on its own errors — would be as informative as the positive one.

Cross-lingual transfer fine-tuning: using high-resource language pairs to improve low-resource pairs within the same model. The paper's zero-shot cross-lingual transfer experiment (Table 8) tests whether an English-German fine-tuned model can translate English→Arabic without Arabic-specific training — a setting that requires the model to transfer both the translation task and the target language generation to a new language pair. A less ambitious but more practical transfer scenario is multi-language-pair fine-tuning: train a single model on parallel documents for multiple language pairs simultaneously (e.g., English→{German, French, Dutch, Romanian} — all pairs where the LLM-based models succeed), then evaluate on held-out pairs (e.g., English→{Arabic, Chinese} — failing pairs). The hypothesis is that training on multiple source-target language mappings teaches the model a more abstract "document translation" skill that is less reliant on per-language-pair representation quality, and that the multilingual training signal may improve target-language representations even for languages not directly trained on (via shared subword tokens or cross-lingual parameter sharing). The experiment would compare: (a) single-pair fine-tuning on English→Arabic only (the failing baseline), (b) multi-pair fine-tuning on English→{De, Fr, Nl, Ro} with zero-shot evaluation on English→Arabic, and (c) multi-pair fine-tuning including English→Arabic. The paper's existing zero-shot transfer results (Table 8) show that LLAMA2-7B transfers positively (+36.3 COMET on Arabic after English-German fine-tuning), suggesting that multi-pair training could substantially close the gap to single-pair supervised performance.

Verifier-guided decoding: using a quality estimation model to detect and reject off-target or low-quality translations during autoregressive generation. The paper's REGEN strategy regenerates all context translations, regardless of whether they are actually erroneous — the 4× cost is paid uniformly. A more efficient approach would use a lightweight quality estimation (QE) model or reference-based COMET scorer to evaluate each generated translation immediately after it is produced. If the QE score falls below a threshold (indicating likely off-target or poor-quality output), the system triggers REGEN for that sentence; otherwise, it proceeds with REUSE. The QE model could be as simple as a language ID classifier (fastText, as used in the paper) combined with a confidence estimate from the LLM itself (e.g., average token probability under beam search). The key challenge is setting the threshold to balance false positives (unnecessary regeneration, wasting compute) against false negatives (missed off-target translations, allowing error propagation). The experiment would sweep QE thresholds and measure the tradeoff curve between inference cost (relative to REUSE and REGEN) and translation quality (BLEU/COMET). This direction is enabled by the paper's quantification of the off-target problem (Tables 3, 16, 17) and the REGEN cost baseline (4×), which together provide precise targets for how much cost reduction is achievable without quality degradation.

Replication on other LLM architectures and at larger scales to test the generality of the off-target robustness finding. The paper establishes that multilingual pre-training (BLOOM-7B) provides off-target robustness that English-centric pre-training (LLAMA2-7B) lacks, but this finding is from a single multilingual model (BLOOM) compared to a single English-centric model (LLAMA2) at a single scale (7B). A systematic replication would test: (a) other English-centric models (Mistral-7B, Falcon-7B, Gemma-7B), (b) other multilingual models (mT5-based decoders, Aya-23, Tower), and (b) larger scales of both types (LLAMA2-13B/70B vs. BLOOM-176B or instruction-tuned variants) to determine whether the robustness advantage of multilingual pre-training is a general property or specific to BLOOM's particular training data mixture. The experiment would also test whether the robustness gap narrows at larger scales — GPT-4-TURBO's strong performance on all language pairs (Table 2, no off-target issues reported) suggests that scale alone may eventually confer robustness, and the scaling threshold at which this occurs is practically important. The paper's finding that VICUNA-7B (instruction-tuned LLAMA2-7B) shows worse cross-lingual transfer than base LLAMA2-7B (Table 8) should also be replicated with other instruction-tuned variants to test whether the negative transfer is specific to VICUNA's ShareGPT-based tuning or general to instruction tuning.

Error-type-specific evaluation beyond pronoun resolution to verify the MQM-based GPT-4 finding that LLMs produce fewer document-level errors despite lower BLEU. The paper's error analysis (Figure 3) uses GPT-4-TURBO to classify errors according to MQM categories and finds that L-7B-LORA and L-7B-FFT produce fewer cohesion and coherence errors than Google Translate and DOC2DOC-MT5-1.2B, despite achieving lower BLEU scores on En→X (Table 2). This is a provocative finding because it suggests that current automatic metrics penalize LLM-based models for something other than document-level quality — perhaps disfluency, register mismatch, or lexical choices that differ from the reference but are not incorrect. Validating this finding requires human evaluation (professional translators rating translations on MQM error categories) on a sample of the IWSLT2017 test sets, comparing LLM-based models against the best conventional DOCMT baselines. If human evaluation confirms that LLM-based models produce fewer document-level errors, this would be strong evidence that the metrics themselves need rethinking for DOCMT, and that the LLM-based models' apparent underperformance on En→X in Table 2 is partially a measurement artifact. If human evaluation contradicts the GPT-4-based error analysis, this would reveal either that GPT-4 is an unreliable evaluator for this task or that the error-type analysis prompt (Figure 6, Appendix D) needs refinement. Either outcome would be valuable.

Practical Applications and Downstream Use Cases

Cost-effective multilingual document translation with lower hardware requirements. The paper's finding that fine-tuned 7B models can approach or match GPT-4-TURBO on specific language pairs — B-7B-LORA achieves 30.9 sBLEU on Arabic→English (Table 13) and L-7B-LORA achieves 39.0 on Romanian→English, compared to GPT-4-TURBO's average of 31.7 on X→En — enables a deployment model where organizations run their own translation systems on modest GPU hardware (a single A100 can serve a 7B model) rather than paying per-token API costs to external providers. This is particularly relevant for privacy-sensitive document translation (legal contracts, medical records, internal corporate communications) where sending text to external APIs is prohibited, and for high-volume scenarios where API costs accumulate rapidly. The scaling law finding (Figure 4) that FFT reaches full-dataset performance with ~1% of IWSLT2017 data (~2K sentence pairs) means that even organizations with limited parallel document resources can build capable systems: 2K sentence-aligned parallel documents is an achievable annotation budget for many language pairs, substantially lower than the ~200K used in the full IWSLT2017 training. For languages where BLOOM-7B provides robustness (low off-target rates, Table 3), the system can use standard REUSE decoding with no additional cost; for other backbones, the REGEN strategy at 4× cost provides a fallback, though a more practical solution would combine BLOOM's robustness with selective REGEN for particularly challenging documents.

Domain-adaptive translation systems that generalize better than conventional supervised models on out-of-domain text. The WMT2023 evaluation (Table 7) shows that LLM-based DOCMT models substantially outperform conventional DOCMT models on out-of-domain English-German translation: L-7B-LORA achieves 28.9 dBLEU on WMT2023 En→De vs. 21.2 for IADA-MT5-1.2B, representing a ~36% relative improvement. For organizations that need to translate documents across multiple domains (e.g., a news agency translating articles on politics, sports, science, and culture), a single LLM-based DOCMT model fine-tuned on one domain (here, TED talks) may provide better out-of-the-box generalization to other domains than a conventional DOCMT model trained on the same data. This is particularly valuable when in-domain parallel documents are scarce — the LLM's broader pre-training provides a domain-robustness prior that task-specific DOCMT architectures lack. The practical deployment would involve fine-tuning a 7B LLM on whatever parallel documents are available (even from a different domain), then deploying it across multiple domains with the expectation that quality degrades gracefully rather than collapsing, as conventional models might.

Discourse-sensitive translation where pronoun resolution and lexical cohesion are critical. The contrastive test set evaluation (Table 5) shows that LLAMA2-based DOCMT models achieve substantially higher accuracy on discourse-dependent pronoun translation than conventional DOCMT models: L-7B-LORA achieves 83.1% accuracy on English-German vs. 77.0% for DOC2DOC-MT5-1.2B; 95.1% on English-French vs. 89.9%. For applications where pronoun correctness is critical — translating legal documents where "he" vs. "she" has legal implications, translating medical records where patient gender must be preserved, translating literature where character reference matters — the 6-8 percentage point accuracy improvement is practically significant. The deployment scenario is a targeted translation pipeline where a conventional system handles the bulk of translation (speed, cost) but a fine-tuned LLM re-translates sentences containing pronouns or other discourse-sensitive elements (as identified by a lightweight classifier), providing higher accuracy where it matters while maintaining throughput for the majority of sentences that are discourse-insensitive.

When to Prefer This Method

The paper does not explicitly frame its contribution as a method to be preferred over named alternatives under specific conditions — it is an empirical analysis that compares existing methods (FFT vs. LoRA, two-stage vs. one-stage training, REUSE vs. REGEN decoding, LLAMA2 vs. BLOOM vs. VICUNA backbones) and identifies failure modes. The findings do, however, support several conditional recommendations that practitioners can extract:

• Prefer BLOOM-7B as the backbone when deploying DOCMT systems that must handle diverse language pairs without language-specific tuning or monitoring, because its multilingual pre-training provides off-target translation robustness (average 2.8% off-target rate on X→En with LoRA, Table 3) that English-centric models lack. The cost is potentially lower peak performance on discourse phenomena (Table 5: BLOOM-based models underperform on pronoun resolution), but this tradeoff favors robustness for most production scenarios where silent wrong-language output is unacceptable.

• Prefer FFT over LoRA when parallel document data is extremely scarce (fewer than ~2K sentence pairs per language pair), because FFT reaches near-full-dataset performance with ~1% of IWSLT2017 data while LoRA requires ~10% (Figure 4). The higher computational cost of FFT training (updating all 7B parameters vs. 8M) is justified by the data efficiency gain in low-resource settings. When data exceeds ~20K sentence pairs, LoRA's regularization advantage and lower training cost make it the better choice.

• Prefer base LLMs (LLAMA2-7B) over instruction-tuned variants (VICUNA-7B) for task-specific DOCMT fine-tuning, because instruction-tuned backbones show no consistent quality advantage (Table 2) and substantially worse zero-shot cross-lingual transfer (Table 8: −8.9 vs. +29.4 average COMET change). This recommendation is specific to fine-tuning on a fixed-format dataset; for zero-shot or few-shot DOCMT without fine-tuning, instruction-tuned models may still be preferable due to their ability to follow prompt instructions without additional training.

• Prefer two-stage training (monolingual, then parallel documents) over one-stage or three-stage variants (Table 6), because the monolingual stage provides necessary target-language grounding and the additional sentence-level intermediate stage provides no benefit. The monolingual stage cost (full-parameter fine-tuning on 1B tokens) should be weighed against the performance gain, particularly for language pairs with sufficient parallel document data where the gain may be modest.