Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation

🎯 Pitch

Despite two decades of methodological revolution in NLG—from hand-crafted pipelines to neural end-to-end models—this survey reveals that no single approach triumphs universally, as the paper finds that effectiveness depends crucially on application domain and input type, while exposing a striking disconnection: the evaluation metrics that dominate the field often fail to correlate with human quality judgments.

1. Executive Summary

This paper surveys the state of the art in Natural Language Generation (NLG), defined as generating text or speech from non-linguistic input, synthesizing research on the field's core tasks, architectures, and evaluation methods as they have evolved over the two decades since Reiter and Dale (2000). The survey analyzes the traditional modular pipeline of six sub-tasks—content determination (choosing which information to include), text structuring (ordering that information), sentence aggregation (grouping related messages into sentences), lexicalisation (selecting words and phrases), referring expression generation (identifying domain entities), and linguistic realisation (combining words into well-formed sentences)—while documenting the field's steady shift from hand-crafted, rule-based approaches toward data-driven, stochastic methods that blur these task boundaries, including planning-based architectures that treat generation as goal-directed action, global optimisation frameworks that jointly solve multiple sub-tasks, and neural encoder-decoder models that learn direct mappings from input data to output text. In highlighting relatively recent research topics arising from synergies with computer vision (image captioning from dense CNN feature vectors), stylistics (generating text conditioned on personality traits or affective states), and computational creativity (generating puns, metaphors, and narratives), the survey establishes that the effectiveness of any given NLG approach is heavily dependent on the application domain and the nature of the input data, with data-driven methods showing particular promise but also raising open questions about scalability, the trade-off between output quality and efficiency, and the need for multiple, complementary evaluation methods given that intrinsic corpus-based metrics and human judgements frequently do not correlate—a finding that implies no single evaluation paradigm adequately captures all dimensions of generated text quality.

2. Context and Motivation

The Core Problem: NLG Has Changed Radically, But No One Has Mapped the New Territory

The fundamental problem this paper addresses is straightforward to state but vast in scope: the field of Natural Language Generation has transformed over the past two decades, and no comprehensive survey exists to document what happened. The last complete survey—Reiter and Dale (2000)—was published when rule-based, modular pipeline architectures dominated the field. Since then, the landscape has shifted dramatically along multiple axes simultaneously: the methods used (from hand-crafted rules to statistical learning and neural networks), the architectures employed (from strict pipelines to integrated, end-to-end systems), the application domains tackled (from weather reports to image captions and creative narratives), and the evaluation paradigms adopted (from ad-hoc human judgements to shared tasks and automatic metrics). Gatt and Krahmer identify this gap explicitly in Section 1.2, noting that Reiter and Dale (2000) contains no discussion of statistical methods, no coverage of vision-to-text generation, no treatment of stylistic variation or creativity, and "notably absent... the lack of discussion of evaluation methodology." The field has outgrown its canonical reference, and the absence of an updated synthesis means that researchers—both newcomers and established practitioners—lack a shared map of the terrain.

This gap matters for several reasons the authors establish throughout their introductory framing:

The boundaries of the field are blurring. What counts as NLG is no longer self-evident. Text summarisation was characterized as text-to-text generation in this survey's introduction (Section 1), yet many abstractive summarisation systems use techniques also found in data-to-text generation. Conversely, data-to-text systems can incorporate text-to-text methods for lexicalisation or stylistic variation. Image captioning—now a major research area at the intersection of computer vision and NLP—is, as the authors argue in Section 4, "arguably a paradigm case of data-to-text generation, where the input comes in the form of an image." Without an updated survey, it is unclear whether these cross-pollinating sub-communities are converging on shared solutions or merely borrowing techniques opportunistically.
Architectural assumptions are under pressure. The modular pipeline architecture that Reiter (1994) identified as a "consensus" has been repeatedly challenged: by blackboard architectures that allow interactive feedback between tasks, by revision-based architectures that permit downstream modules to correct upstream decisions, by planning-based approaches that treat all generation choices as part of a unified goal-directed process, and by statistical models that learn direct mappings from input data to output text without explicit task decomposition. The authors note in Section 3.1 that this consensus was "contested repeatedly," but the field has never systematically assessed which of these challenges represent genuine architectural advances and which are domain-specific workarounds.
The evaluation crisis in NLG mirrors broader NLP but with unique complications. As the authors detail in Section 7, NLG systems produce open-ended textual outputs for which there is rarely a single correct answer. This makes evaluation fundamentally harder than in tasks with well-defined gold standards (e.g., named entity recognition, where annotated spans provide ground truth). The introduction of automatic metrics borrowed from machine translation (BLEU, METEOR) and summarisation (ROUGE) has enabled rapid, cheap evaluation, but a recurring finding—documented across multiple studies that the survey synthesises—is that these metrics frequently do not correlate with human judgements. The problem is not merely academic: without reliable evaluation, progress is unmeasurable, and the field cannot distinguish genuine improvements from artifacts of metric choice.

Why This Matters: Real-World Deployment and Theoretical Understanding

The practical significance of this gap is underscored by the range of deployed NLG applications the authors enumerate in Section 1. The technology is no longer confined to research prototypes. It generates:

Journalistic content: The Los Angeles Times' Quakebot system published an earthquake report within three minutes of the event, converting seismographic sensor data into prose through template filling (Section 1). Companies like Narrative Science and Automated Insights produce sports reports and financial summaries at commercial scale.
Clinical decision support: Systems such as BT-Nurse (Hunter et al., 2012) generate nursing shift summaries from neonatal intensive care data, a domain where information overload from continuous physiological monitoring can impair clinical decision-making.
Environmental communication: Generated texts describe wildlife tracking data (Tag2Blog, Ponnamperuma et al., 2013) and provide personalised environmental information (Wanner et al., 2015), aiming to increase public engagement with ecological issues.
Persuasive and behavioural interventions: Systems generate tailored smoking cessation letters (Reiter et al., 2003) and arguments for housing options (Carenini & Moore, 2006), where the quality of generated language directly impacts health outcomes or financial decisions.

In each of these domains, the choice of NLG architecture and methods has direct consequences for output quality, and by extension, for whether users trust, understand, and act on generated information. The survey's comprehensive mapping of available techniques therefore serves a practical function: it helps practitioners make informed methodological choices rather than defaulting to whatever approach happens to be fashionable or familiar.

Equally important is the theoretical significance. The authors position NLG not merely as an engineering discipline but as a testbed for theories of language production. The survey repeatedly draws connections between computational approaches and psycholinguistic models: Levelt's (1989) blueprint of the speaker, which distinguishes between conceptualisation, formulation, and articulation, maps naturally onto the NLG pipeline (Section 3.1); Gricean maxims underpin algorithms for referring expression generation (Section 2.5); theories of politeness (Brown & Levinson, 1987) and personality (the Big Five model, John & Srivastava, 1999) inform stylistic variation systems (Section 5). The authors argue in Section 8.3 that NLG implementations can "shed light on a number of underspecified parameters in the original model," citing Poesio et al.'s (2004) computational implementation of Centering Theory as a precedent. An updated survey that documents which theories have proven computationally tractable—and which have resisted formalisation—provides valuable feedback to theoretical linguistics and cognitive science.

What Prior Approaches Existed, and Where They Fall Short

The survey identifies four categories of prior work that fail to adequately cover the contemporary NLG landscape:

Reiter and Dale (2000) remains the canonical reference, but is obsolete. The authors are explicit about this: "the field of NLG has changed drastically in the last 15 years, with the emergence of successful applications generating tailored reports for specific audiences, and with the emergence of text-to-text as well as vision-to-text generation applications, which also tend to rely more on statistical methods than traditional data-to-text. None of these are covered by Reiter and Dale (2000)." The obsolescence is not merely a matter of missing citations—it is structural. Reiter and Dale's framework assumes a rule-based, modular pipeline with crisp separation between strategic (content selection, text structuring) and tactical (lexicalisation, realisation) decisions. This framework cannot accommodate the joint optimisation approaches that have become central to data-driven NLG, where decisions about what to say and how to say it are entangled in a single probabilistic model.

Subsequent overviews are either too narrow or too dated. The authors acknowledge several post-2000 overview texts (Section 1.2): Bateman and Zock (2005) cover cognitive, social, and computational dimensions of NLG; McDonald (2010) offers a general characterisation; Wanner (2010) focuses on report generation; Di Eugenio and Green (2010) examine education and healthcare applications; Krahmer and Theune (2010) collect articles on data-driven approaches; Bangalore and Stent (2014) focus on interactive systems. The authors also note several unpublished technical reports and specialised surveys—Theune (2003) on dialogue systems, Piwek (2003) and Belz (2003) on affective NLG, Gkatzia (2016) on content selection. While useful, these resources are acknowledged as being either "covered in other surveys" (for text-to-text applications like summarisation and paraphrasing) or not offering "a comprehensive review." The field has accumulated a patchwork of partial maps, but no single work has attempted to synthesise them into a coherent picture of the post-2000 landscape.

The treatment of evaluation in prior work is inadequate. This is perhaps the most consequential gap. Reiter and Dale (2000) simply did not discuss evaluation methodology in any depth. Yet, as the authors document across Section 7, the period since 2000 has seen the establishment of multiple NLG shared tasks (TUNA for referring expressions, GIVE for instruction generation in virtual environments, the Surface Realisation Shared Task, the KBGen Challenge for content selection), the adoption and adaptation of automatic metrics from machine translation and summarisation, the development of task-based extrinsic evaluation paradigms, and a growing body of meta-evaluation research examining the relationship between these different methods. Any survey that fails to synthesise this methodological work leaves practitioners without guidance on the most basic question: how do you determine whether your system is any good?

The relationship between data-to-text and text-to-text generation is underexplored. The authors take a deliberate position that distinguishes their survey: they define NLG in terms of its non-linguistic input (Section 1.1), focusing primarily on data-to-text generation while acknowledging the blurred boundaries with text-to-text tasks. However, they also argue that treating these as entirely separate enterprises is misleading. Data-to-text systems can incorporate text-to-text techniques (e.g., Kondadadi et al., 2013, using statistical methods learned from text-to-text to inform sentence planning), and text-to-text systems face many of the same architectural and evaluation challenges. The authors note that "doing full justice to recent developments in the various text-to-text generation applications is beyond the scope of a single survey," but they "will in various places discuss connections between data-to-text and text-to-text generation, both because—as noted above—the boundaries are blurred, but also, and perhaps more importantly, because text-to-text systems have long been couched in the data-driven frameworks that are becoming increasingly popular in data-to-text generation." Prior surveys have not systematically explored this convergence.

How This Paper Positions Itself Relative to Existing Work

The survey positions itself not as a replacement for Reiter and Dale (2000) but as a necessary update that covers the territory that has emerged since that work was published, while maintaining continuity with the traditional task decomposition that remains conceptually useful even as architectures diverge from it. The authors are explicit about three goals stated in Section 1.3:

First, to provide an up-to-date synthesis of core tasks and architectures, with emphasis on data-driven techniques. This goal produces Sections 2 and 3, which together constitute what the authors call the "foundational" part of the survey. Section 2 revisits the six canonical NLG tasks from Reiter and Dale (1997, 2000), but for each task, the survey documents how the approach has evolved: from hand-crafted domain-specific rules to data-driven, domain-independent methods. For example, content determination (Section 2.1) moved from schema-based approaches (McKeown, 1985) to Hidden Markov Models for topic shift detection (Barzilay & Lee, 2004), weakly supervised alignment techniques (Liang et al., 2009), and multi-resolution grounding models (Koncel-Kedziorski et al., 2014). Section 3 then examines how these tasks are organised into architectures, surveying modular, planning-based, and stochastic approaches, with the latter receiving the most extensive treatment given its relative novelty.

Second, to highlight relatively recent research topics arising from synergies with other areas of AI. This goal produces Sections 4, 5, and 6, which the authors treat as exemplars of NLG's expanding boundaries rather than exhaustive treatments of every sub-area. The selection is deliberate: image captioning (Section 4) represents the vision-language interface, where NLG must ground language in perceptual data; stylistic variation (Section 5) represents the interface with affective computing and personality psychology, where generation must be sensitive to speaker identity and emotional state; and computational creativity (Section 6) represents the interface with creative AI, where generation must produce artefacts (jokes, metaphors, narratives) that are not merely correct but interesting, surprising, or aesthetically pleasing. The authors acknowledge that these sections are shorter and more focused than the foundational material, but argue they are essential for understanding where NLG is heading.

Third, to draw attention to evaluation challenges, with emphasis on the relationship between different methods. This goal produces Section 7, which is arguably the most original contribution of the survey. Rather than simply cataloguing evaluation metrics and shared tasks—which other surveys have done for specific sub-areas—the authors organise their discussion around a fundamental methodological tension: intrinsic versus extrinsic evaluation, where intrinsic methods assess text quality in isolation (fluency, adequacy, genre compatibility) and extrinsic methods measure whether the text achieves its communicative purpose in a real task. The authors systematically document the finding, replicated across multiple domains (weather forecasting, referring expression generation, image captioning, text simplification), that these different evaluation paradigms frequently do not agree. They then draw practical recommendations from this meta-evidence, arguing that "the choice of method has a direct impact on the way in which results can be interpreted" and that "NLG practitioners, where evaluation is concerned, [should] err in favour of diversity, by using multiple methods, as far as possible, and reporting not only their results, but also the correlation between them" (Section 7.5).

The survey closes with a forward-looking discussion (Section 8) that identifies open questions and emerging directions, many of which have proven prescient: the role of social media data as an input source (anticipating the explosion of work on dialogue systems and chatbots), the importance of situated language generation (anticipating work on embodied agents and human-robot interaction), the potential of generation from structured knowledge bases (anticipating the semantic web and knowledge graph communities' growing interest in NLG), and the need for scalable data-driven techniques that can be deployed in commercial applications—a challenge that remains central to the field nearly a decade later.

3. Technical Approach

3.1 Reader Orientation

This paper is a comprehensive literature survey, not a technical contribution proposing a new model or system. The "system" being built is the survey itself—a structured framework for understanding how the field of Natural Language Generation has evolved from rule-based modular pipelines to data-driven integrated architectures. The core problem it solves is the fragmentation of knowledge: after two decades of rapid progress following Reiter and Dale (2000), no single document had synthesised the new methods, applications, and evaluation paradigms into a coherent picture. The shape of the solution is a three-part exposition that first revisits the traditional task decomposition (content determination through realisation), then examines how these tasks are organised into architectures (modular, planning-based, stochastic), and finally maps the expanding boundaries of the field into new application areas (vision-language, stylistic variation, computational creativity) and evaluation methodologies.

3.2 Big-Picture Architecture (Diagram in Words)

The survey organises the NLG landscape through a layered taxonomic structure with three orthogonal dimensions:

Task Decomposition (Section 2): The six canonical sub-tasks that most NLG systems perform, regardless of architecture—content determination (what to say), text structuring (in what order), sentence aggregation (how to group information into sentences), lexicalisation (which words to use), referring expression generation (how to identify entities), and linguistic realisation (how to form grammatical sentences). These tasks range from "early" (data-oriented, strategic) to "late" (language-oriented, tactical).
Architectural Organisation (Section 3): How the six tasks are combined into a working system. Three broad families are identified:
- Modular architectures (pipeline, blackboard, revision-based) that maintain task boundaries but vary in how information flows between modules.
- Planning-based architectures that treat generation as goal-directed action, using formalisms from AI planning (STRIPS, PDDL, Reinforcement Learning) to unify strategic and tactical decisions.
- Stochastic/integrated architectures that learn direct mappings from input data to output text, subsuming multiple tasks into a single probabilistic model (log-linear models, synchronous grammars, encoder-decoder neural networks).
Application Extensions (Sections 4–6) and Evaluation (Section 7): New domains where NLG interfaces with other AI subfields (computer vision, affective computing, computational creativity), and the methodological frameworks for measuring system quality (intrinsic vs. extrinsic, human judgements vs. automatic metrics, black-box vs. glass-box).

Information flows through the survey as follows: the task decomposition establishes the conceptual vocabulary → the architectural discussion shows how this vocabulary is operationalised in different system designs → the application sections demonstrate how the architecture-task relationship varies across input modalities and communicative goals → the evaluation section provides the methodological toolkit for assessing any configuration of the above.

3.3 Roadmap for the Deep Dive

First, the traditional task decomposition (Section 2 of the paper): Understanding the six canonical NLG tasks is prerequisite to understanding any architecture, because every system—whether modular pipeline or end-to-end neural model—must solve these sub-problems at some level. This section walks through content determination, text structuring, sentence aggregation, lexicalisation, referring expression generation, and linguistic realisation, tracing each from hand-crafted rules to data-driven methods.
Second, modular architectures and their alternatives (Section 3.1): Starting with the classical three-stage pipeline (Text Planner → Sentence Planner → Realiser), this section examines why this architecture became a "consensus" and what problems motivated alternatives: the generation gap (upstream decisions having unforeseen downstream consequences), blackboard architectures (reactive, shared-data structures), and revision-based architectures (limited feedback loops).
Third, planning-based approaches (Section 3.2): These treat NLG as goal-directed action, cutting across task boundaries. Two sub-families are examined: grammar-as-planning (using LTAG elementary trees as STRIPS-like operators with preconditions and effects) and stochastic planning under uncertainty using Reinforcement Learning (modelling generation as a Markov decision process with reward functions learned from interaction).
Fourth, stochastic and integrated approaches (Section 3.3): The most extensive treatment, covering data acquisition (alignment techniques, weak supervision), NLG as sequential stochastic process (n-gram models, Maximum Entropy), NLG as classification and optimisation (classifier cascades, Integer Linear Programming for joint optimisation), NLG as inverted parsing (probabilistic CFG rules, CYK decoding), and deep learning methods (encoder-decoder architectures, conditioned language models, attention mechanisms).
Fifth, application-specific architectures (Sections 4–6): These sections examine how the general architectural principles are adapted to specific input modalities: dense CNN feature vectors for image captioning, personality trait vectors for stylistic variation, and knowledge bases of semantic conventions for creative language (jokes, metaphors, narratives).
Sixth, evaluation architectures (Section 7): The methodological frameworks for measuring NLG quality—intrinsic vs. extrinsic, human judgements vs. automatic metrics, black-box vs. glass-box—and the meta-evidence that these methods frequently disagree, with practical recommendations for triangulation.

3.4 Detailed, Sentence-Based Technical Breakdown

This paper is a survey, so its "technical approach" is a taxonomic and synthetic methodology rather than an algorithmic contribution. The core idea is that the bewildering variety of NLG systems can be understood by cross-classifying them along three axes: which sub-tasks they perform (the traditional decomposition from Reiter & Dale, 2000), how those tasks are organised (the architectural dimension from modular to integrated), and what input-output relationship they model (the methodological dimension from rule-based to data-driven). The survey's intellectual contribution is to show that these axes are orthogonal—a system can be modular yet data-driven, or integrated yet symbolic—and that the field's evolution has been driven not by wholesale replacement of one paradigm by another, but by the gradual infusion of statistical techniques into existing architectures, followed by architectural redesign when the old boundaries no longer fit the new methods.

The Traditional Task Decomposition as a Conceptual Scaffold (Section 2)

The survey begins by re-establishing the six canonical NLG tasks from Reiter and Dale (1997, 2000), not because all modern systems implement them as separate modules—many do not—but because they provide a conceptual vocabulary for describing what any NLG system must accomplish, regardless of architecture. These six tasks are:

Content Determination (Section 2.1): Deciding which information from the input data should be included in the output text, and which should be omitted or abstracted. The input is typically a structured or semi-structured data source (database records, sensor readings, logical forms) containing more information than a reader needs or wants. The output is a set of preverbal messages—semantic representations of the selected information, often expressed in a formal representation language such as attribute-value matrices, logical forms, or graph structures.

The survey traces the evolution of content determination approaches from early, hand-crafted domain-dependent rules (McKeown's schemata, 1985) through to data-driven techniques. The key transition point is the work of Barzilay and Lee (2004), who used Hidden Markov Models to model topic shifts in a particular domain of discourse (e.g., earthquake reports), where the hidden states represented "topics" modelled as sentences clustered by similarity. This approach required only a corpus of texts, not manually annotated training data, and could generalise across domains sharing similar discourse structures.

Subsequent work addressed the alignment problem: given a database of records and corresponding free text, how to automatically identify which portions of text correspond to which database entries. Liang et al. (2009) formalised this as a generative model defining a distribution $p(w \mid s)$ for sequences of words $w$ and input states $s$ , with latent variables specifying the correspondence between $w$ and $s$ in terms of three components: (i) the likelihood of database records being selected given $s$ , (ii) the likelihood of certain fields being chosen for some record, and (iii) the likelihood that a string of a certain length is generated given the records, fields, and states. The parameters are estimated using the Expectation Maximisation (EM) algorithm, which iteratively refines the alignment by treating the latent correspondences as missing data.

The survey highlights a crucial property of this approach: the alignment is learned with weak supervision. No human-annotated alignments are required—only the raw text and corresponding database records. This makes the approach scalable to new domains, but also introduces a limitation: the alignment quality depends on the strength of the statistical signal, which may be weak when text and data are loosely coupled (e.g., a single sentence summarising multiple database entries).

Text Structuring (Section 2.2): Determining the order in which selected messages will be presented to the reader. The input is the set of preverbal messages from content determination; the output is a discourse plan or text plan—a structured, ordered representation of those messages, where the structure reflects discourse relations such as temporal sequence, contrast, elaboration, or causality.

The survey documents the shift from hand-crafted domain-dependent ordering rules (McKeown's schemata, 1985) and Rhetorical Structure Theory (Mann & Thompson, 1988) to general, corpus-based approaches for information ordering. The key contribution is the work of Lapata (2006), who frames text structuring as a ranking problem: given a set of information-bearing items (messages or sentences), learn a model that can order them as a human writer would. The approach uses a probabilistic model of item transitions, where the probability of item $b$ following item $a$ is estimated from corpus co-occurrence statistics, optionally enriched with features capturing discourse relations, temporal information, and entity-based coherence (e.g., maintaining a consistent topic focus across adjacent sentences).

The survey notes an important architectural implication: information ordering approaches can be applied both to text structuring in data-to-text systems (ordering preverbal messages) and to sentence ordering in multi-document summarisation (ordering extracted sentences). This cross-task transferability is a hallmark of the data-driven turn—methods become more general as they rely more on learned statistical patterns and less on domain-specific rules.

Sentence Aggregation (Section 2.3): Deciding which messages to express in individual sentences—that is, which messages to combine into a single sentence and which to keep separate. The input is the ordered text plan; the output is a set of sentence plans, each specifying a group of messages to be realised as a single sentence.

The survey distinguishes between two types of aggregation, following Reape and Mellish (1999): semantic aggregation (combining messages at the level of their meaning representations, e.g., merging three separate bradycardia events into a single existential predication) and syntactic aggregation (combining linguistic structures after realisation, e.g., conjoining clauses and eliding repeated constituents). The former is more powerful but more domain-dependent; the latter is more general but achieves only modest reductions in redundancy.

The key data-driven contribution is the work of Barzilay and Lapata (2006), who frame aggregation as a set partitioning problem solvable via Integer Linear Programming (ILP). The approach proceeds in two stages:

A binary classifier is trained to predict, for any pair of database entries $\langle e_i, e_j \rangle$ , whether they should be aggregated in the same sentence, based on pairwise similarity features (e.g., do the entries share the same event type? the same arguments?).
An ILP solver finds a globally optimal set of pairwise links satisfying transitivity constraints (if $\langle e_i, e_j\rangle$ and $\langle e_j, e_k\rangle$ are linked, then $\langle e_i, e_k\rangle$ should also be linked) and global constraints (e.g., the total number of sentences in the document, the maximum number of entries per sentence).

The advantage of the ILP formulation is that it avoids greedy, locally optimal decisions that can produce globally suboptimal aggregations—a direct parallel to the error propagation problem in pipeline architectures.

Lexicalisation (Section 2.4): Choosing the words and phrases to express the content of each sentence plan. The input is a sentence plan specifying a set of messages and their relations; the output is a lexicalised sentence plan where domain concepts have been mapped to lexical items (nouns, verbs, adjectives, prepositions) and open-class word forms have been selected.

The survey identifies two main sources of complexity in lexicalisation: semantic similarity (choosing among near-synonyms or taxonomically related words, e.g., animal vs. dog) and vagueness (selecting gradable adjectives like tall or temporal expressions like midnight, which require reasoning about context-dependent standards of comparison). The latter is particularly well-illustrated by Reiter et al. (2005), who found that human weather forecasters show considerable variation in how they express timestamps like 00:00 (late evening, midnight, or simply evening), and that automatically generated texts with more consistent lexicalisation were preferred by readers because they avoided seemingly arbitrary variation.

The survey notes a trend toward merging lexicalisation with syntactic realisation, tracing back to Elhadad et al. (1997), who described a unification-based approach where conceptual representations (preverbal messages) are unified with grammar rules encoding both lexical and syntactic choices. This trend has accelerated with data-driven approaches: in many neural models, there is no separate lexicalisation component—the mapping from concepts to words is learned implicitly as part of the sequence-to-sequence generation process.

Referring Expression Generation (REG) (Section 2.5): Selecting words or phrases to identify domain entities for the reader. The survey identifies this as "among the tasks within the field of automated text generation that has received most attention in recent years" because it "can be separated relatively easily from a specific application domain and studied in its own right."

The task decomposes into two sub-decisions: (a) referential form: whether to use a pronoun, a proper name, an indefinite description, or a definite description, depending on the entity's salience or "focus" status in the discourse; and (b) referential content: for descriptions, which properties to include to distinguish the target referent from other entities in the context.

The survey presents the classical algorithmic approaches to content determination—the Full Brevity algorithm (exhaustive search for the smallest distinguishing set of properties), the Greedy Heuristic (incrementally select the property ruling out the most remaining distractors), and the Incremental Algorithm (Dale & Reiter, 1995) (incrementally select properties based on a domain-specific preference order, stopping as soon as the target is distinguished). These algorithms are all implementations of the Gricean maxim of Quantity: be as informative as required, but not more so.

The key conceptual evolution has been toward probabilistic models that can capture the variability observed in human-produced referring expressions. For example, FitzGerald et al. (2013) frame REG as estimating a log-linear distribution over a space of logical forms representing expressions for sets of objects, where the probability of a description depends on features like the number of properties, their cognitive salience, and the discourse context. This allows the model to generate descriptions that are sometimes "overinformative" (including redundant properties), matching human behaviour while still being generally minimal.

Linguistic Realisation (Section 2.6): Combining all words and phrases into well-formed sentences, including ordering constituents, generating morphological forms (verb conjugations, agreement), inserting function words and punctuation. The input is a lexicalised sentence plan; the output is a surface string.

The survey distinguishes three broad approaches:

Templates (Section 2.6.1): Fixed strings with variables that are filled using application-specific rules. The example given is $player scored for $team in the $minute minute, which can generate Ivan Rakitic scored for Barcelona in the 4th minute. The advantage is full control and guaranteed grammaticality; the disadvantage is limited variation and poor scalability.
Hand-crafted grammar-based systems (Section 2.6.2): Domain-independent realisers like FUF/SURGE, KPML, NIGEL, and RealPro that use manually written grammars to generate sentences from detailed input specifications. The survey notes that these require "very detailed input," often including information that earlier pipeline stages do not naturally produce—an instance of the generation gap.
Statistical approaches (Section 2.6.3): These subdivide further into (a) overgenerate-and-rank methods (Langkilde & Knight's Nitrogen/Halogen systems) where a hand-crafted grammar generates many alternative realisations represented as a forest, and a statistical model (typically an n-gram language model or log-linear reranker) selects the best candidate; (b) direct statistical generation where probabilities are used at choice points during generation, not just for post-hoc filtering (Belz's PCRU system); and (c) fully data-driven grammar acquisition from treebanks, where both the grammar rules and the statistical model are learned from annotated corpora (e.g., OpenCCG, which learns Combinatory Categorial Grammar derivations from the Penn Treebank).

The survey emphasises that the existence of wide-coverage stochastic realisers has "motivated a greater focus on subtle choices, such as how to avoid structural ambiguity, or how to handle choices such as explicit complementiser insertion in English." This is a concrete example of how improving one component (realisation) raises the bar for other components (sentence planning, lexicalisation) by making their decisions more visible in the final output.

Modular Architectures and the Pipeline Consensus (Section 3.1)

The survey establishes the classical three-stage pipeline architecture (Reiter, 1994) as a "de facto standard" that was "a generalisation based on actual practice" rather than a theoretically motivated design:

Text Planner (Document Planner, Macroplanner): Combines content determination and text structuring. Input is the raw or pre-processed data; output is a text plan—a structured, ordered representation of preverbal messages. This module handles strategic generation: deciding what to say.
Sentence Planner (Microplanner): Combines sentence aggregation, lexicalisation, and referring expression generation. Input is the text plan; output is a sentence plan—a lexicalised, aggregated representation where referring expressions have been chosen, but surface syntax and morphology are not yet determined. This module handles the transition from strategic to tactical generation: deciding how to say it.
Linguistic Realiser: Converts sentence plans into surface strings by applying syntactic and morphological rules. Input is the sentence plan; output is the final text.

The pipeline shares characteristics with two related architectures that the survey uses to situate it within a broader intellectual context:

Text summarisation pipelines (Mani, 2001): These decompose into (a) analysis of source texts and selection of information, (b) transformation of selected information to enhance fluency, and (c) synthesis of the summary. The parallel is structural: both decompose the generation problem into content selection, transformation, and realisation stages.
Levelt's (1989) model of human speech production: This psycholinguistic model makes a similar distinction between conceptualisation (deciding what to say), formulation (determining how to say it, including grammatical encoding and phonological encoding), and articulation (producing the speech signal). Levelt's model allows for "a limited degree of self-monitoring through feedback loops, a feature that is absent in Reiter's NLG pipeline."

The survey identifies two fundamental problems with the pipeline architecture that motivate alternatives:

The Generation Gap (Meteer, 1991): Mismatches between strategic and tactical components, where early decisions in the pipeline have unforeseen negative consequences further downstream. The example given from Inui et al. (1992) is illustrative: a generation system might determine a particular sentence ordering during sentence planning, but this ordering might turn out to be ambiguous once sentences have actually been realised and orthography (punctuation, paragraph breaks) has been inserted. The problem is structural—earlier modules cannot anticipate how their output will be transformed by later modules, because they operate on different representations (preverbal messages vs. surface strings).

Generating Under Constraints (Reiter, 2000): When the output must satisfy requirements that are only definable in surface terms, such as a maximum length in words or characters. These constraints are "much harder at the earlier stages, where the representations are pre-linguistic and their mapping to the final text are potentially unpredictable."

The survey documents three alternative architectures that partially address these problems, all predating the data-driven turn and all symbolic in their methodological orientation:

Blackboard Architectures: Task-specific procedures are not rigidly pre-organised but perform their tasks reactively as the output, represented in a shared data structure (the blackboard), evolves. The example given is Nirenburg et al. (1989). The advantage is maximum flexibility—any module can contribute at any time based on the current state of the blackboard. The disadvantage is reduced efficiency, since modules must constantly monitor the blackboard for relevant changes.

Revision-Based Architectures: A limited form of feedback between modules under monitoring, with the possibility of altering choices that prove to be unsatisfactory. The survey cites Inui et al. (1992) and Mann and Moore (1981). The key insight is that this "has the advantage of not requiring 'early' modules to be aware of the consequences of their choices for subsequent modules, since something that goes wrong can always be revised." Robin (1993) provides a concrete example: an initial draft of a sports summary was generated, then revised to add historical background information that was made relevant by the events reported in the draft, with decisions about where to place the background information relative to the main text.

The survey also notes Reiter's (2007) extension of the pipeline "backwards" to accommodate systems whose input is raw, unstructured numerical data (such as physiological signals in neonatal intensive care, or numerical weather predictions). These systems require two additional stages before the traditional Text Planner:

Data abstraction: Identifying broad trends, patterns, or events in the raw data (e.g., "three successive bradycardia events occurred between 17:01 and 17:06" from a continuous heart rate signal). Techniques range from signal processing (Portet et al., 2009) to fuzzy set theory (Ramos-Soto et al., 2015).
Data interpretation: Assigning meaning to the abstracted patterns (e.g., "this cluster of events is clinically significant because..."). This stage bridges the gap between domain-agnostic pattern detection and domain-specific knowledge.

The survey then introduces Mellish et al.'s (2006) "objects-and-arrows" framework as a formalism for describing architectures without imposing a specific pipeline structure. Rather than prescribing a particular task decomposition, this framework allows the analyst to specify "different types of information flow between NLG sub-tasks," accommodating systems where, for example, referring expression generation is split between sentence planning (which determines content) and realisation (which determines surface form). The framework "retains the principle that the tasks, irrespective of their organisation, are relatively well-defined and distinguished"—it is a language for describing variability, not an argument for abolishing task boundaries entirely.

Planning-Based Approaches (Section 3.2)

The survey presents planning-based NLG as a fundamentally different way of organising the generation process, one that cuts across the task boundaries of the pipeline architecture. The unifying idea is that text generation is goal-directed action: the system has a communicative goal (e.g., inform the reader about an earthquake, persuade a user to choose a restaurant), and generation proceeds by selecting actions (speech acts, linguistic choices) whose predicted effects move the discourse context closer to that goal.

This perspective connects NLG to a long tradition in AI planning, originating with STRIPS (Fikes & Nilsson, 1971), where actions are represented as tuples of preconditions (what must be true for the action to be applicable) and effects (what becomes true after the action is executed). In the NLG context, an action might be "generate a referring expression for entity X," with preconditions that X is salient in the discourse and effects that the reader can now identify X. The survey traces this idea back to Appelt's (1985) KAMP system, which generated referring expressions by reasoning about interlocutors' beliefs and mutual knowledge, producing referential NPs whose predicted effect was to alter the hearer's belief state about the referent.

The survey identifies two major sub-paradigms within planning-based NLG:

Planning through the Grammar (Section 3.2.1): Rather than using general-purpose AI planning formalisms with separate domain theories, this approach encodes planning operators directly in the grammar. The primary formalism is Lexicalised Tree Adjoining Grammar (LTAG; Joshi & Schabes, 1997), where elementary trees (lexicalised fragments of syntactic structure) are augmented with:

Semantic preconditions: What must be true in the knowledge base for the lexical item to be felicitously used (e.g., for the verb likes, the agent must be animate, and the proposition that X likes Y must be supported by the knowledge base).
Pragmatic effects: What goals the use of that lexical item will achieve (e.g., inserting likes into the derivation sets up NP nodes for the agent and patient that must subsequently be filled).

The survey provides a worked example based on Garoufi (2014) for the sentence Mary likes the white rabbit. The lexical item likes is represented as a STRIPS-like operator with preconditions (the proposition is in the KB, the agent is animate, the current utterance can be substituted into the derivation) and effects (the utterance is now part of the sentence, new NP nodes for arguments have been created and need filling). Referring expression generation is then accomplished not by a separate module but by associating additional preconditions on the linguistic operators that fill these argument positions: the entity must be in the hearer's knowledge state, and each added word (e.g., rabbit, white) has the effect of excluding distractors from the set of possible referents.

The practical advantage is that this formulation is compatible with off-the-shelf AI planners, since the NLG task can be expressed in the Planning Domain Definition Language (PDDL; McDermott, 2000). The survey notes, however, that "planners remain beset by problems of efficiency." Koller and Petrick (2011) found in experiments on NLG tasks of differing complexity that "planners tend to spend significant amounts of time on preprocessing, though solutions could often be found efficiently once preprocessing was complete."

Stochastic Planning under Uncertainty using Reinforcement Learning (Section 3.2.2): This approach addresses a limitation of classical planning: the assumption that the effects of actions are deterministic and known in advance. In dialogue systems, for example, the user's response to a system utterance is not perfectly predictable—there is inherent uncertainty about whether a recommendation will be accepted, a question will be answered, or a misunderstanding will occur.

The survey describes how NLG in dialogue can be modelled as a Markov Decision Process (MDP) within a Reinforcement Learning (RL) framework. In this formulation:

States are representations of the discourse context (what has been said, what the user has done, what information the system has about the user's goals and knowledge).
Actions are generation decisions (which content to include, which words to use, which dialogue strategy to adopt).
State-action pairs are associated with a transition probability $P(s_{t+1} \mid s_t, a_t)$ —the probability of moving to a new discourse state given the current state and the system's action.
Rewards are provided by a reward function that quantifies the optimality of the generated output (e.g., negative reward for verbose utterances, positive reward for successful task completion).

The key advantage over supervised learning is that RL "enables adaptation in a changing context" (Rieser & Lemon, 2009), because the policy is learned through interaction (or simulation) rather than from a fixed dataset. The survey cites several applications:

Rieser et al. (2011) optimised information presentation when generating restaurant recommendations, learning when to provide additional details versus when to make a concise recommendation.
Janarthanam and Lemon (2014) optimised the choice of information in referring expressions, learning to adapt the user model as the user acquires new knowledge during a dialogue.
Cuayáhuitl and Dethlefs (2011) used hierarchical RL to jointly optimise route finding and route description, learning strategies that guide users past familiar landmarks while avoiding confusing junctions.

The survey highlights joint optimisation as a major theme in this work, where a single policy "satisfies multiple constraints arising from different sub-tasks of the generation process, by sharing knowledge across the sub-tasks." Dethlefs and Cuayáhuitl (2015) provide the strongest evidence: a hierarchical model with separate learning agents for content selection through realisation, where a joint framework in which agents share knowledge outperforms an isolated learning framework where each task is modelled separately. For example, "the joint policy learns to give high-level navigation instructions, but switches to low-level instructions if the user goes off-track," producing utterances that are "less verbose and lead to shorter interactions overall."

The survey notes that the joint optimisation framework is not unique to planning-based approaches—it also appears in the ILP-based work of Barzilay and Lapata (2005, 2006) and Marciniak and Strube (2005)—but argues that the RL formulation provides a more natural way to handle the dynamic, interactive nature of dialogue.

Stochastic Approaches to NLG (Section 3.3)

This is the most extensive subsection of the survey, reflecting the dominance of data-driven methods in contemporary NLG. The survey organises these approaches by their underlying statistical framework, tracing an evolution from models that maintain task boundaries to those that abolish them entirely.

Acquiring Data (Section 3.3.1): A prerequisite for statistical NLG is training data—pairs of inputs (data) and outputs (text) from which models can learn mappings. The survey documents the landscape of available resources:

Existing aligned corpora: These were initially limited to specific domains—weather forecasting (Reiter et al., 2005; Belz, 2008; Liang et al., 2009), sports summaries (Barzilay & Lapata, 2005; Chen & Mooney, 2008), restaurant recommendations (Mairesse et al., 2010). They "usually consist of database records paired with free text."
Automatic alignment techniques: The survey describes the influential framework of Liang et al. (2009) in detail. This is a generative model that defines a distribution $p(w \mid s)$ for word sequences $w$ and input states $s$ , with latent variables specifying the correspondence. Three components are modelled: (i) the likelihood that a database record is selected given the state, (ii) the likelihood that specific fields within that record are chosen for verbalisation, and (iii) the likelihood of generating a word string of a given length given these selections. Parameters are estimated via Expectation Maximisation (EM), which alternates between estimating the latent alignment (E-step) and updating the model parameters to maximise the likelihood of the observed text (M-step).
Multi-resolution alignment: The survey describes the work of Koncel-Kedziorski et al. (2014), which addresses the fact that "there is no one-to-one correspondence between, for example, soccer events in data and sentences in associated soccer reports." A sentence like Chamakh rises highest and aims a header towards goal which is narrowly wide corresponds to two separate database entries (an aerial pass and a miss), but can be further decomposed into sub-parts that individually map to these sub-events. The model "exploits linguistic structure to align at varying resolutions," using constituency or dependency parses to identify sentential sub-components and align them to data fragments.
Crowd-sourcing: The survey cites Mairesse and Young (2014), who "use crowd-sourcing techniques to elicit realisations for semantic/pragmatic inputs describing dialogue acts in the restaurant domain." The key to success is "the development of a semantics that is sufficiently transparent for use with non-specialists"—lay annotators must be able to understand the meaning representation well enough to produce natural paraphrases.
Active learning for data reduction: Mairesse et al. (2010) describe using uncertainty sampling (Lewis & Catlett, 1994) to reduce the amount of training data required. A model is trained on a small initial dataset, applied to new data, and the system "samples the cases of which it is least certain, forwarding these to a (possibly human) oracle for feedback, which potentially leads to a new training cycle."

The survey observes that neural network approaches (Section 3.3.5) have partially obviated the need for explicit alignment, because "looser input-output couplings can constitute adequate training data, especially in models that incorporate attentional mechanisms." The attention mechanism learns which parts of the input to attend to when generating each output word, effectively performing soft alignment as a by-product of the generation process.

NLG as a Sequential, Stochastic Process (Section 3.3.2): This family of approaches models generation as a sequence of decisions where each choice is conditioned on the history of previous choices and the input data. The foundational approach is the Markov model:

n-gram language models augmented with input features: Oh and Rudnicky (2002) constructed separate n-gram models for each utterance class (dialogue act type) and for word classes corresponding to input attributes (e.g., words for departure city). Content planning is handled by a separate model predicting which attributes should be included based on dialogue history. Realisation uses n-gram models to predict word sequences given the chosen attributes. This architecture "maintains a crisp separation between strategic and tactical components," using statistical methods within a modular framework.
Log-linear (Maximum Entropy) models: These address a limitation of n-gram models—the local history assumption, which limits the extent to which prior selections can influence current choices. Log-linear models can condition on "arbitrarily long histories of previous decisions" by defining features that capture long-range dependencies. The survey highlights Ratnaparkhi (2000) as foundational: two systems (NLG2 and NLG3) were compared, where NLG2 uses a conditional language model predicting words given both local history and remaining semantic attributes, and NLG3 augments this with syntactic dependency relations, recursively predicting the left and right children of a constituent.
End-to-end log-linear generation with task separation: Angeli et al. (2010) "maintain a separation between content selection, sentence planning and realisation, modelling each process as a sequence of decisions in a log-linear framework." The separation is maintained by using "a different set of features" for decisions at each stage—content selection features capture which database records are important, sentence planning features capture template choice likelihoods, and realisation features capture local word order. Templates are acquired from corpus data: "a template is selected based on its likelihood given the database fields selected during content selection."
Factored Language Models for joint generation: Mairesse and Young (2014) use a tree-structured representation (Figure 5 in the paper) where leaves correspond to word sequences and non-terminals are semantic stacks (the pieces of input to which the words correspond). A Factored Language Model (FLM) extends n-gram models by "conditioning probabilities on different utterance contexts, rather than simply on word histories." Generation works in three stages using Viterbi search: (a) identify mandatory semantic stacks for the dialogue act, (b) enrich with optional stacks (function words), and (c) find realisations (word sequences) for the stack sequence. This approach "seeks a global solution to generation, without a crisp distinction between strategic and tactical components."

NLG as Classification and Optimisation (Section 3.3.3): If each NLG decision can be framed as choosing among a discrete set of options, then the entire generation process can be modelled as a cascade of classifiers, where the output is constructed incrementally:

Classifier cascades: Marciniak and Strube (2004) divided route description generation into "a series of eight classification problems, from determining the linear precedence of discourse units, to determining the lexical form of verbs and the type of their arguments." Each classifier uses the instance-based KStar algorithm, trained on a small corpus of manually annotated texts. A key architectural insight is that the cascade can be viewed as "a weighted, multi-layered lattice, where generation amounts to a best-first traversal" (Marciniak & Strube, 2005)—each classifier produces the most likely output, which leads to the next classifier along the most probable path.
Error propagation as the central problem: The survey notes that "a sequential organisation of NLG tasks is susceptible to error propagation, whether this takes the form of classifier errors, or decisions in a rule-based module that have a negative impact on downstream components." Zarrieß and Kuhn (2013) provide a concrete demonstration: their system performs syntactic mapping and referring expression insertion using SVM-based ranking classifiers, and they find that "the performance of either task is order-dependent, in that both classification tasks perform worse when they are second in the sequence." A revision-based architecture (syntax → REG → syntax revision) achieves the best performance.
Integer Linear Programming for joint optimisation: The survey presents ILP as a solution to error propagation that "seek[s] a global labelling solution that minimises the overall cost." The framework is:
1. Each NLG task is modelled as binary label assignment (e.g., a label for "include this database record" or "use this template"), with a cost based on the label's probability in training data.
2. Pairs of tasks with strong interdependencies (e.g., syntactic choices and REG realisations) have a joint cost based on the co-occurrence probability of their labels.
3. An ILP solver finds the global assignment minimising total cost, with constraints enforcing consistency (if one of a pair of correlated labels is selected, the other must be too).
The survey cites Lampouras and Androutsopoulos (2013) who "show that joint optimization using Integer Linear Programming to jointly determine content selection, lexicalisation and aggregation produces more compact verbalisations of ontology facts, compared to a pipeline system."
Imitation learning as an alternative: Lampouras and Vlachos (2016) frame content determination and realisation as classification problems optimised jointly "in an iterative training algorithm using imitation learning," where the model learns to mimic an expert policy (derived from the training data) while being penalised for deviations that cascade into poor downstream decisions.

NLG as 'Parsing' (Section 3.3.4): A conceptually elegant framing treats generation as the inverse of syntactic or semantic parsing—given a meaning representation (analogous to the logical form output by a semantic parser), find the most probable surface string (analogous to the sentence input to a semantic parser) using a probabilistic grammar:

Probabilistic Context-Free Grammars (PCFGs) for generation: Belz (2008) "formalises the NLG problem entirely in terms of CFGs: a base generator expands inputs (bits of weather data in this case) by applying CFG rules; corpus-derived probabilities are then used to control the choice of which rules to expand at each stage of the process." The base generator is hand-crafted, but the probabilities are learned from data, so the system's behaviour adapts to the domain while maintaining grammatical guarantees.
Synchronous grammars from machine translation: The survey describes the work of Wong and Mooney (2007) and Chen and Mooney (2008), who adapt the WASP semantic parser—which uses probabilistic synchronous CFG rules learned from utterance-meaning pairs—for generation. The key contribution is WASPER-GEN, which "seeks the maximally probable sentence given an input MR, as it were, learning a translation model from meaning to text" (the reverse direction from parsing). On the RoboCup sportscasting dataset, WASPER-GEN outperforms WASP on automatic metrics and "is shown to achieve a level of fluency and semantic correctness which approaches that of human text, based on subjective judgements."
Unified hypergraph generation: Konstas and Lapata (2012, 2013) present the most fully integrated model in this family. Rules of the form:
```
R(windSpeed) → FS(temperature), R(rain)
FS(windSpeed,min) → F(windSpeed,max) FS(windSpeed,max)  
FS(windSpeed,min) → W(windSpeed,min)
```
incorporate different types of decisions: content ordering (rule 15: describe wind speed, then temperature, then rain), content selection (rule 16: minimum wind speed should be followed by maximum wind speed with some probability), and lexical realisation (rule 17: expand a database field to a word sequence). These rules are packed into a hypergraph, and generation proceeds via an adaptation of the CYK parsing algorithm to find the maximally likely word sequence. The decoder is interleaved with "two further sources of linguistic knowledge: (a) a weighted finite-state automaton (representing an n-gram language model); and (b) a dependency model," which address the fact that the CFG rules alone do not capture surface fluency.

Deep Learning Methods (Sections 3.3.5–3.3.7): The survey's treatment of neural approaches, while noting that the area was "comparatively small (but steadily growing)" at the time of writing, identifies the key architectural patterns that have since become dominant:

Recurrent Neural Networks (RNNs) with Long Short-Term Memory (LSTM): The survey explains that LSTMs (Hochreiter & Schmidhuber, 1997) are "the dominant type of RNN for language modelling tasks" because they "handle sequences of varying lengths, while avoiding both data sparseness and an explosion in the number of parameters through the projection of histories into a low-dimensional space, so that similar histories share representations." The foundational demonstration for NLG was Sutskever et al. (2011), who used a character-level LSTM to generate grammatical English sentences, though this focused exclusively on realisation without semantic conditioning.
Encoder-Decoder Architectures (Section 3.3.6): This framework "decouples" input processing from output generation: an RNN encoder maps the input (data records, meaning representations, images) into a fixed-length vector representation, which then serves as the auxiliary input to an RNN decoder that generates the output word by word. The survey highlights several architectural variants:
- Semantically conditioned LSTMs (Wen et al., 2015): The input dialogue act is encoded and used to condition the LSTM that generates the next dialogue act in the output sequence.
- Sequence-to-sequence (seq2seq) with attention (Dušek & Jurčíček, 2016): An attention mechanism "force[s] the encoder, during training, to weight parts of the input encoding more when predicting certain portions of the output during decoding," which "obviates the need for direct input-output alignment, since attention-based models are able to learn input-output correspondences based on loose couplings of input representations and output texts."
- Coarse-to-fine alignment with bidirectional LSTMs (Mei et al., 2016): A bidirectional LSTM encoder maps input records to hidden states. An attention-based aligner then models content selection by "determining which records to mention as a function of their prior probability and the likelihood of their alignment with words in the vocabulary." A refinement step "weights the outcomes of the alignment with the priors, making it more likely that more important records will be verbalised."
The survey draws an explicit connection between this architecture and earlier log-linear work: the LSTM "learn[s] long-range dependencies between records and descriptors, which the log-linear model of Angeli factored in explicitly," suggesting that deep learning automates feature engineering that earlier approaches required manual specification.
Conditioned Language Models (Section 3.3.7): A simpler but effective approach where "output is generated by sampling words or characters from a distribution conditioned on input features, which may include semantic, contextual or stylistic attributes." The survey example is Lebret et al. (2016), who use a feedforward neural network to generate the first sentence of Wikipedia biographies from infobox tables, "conditioning output word probabilities on both local context and global features obtained from the input table." The global features "bias the model towards full coverage of the contents of a field"—for example, ensuring that a multi-word person name is generated as a complete unit rather than truncated.

The survey closes this subsection with two forward-looking observations. First, Manning's (2015) argument that "the contribution of deep learning to NLP has to date been mainly due to the power of distributed representations, rather than the exploitation of the 'depth' of multi-layered models," but that "greater depth can confer representational advantages." Second, the speculation that as complex architectures "self-organise during training by minimising a loss function, it might turn out that different components of such architectures acquire core representations pertaining to different aspects of the problem at hand"—a form of emergent modularity that would parallel the traditional task decomposition, but learned rather than designed.

The Vision-Language Interface: Image Captioning (Section 4)

The survey treats image captioning as a case study in how NLG architectures adapt to a specific input modality—visual data—and how the architectural choices for NLG interact with choices from computer vision. The section is organised around the two logically distinguishable sub-tasks:

Image Analysis (Section 4.2.1): Three approaches to extracting features from images:

Detection: Using computer vision methods to detect and label objects, attributes, "stuff" (mass nouns like grass), spatial relations, and actions. Output is structured: detected entities with their properties and relations. This approach makes the mapping to linguistic structures relatively direct—objects become nouns, spatial relations become prepositions—but its performance "depends on the coverage and accuracy of detectors."
Holistic scene analysis: Characterising images using global features (RGB histograms, SIFT, GIST) that do not identify individual objects but capture the overall scene type. Often used by retrieval-based systems that match a query image to training images and return the associated captions.
Dense image feature vectors from CNNs: The dominant approach, using an activation layer from a pre-trained CNN (AlexNet, VGG, Caffe) as the input to the text generation component. This provides a dense, continuous representation that captures visual content without explicit object detection.

Text Generation or Retrieval (Section 4.2.2): Four approaches to converting visual features to text:

Templates or trees: Systems relying on detectors map their output to linguistic structures in a sentence planning stage. Kulkarni et al. (2013) experiment with both templates and n-gram language models, "finding that the former are more fluent, but suffer from lack of variation." Mitchell et al.'s (2012) Midge system represents images as triples ⟨noun, verb, preposition⟩ and realises them using a tree substitution grammar, enhanced with the ability to insert words not directly grounded in detections but having high corpus probability.
Language models (including neural): These model caption generation as word-by-word prediction, where each word's probability is conditioned on both the caption history and the image features. Most deep learning architectures follow this pattern, varying in how they combine image features with the language model: Donahue et al. (2015) inject CNN features into the LSTM at each time-step; Mao et al. (2015a) combine linguistic and visual features in a multimodal layer after the recurrent layer.
Caption retrieval and recombination: Rather than generating captions, systems retrieve them from training data based on image similarity, potentially recombining fragments from multiple retrieved captions. The advantage is guaranteed fluency; the disadvantage is that "captions in the training data may not be well-matched to a query image," resulting in generic or irrelevant outputs.

The survey draws particular attention to attention-based models (Xu et al., 2015) as a development that bridges the two sub-tasks: "the idea is to allocate variable weights to portions of captions in the training data, depending on the current context, to reflect the 'relevance' of a word given previous words and an image region." This allows the model to learn fine-grained correspondences between words and image regions without explicit object detection—a form of soft grounding.

Generating Text with Style, Personality, and Affect (Section 5)

This section examines architectures where the output text must vary not just in content but in manner of expression, conditioned on variables like author identity, personality traits, or emotional state. The survey identifies two main paradigms:

Rule-based stylistic control: Early work defined stylistic features using hand-crafted rules. DiMarco and Hirst (1993) proposed a "stylistic grammar for English and French" that explicitly encoded choices like passive vs. active voice, complementiser insertion, and lexical formality. The limitation is that "certain features, once selected, may 'cancel' or obscure the stylistic effect of other features" (Mairesse & Walker, 2011), making rule interactions difficult to manage.

Data-driven stylistic variation: The dominant modern approach, which learns correlations between stylistic dimensions and linguistic features from corpora:

Multidimensional style models: Paiva and Evans (2005) used Biber's (1988) framework, conducting factor analysis on linguistic features to identify stylistic dimensions. Their system generated a large number of candidate texts varying its decisions at choice points, scored each text on the stylistic dimensions, and trained a linear regression model to predict dimension scores from decisions. At test time, a best-first search identified the sequence of local decisions most likely to maximise the desired stylistic effect.
Personality-conditioned generation (PERSONAGE): Mairesse and Walker (2010, 2011) model personality using the "Big Five" trait model (extraversion, agreeableness, conscientiousness, neuroticism, openness). The input is a pragmatic goal (e.g., recommend restaurants) and real-valued scores on the five personality dimensions. The system "estimates generation parameters for stylistic features based on the input traits, using machine-learned models acquired from a dataset pairing sample utterances with human personality judgements." An extraverted utterance might be more verbose with expletives; an introverted one might show more hedging and uncertainty.

The survey notes a crucial finding from evaluation: "readers vary significantly in their judgements of what personality is actually reflected by a given text," suggesting that "the relationship between such psychological features and their linguistic effects is far from straightforward."

Neural approaches to stylistic control (Section 5.3): The survey documents the emerging trend of conditioning neural language models on stylistic attributes:

Persona-based response generation (Li et al., 2016): A seq2seq model where the decoder LSTM is conditioned on speaker embeddings learned from social media data. An alternative version conditions on both speaker and addressee profiles.
Personality-conditioned LSTMs (Herzig et al., 2017): Conditioning on personality trait embeddings rather than speaker-specific embeddings, enabling the generator to be tuned to specific personality settings without retraining for each speaker.
Attribute-controlled generation (Hu et al., 2017; Ficler & Goldberg, 2017): Using variational auto-encoders or directly conditioning on attribute vectors to control sentiment, tense, length, descriptiveness, and other stylistic parameters independently.

The survey identifies adaptivity as a key open question: "the way speakers (or systems) alter their linguistic choices as a result of their interlocutors' utterances"—a dynamic form of style that goes beyond static attribute conditioning.

Generating Creative and Entertaining Text (Section 6)

This section surveys approaches to generating language that is not merely correct but interesting, surprising, or aesthetically pleasing. The architectural patterns are notably different from standard NLG, often relying on explicit representations of semantic conventions and their creative violation:

Puns and Jokes (Section 6.1): The canonical example is Binsted and Ritchie's (1994, 1997) JAPE system, which generates punning riddles like What's the difference between money and a bottom? One you spare and bank, the other you bare and spank. The system relies on a template-based architecture combined with lexical resources: a pronunciation dictionary (to find words with similar sounds, e.g., bizarre and bazaar) and WordNet (to find semantically related words, e.g., bazaar and market). The creative mechanism is juxtaposition: placing related words next to each other and treating the combination as a normal construction, while checking that the combination is novel.

The survey highlights Petrović and Matthews (2013) as an "unsupervised alternative to this earlier work, which does not require labelled examples or hard-coded rules." Their system generates jokes of the form I like my X like I like my Y, Z (e.g., I like my coffee like I like my war, cold). The approach quantifies three principles for "good" triples ⟨X, Y, Z⟩: (a) the attribute Z should describe both nouns X and Y; (b) Z should be both common and ambiguous; (c) X and Y should be dissimilar. These are operationalised using WordNet and the Google n-gram corpus, and scored without any labelled joke data.

Metaphors and Similes (Section 6.2): Veale and Hao's (2007, 2008) Sardonicus system generates metaphors by searching the web for nouns associated with a target property (e.g., stick, pole, pencil for skinniness), filtering out inappropriate ones (e.g., cadaver), and ranking them by their occurrence in web queries of the form N-like T (e.g., stick insect-like Paris Hilton). The creative mechanism is category inclusion (Glucksberg, 2001): the source noun must be a prototypical member of the category denoted by the property, so that the metaphor is interpretable rather than merely arbitrary.

The survey also describes Veale's (2013) extension to affective metaphor generation, where stereotypes (e.g., babies are cute, cowboys are lonesome) are assigned positive or negative valence by mining coordination patterns in the Google n-gram corpus (happy and X, sad and X). This enables generating similes like cute as a baby (positive) or crying like a baby (negative), and even composing them into poems by combining multiple metaphors under a unifying trope.

Narratives (Section 6.3): The survey documents the fundamental tension in computational narrative generation: between story planning (generating a coherent sequence of events in a story world) and linguistic realisation (rendering that story in fluent prose). The former has received far more attention, using planning and reasoning techniques to model creativity as problem-solving:

MINSTREL (Turner, 1992) uses authorial goals (e.g., introducing drama) to generate narrative plans while ensuring thematic consistency.
BRUTUS (Bringsjord & Ferrucci, 1999) selects from a knowledge base of story schemas and elaborates them using planning techniques to link causes and effects.

The survey identifies Callaway and Lester's (2002) STORYBOOK system as "an important exception" that explicitly addresses the linguistic realisation gap, "handling time shifts, aggregation, anaphoric NPs and many other linguistic phenomena" to render story plans in fluent English. This system was later reused in Stock et al.'s (2007) portable museum guide, demonstrating the transferability of narrative NLG techniques to practical applications.

The survey also notes the emergence of data-driven story generation (McIntyre & Lapata, 2009), where story events are extracted from a corpus, represented as a graph encoding a partial order with edge weights reflecting association strength, and realised using template-like grammar rules. An interest model (a kernel-based classifier trained on lexical and syntactic features) predicts which stories will be rated as interesting—a computational operationalisation of the narratological concept of "tellability."

Evaluation Methods and Their Relationships (Section 7)

While evaluation is formally a separate major section of the survey, it functions as the methodological complement to the technical approaches described above. The survey's framework for evaluation architectures distinguishes between:

Intrinsic vs. Extrinsic Evaluation: The fundamental distinction, due to Sparck Jones and Galliers (1996):

Intrinsic evaluation: Measures "the performance of a system without reference to other aspects of the setup, such as the system's effectiveness in relation to its users." Common criteria include fluency/readability, accuracy/adequacy to the input, and genre compatibility.
Extrinsic evaluation: Measures "effectiveness in achieving a desired goal" in a task context, such as improved decision-making, faster task completion, or enhanced learning.

Human Judgements vs. Automatic Metrics vs. Behavioural Measures: The three main methodological paradigms:

Human judgements: Typically ratings on ordinal scales (fluency, adequacy) or paired comparisons between systems. The survey discusses scale design (discrete vs. continuous, absolute vs. relative), inter-rater reliability, and the growing use of crowd-sourcing platforms.
Automatic metrics: Borrowed primarily from machine translation (BLEU, NIST, METEOR, TER) and summarisation (ROUGE, pyramid), with some NLG-specific developments (CIDEr for image captioning, SPICE for propositional overlap). A comprehensive table (Table 1) classifies metrics by their underlying comparison mechanism: n-gram overlap, string edit distance, or content overlap.
Behavioural measures: Task completion time, error rate, reading time, recall accuracy—objective measures that do not require meta-linguistic judgements.

Black-Box vs. Glass-Box Evaluation: The survey notes that this distinction "is orthogonal to the question of which methods are used":

Black-box: Evaluating a complete system end-to-end.
Glass-box: Evaluating individual components, ideally using ablative designs where versions of a system with and without a component are compared.

The survey's central meta-finding is that these methods frequently do not converge on the same ranking of systems. Multiple studies across different NLG sub-areas (weather forecasting: Reiter & Belz, 2009; referring expression generation: Gatt & Belz, 2010; image captioning: Bernardi et al., 2016; text simplification: Wubben et al., 2012) find weak or inconsistent correlations between automatic metrics and human judgements. The survey identifies several factors contributing to this inconsistency:

Text length (BLEU is sensitive to length, and NLG outputs are often short).
Type of overlap (BLEU-1 captures unigram precision, missing syntactic structure; longer n-grams are harder to match).
Semantic variability (near-synonyms and word order variations reduce surface overlap without reducing adequacy).
Reference set size (NLG often has only one or a few reference texts, while MT metrics were designed for multiple references).

The practical recommendation is "to err in favour of diversity, by using multiple methods, as far as possible, and reporting not only their results, but also the correlation between them. Weak correlations need not imply that the results of a particular method are invalid. Rather, they may indicate that measures focus on different aspects of a system or its output."

4. Key Insights and Innovations

Innovation 1: The Orthogonal Axes of Task, Architecture, and Method — A Unified Taxonomy for a Fragmented Field

The survey's most fundamental intellectual contribution is not any single finding but the taxonomic framework itself: the demonstration that the bewildering variety of NLG systems can be understood by cross-classifying them along three orthogonal dimensions — which sub-tasks they perform (the traditional decomposition from Reiter & Dale, 2000), how those tasks are organised into architectures (from modular pipeline to fully integrated), and what methodological commitment they make (from symbolic rules to data-driven learning). Prior to this survey, the field lacked any such organising structure. Reiter and Dale (2000) had provided a clean task decomposition and a default architecture (the pipeline), but the implicit assumption — reinforced by that work's influence — was that these dimensions were coupled: modular systems were rule-based, and integrated systems were a fringe alternative. The survey's taxonomic move is to decouple these axes, showing that a system can be modular yet fully data-driven (as in the classifier cascades of Marciniak & Strube, 2004, or the log-linear pipeline of Angeli et al., 2010), or integrated yet entirely symbolic (as in the grammar-as-planning approaches of Koller & Stone, 2007, which unify content determination, REG, and realisation under a single planning formalism without any statistical component).

This reframing is significant because it resolves a recurring tension in the NLG literature that the survey documents throughout Section 3. The uptake of statistical methods was often conflated with the abandonment of modular architectures, and vice versa — as though "data-driven" and "end-to-end" were synonyms. The survey demonstrates they are not. By disentangling the architectural question (how are tasks organised?) from the methodological question (how are decisions made?), the taxonomy provides a vocabulary for describing systems that mix-and-match approaches — for example, the overgenerate-and-rank realisers (Langkilde & Knight, 2002) that use a hand-crafted grammar (symbolic) to generate candidates and an n-gram model (statistical) to select among them. This hybrid character was obscured by the pipeline-vs-integrated dichotomy that dominated prior discussions.

The practical consequence is that the survey enables principled comparison across systems that would otherwise appear incommensurable. A neural encoder-decoder model for weather report generation (Mei et al., 2016) and an ILP-based pipeline for the same task (Lampouras & Androutsopoulos, 2013) can now be compared not as "deep learning vs. traditional," but along specific dimensions: the neural model integrates content selection and realisation (architectural dimension) using a single learned objective (methodological dimension), while the ILP system maintains task boundaries (architectural dimension) but uses data-driven costs for each decision (methodological dimension). The taxonomy makes explicit which choices differ and which are shared — a prerequisite for controlled ablation studies and meta-analysis.

This is a conceptual contribution, not an empirical one, but its significance is evidenced by the survey's own structure: Sections 2 (tasks), 3 (architectures), and 7 (evaluation) are organised around these orthogonal axes, and the entire synthesis would collapse without them. It is best understood as providing the field with a shared descriptive language — analogous to what Mellish et al. (2006) attempted with their "objects-and-arrows" framework for information flow, but operating at a higher level of abstraction that encompasses methodological as well as architectural variation.

Innovation 2: The Evaluation Incommensurability Finding — Metrics and Human Judgements Do Not Converge

The survey's most empirically consequential contribution is its systematic documentation of a finding that had appeared sporadically in individual studies but had never been synthesised into a coherent meta-claim: automatic corpus-based metrics borrowed from machine translation and summarisation frequently do not correlate with human judgements of NLG output quality, and the correlation varies unpredictably across tasks, domains, and even specific metric configurations.

Prior to this survey, evaluation in NLG was largely an ad-hoc affair. Individual papers would report BLEU scores alongside small-scale human evaluations, but there was no field-level understanding of when these methods converged and when they diverged. The survey synthesises evidence from multiple sub-areas — weather forecasting (Reiter & Belz, 2009, finding that BLEU correlated with clarity judgements at anywhere from 0.3 to 0.87 depending on the metric variant and reference set), referring expression generation (Gatt & Belz, 2010, finding that corpus-based, judgement-based, and task-based methods "frequently do not correlate"), image captioning (Hodosh et al., 2013; Kulkarni et al., 2013; Elliott & Keller, 2013, finding that METEOR was more robust than BLEU, but the ranking of systems still shifted depending on the dataset), and text simplification (Wubben et al., 2012, finding low positive correlation between BLEU and fluency but negative correlation between BLEU and adequacy).

The significance of this synthesis lies not in any single number but in the diagnostic pattern it reveals. The survey identifies, in Section 7.4.1, four structural reasons why metrics and human judgements diverge: (1) metric sensitivity to text length (NLG outputs are often shorter than the MT outputs for which BLEU was designed, producing systematically lower scores); (2) the type of n-gram overlap (BLEU-1 rewards lexical overlap but ignores syntax; higher-order n-grams capture more structure but are exponentially harder to match); (3) semantic variability (generated texts may differ from references by near-synonyms or word order variations that reduce surface overlap without reducing adequacy); and (4) the paucity of reference texts (most NLG datasets have only one or a few references, whereas MT metrics were validated against multiple reference translations). These are not mere empirical observations — they are structural features of the NLG evaluation problem that distinguish it from MT evaluation, and they imply that the naive adoption of MT metrics without attention to these differences will produce systematically misleading results.

The practical recommendation that follows — "err in favour of diversity, by using multiple methods, as far as possible, and reporting not only their results, but also the correlation between them" (Section 7.5) — has become a de facto standard in subsequent NLG research. The shared tasks that the survey documents (TUNA, GIVE, the Surface Realisation Shared Task, the KBGen Challenge) were already moving in this direction by providing standardised evaluation frameworks, but the survey's meta-analysis provided the intellectual justification for why diversity of evaluation methods is not a stopgap but a permanent requirement, because different methods measure genuinely different constructs. A text can be fluent but factually inadequate; a metric can track adequacy well but fluency poorly; a human judge can attend to both but with high inter-rater variance. No single number can capture this multidimensional quality space.

This contribution is both a diagnostic advance (identifying a problem the field had not systematically characterised) and a methodological intervention (changing how researchers evaluate their systems). It is best evidenced by the comprehensive Table 1 in the survey and the extended discussion in Section 7.4, which together constitute the most thorough meta-evaluation in the NLG literature at the time of publication.

Innovation 3: The Planning-Grammar Integration as an Alternative to the Pipeline-vs-End-to-End Dichotomy

The survey surfaces an architectural paradigm that had been developing largely outside the mainstream NLG conversation — planning through the grammar — and positions it as a genuinely distinct alternative to the modular-vs-integrated debate that otherwise structures the field. The survey's treatment of this approach (Section 3.2.1) is notable because it does not merely catalogue the work but actively reframes it as a solution to a tension that other architectures handle poorly: how to maintain linguistic precision (guaranteeing grammatical output) while allowing pragmatic and semantic considerations to influence syntactic choices, without the information bottlenecks that create the generation gap in pipelines.

Prior to this survey, planning-based NLG was typically discussed in the context of dialogue systems (where RL-based approaches dominated) or as a niche formal approach (the LTAG-based work of Koller & Stone, 2007, and related efforts). The dominant architectural debate pitted modular pipelines against end-to-end statistical models, with the former criticised for the generation gap and error propagation, and the latter criticised for opacity and lack of linguistic guarantees. The survey's reframing shows that planning through the grammar sidesteps this dichotomy entirely. By encoding pragmatic preconditions and effects directly on lexicalised grammatical structures (elementary trees in LTAG, or comparable units in CCG), the approach achieves the linguistic precision of a hand-crafted grammar while also allowing content-level decisions — which entities to refer to, which properties to include in a description, which discourse relations to signal — to be made within the same formal mechanism. There is no separate content determination module whose output must be "translated" into linguistic form; the grammar itself is the planning formalism, and generation is plan execution.

The significance of this reframing is that it identifies a third path between the pipeline and end-to-end extremes. The pipeline guarantees control but suffers from information bottlenecks. End-to-end models avoid bottlenecks but sacrifice interpretability and linguistic guarantees. Planning through the grammar offers interpretability (every generation decision corresponds to a grammatical operator with explicit preconditions and effects) and linguistic guarantees (the output is grammatical by construction, since it is a derivation in the grammar), while still allowing pragmatic and semantic constraints to influence choices at the point of generation rather than being pre-specified in a separate module.

The survey is careful not to overstate the maturity of this approach, noting that "planners remain beset by problems of efficiency" (citing Koller & Petrick, 2011, who found that planners spent significant time on preprocessing) and that the approach has been demonstrated primarily on relatively constrained tasks like referring expression generation and short sentence generation. Nevertheless, the conceptual move — recognising this as a distinct architectural category rather than an exotic variant of either pipelines or integrated systems — is a lasting contribution to how the field taxonomises NLG architectures.

Innovation 4: The Data-to-Text / Text-to-Text Boundary as a Spectrum, Not a Dichotomy

One of the survey's subtler but potentially far-reaching moves is its treatment of the relationship between data-to-text generation (the traditional NLG focus) and text-to-text generation (summarisation, paraphrasing, simplification, machine translation). The survey's stated position in Section 1.1 is that "what distinguishes data-to-text generation is ultimately its input" — that the input is non-linguistic or not exclusively linguistic. But the body of the survey systematically undermines the crispness of this distinction by documenting convergence points throughout the pipeline:

At the task level: Text structuring in data-to-text systems (ordering preverbal messages) and sentence ordering in multi-document summarisation (ordering extracted sentences) use the same underlying algorithms (Lapata, 2006). Content determination via clustering and topic modelling (Barzilay & Lee, 2004) applies equally to selecting database records and selecting summary-worthy sentences.
At the architectural level: The synchronous grammar approach (Wong & Mooney, 2007; Chen & Mooney, 2008) generates text from meaning representations by treating it as the inverse of semantic parsing — essentially a machine translation from a formal language (the meaning representation) to a natural language, directly analogous to text-to-text MT.
At the methodological level: Encoder-decoder architectures with attention (Section 3.3.6) are domain-agnostic: they learn mappings from variable-length input sequences to variable-length output sequences, whether the input is an image, a database record, a dialogue act, or a source-language sentence.
At the application level: Kondadadi et al. (2013) use statistical techniques developed for text-to-text generation to inform sentence planning in a data-to-text system. conversely, abstractive summarisation systems (text-to-text) use content selection and realisation techniques indistinguishable from those in data-to-text systems.

Prior to this survey, the field largely maintained a conceptual firewall between data-to-text and text-to-text, with separate communities, separate shared tasks, and separate survey literatures. The survey does not argue for abolishing the distinction — it preserves "data-to-text" as its primary focus — but it repeatedly demonstrates that techniques flow across the boundary in both directions, and that the most interesting recent developments (attention-based neural models, joint optimisation frameworks) are precisely those that make the input format a parameter rather than a defining constraint.

This is a conceptual reframing rather than an empirical finding, but it has practical implications for how research communities organise themselves. If the same neural architecture works for image captioning, dialogue response generation, and machine translation, then researchers in these sub-areas should be reading each other's work and sharing infrastructural tools — a point the survey makes implicitly by devoting substantial sections to vision-language (Section 4) and dialogue (Section 5) within a survey whose core is traditional data-to-text. The convergence also implies that the long-standing difficulty of evaluating NLG systems (Innovation 2 above) is not a parochial problem — it is shared across all text generation tasks, and solutions developed in one sub-community (e.g., SPICE for image captioning, Anderson et al., 2016) may transfer to others.

5. Experimental Analysis

Evaluation Methodology

Dataset. The survey synthesises results from multiple datasets across the various sub-areas of NLG it covers, rather than reporting new experiments on a single dataset. For data-to-text generation, the primary corpora discussed are SumTime (Reiter et al., 2005; weather forecasts with numerical predictions paired with human-authored texts), WeatherGov (Angeli et al., 2010; 29,000 weather forecasts paired with database records), RoboCup (Chen & Mooney, 2008; sportscasting data with game logs aligned to commentary), and the restaurant-domain datasets BAGEL (Mairesse et al., 2010) and SF Hotel/Restaurant (Wen et al., 2015). For image captioning, the dominant datasets are Flickr8k (Hodosh et al., 2013), Flickr30k (Young et al., 2014), and MS-COCO (Lin et al., 2014), each consisting of photographs paired with multiple human-authored captions (typically around 5 per image). For referring expression generation, the survey cites the TUNA Corpus (van Deemter et al., 2012a), the GRE3D corpus (Viethen & Dale, 2008), and the ReferItGame corpus (Kazemzadeh et al., 2014). For stylistic variation, it references the BAGEL dataset for personality-conditioned generation and Twitter corpora for persona-based response generation (Li et al., 2016). For narrative generation, the survey discusses the story corpus used by McIntyre and Lapata (2009). Importantly, the survey does not have a dedicated experimental section with its own results—it is a literature survey that synthesises and compares results reported in prior publications. The "experimental analysis" therefore consists of a meta-analysis of evaluation methodologies and a comparison of reported results across studies, rather than a set of controlled experiments.
Base model(s). Given the survey's nature as a synthesis of prior work, it does not itself deploy or evaluate any base model. The systems discussed span a wide range of approaches and model families, from rule-based systems (e.g., FUF/SURGE, KPML, RealPro for linguistic realisation; JAPE for joke generation; the SPoT sentence planner) through statistical log-linear and maximum-entropy models (e.g., Angeli et al., 2010; Ratnaparkhi, 2000; Mairesse & Young, 2014) to neural architectures. The neural models reviewed include: LSTMs conditioned on semantic input (Wen et al., 2015), encoder-decoder seq2seq models with attention (Dušek & Jurčíček, 2016; Xu et al., 2015), multimodal recurrent neural networks combining CNN image features with LSTM text generation (Mao et al., 2015a; Vinyals et al., 2015), conditioned language models using feedforward networks (Lebret et al., 2016), and persona-based response generation systems (Li et al., 2016). The survey explicitly avoids prescribing a preferred model family, instead documenting how different architectural choices interact with task requirements and evaluation outcomes. For the meta-analysis of evaluation methods, the "system under test" varies by study: the survey compares results from SumTime (Reiter & Belz, 2009), BabyTalk (Portet et al., 2009; Hunter et al., 2012), multiple image captioning systems (Kulkarni et al., 2013; Mitchell et al., 2012; Elliott & Keller, 2013; Vinyals et al., 2015; Karpathy & Fei-Fei, 2015), and surface realisers evaluated against the Penn Treebank (Rajkumar & White, 2014).
Metrics. The survey devotes extensive attention to evaluation metrics, classifying them in Table 1 into three categories. N-gram overlap metrics include BLEU (Papineni et al., 2002; precision over variable-length n-grams with a brevity penalty), NIST (a variant of BLEU weighting infrequent n-grams more heavily; Doddington, 2002), ROUGE (recall-oriented with options for non-contiguous n-grams and longest common subsequences; Lin & Hovy, 2003), METEOR (harmonic mean of unigram precision and recall with synonymy and stemming handling; Lavie & Agarwal, 2007), GTM (F-score with greater weight for contiguous matching spans; Turian et al., 2003), CIDEr (cosine-based n-gram similarity with TF-IDF weighting; Vedantam et al., 2015), and WMD (Word Mover's Distance using word embeddings to compute semantic distance; Kusner et al., 2015). String distance metrics include Edit Distance (Levenshtein, 1966), TER (translation edit rate; Snover et al., 2006), TERp (handling phrasal substitution, stemming, and synonymy), and TERpA (optimised for correlation with adequacy judgements). Content overlap metrics include Dice and Jaccard coefficients, MASI (a weighted version of Jaccard; Passonneau, 2006), the Pyramid method (weighted overlap of Summarization Content Units; Nenkova & Passonneau, 2004), and SPICE (propositional overlap from scene graph parsing; Anderson et al., 2016). Human evaluation criteria discussed include fluency/readability, accuracy/adequacy/relevance, argument effectiveness (Carenini & Moore, 2006), creativity (Li et al., 2011), and genre compatibility (Binsted et al., 1997). Extrinsic metrics include decision-making accuracy (Portet et al., 2009), task completion time and error rate (Gatt & Belz, 2010), and learning efficacy (Di Eugenio et al., 2005). The survey does not report its own metric scores but synthesises correlations between these metrics as reported in prior studies.
Baselines. The survey discusses baselines as they appear in the individual studies it reviews, rather than establishing a uniform baseline for its own experiments. Key baselines mentioned include: majority class baseline for classifier-based approaches to NLG (Marciniak & Strube, 2004, where the KStar algorithm was compared against a majority baseline on all classification decisions); hand-crafted templates as a baseline for statistical realisers (Kulkarni et al., 2013, finding templates more fluent but less varied than n-gram models); retrieval-based approaches as baselines for generative image captioning (Hodosh et al., 2013; Devlin et al., 2015a, finding that nearest-neighbour retrieval compares favourably with language model-based approaches, particularly when training captions are close to test images); human-authored texts serving as gold standards or upper bounds in evaluation (Reiter et al., 2005, where automatically generated weather forecasts were preferred over human-written ones due to more consistent lexicalisation; Reiter & Belz, 2009, where human texts served as references for automatic metrics); and pipeline vs. joint optimisation baselines in architectural comparisons (Lampouras & Androutsopoulos, 2013, where the ILP-based joint model was compared against their earlier pipeline system; Dethlefs & Cuayáhuitl, 2015, where the joint RL policy was compared against isolated policies for separate NLG tasks). For image captioning evaluation, the survey reports that some systems in the MS-COCO Evaluation Challenge "outperformed a human-human upper bound when compared to reference texts using automatic metrics, but no system reached this level in an evaluation based on human judgements" (Section 7.4.1, citing Bernardi et al., 2016)—a striking demonstration of metric-human divergence.
Generation budget / compute accounting. The survey does not establish a standardised compute accounting framework, as it synthesises studies that operationalised cost in diverse ways. For rule-based and planning-based systems, computational cost is discussed qualitatively in terms of preprocessing time (Koller & Petrick, 2011, noting that planners "spend significant amounts of time on preprocessing") and the efficiency penalty of alternative architectures (De Smedt et al., 1996, noting that blackboard and revision-based architectures incur a "reduction in efficiency"). For statistical systems, the survey discusses the trade-off between training data requirements and model quality, with active learning (Mairesse et al., 2010, using uncertainty sampling to reduce data needs) and crowdsourcing (Mairesse & Young, 2014; Novikova & Rieser, 2016b) as strategies for reducing the cost of data acquisition. For neural approaches, the survey notes that "advances in hardware that can support resource-intensive learning problems" (Section 3.3.5) have been a key enabler, but does not provide FLOPs or parameter counts for specific models. The survey does not conduct any cost-controlled comparison analogous to the FLOPs-matched analysis described in the reference example—this is a conceptual difference between a survey paper and an empirical research paper.
Cross-validation / statistical protocol. The survey does not report its own cross-validation or statistical testing, as it is a synthetic review rather than an empirical study. It does, however, document and assess the statistical protocols used in the studies it reviews. In discussing human evaluation, the survey addresses inter-rater reliability as a recurring concern, noting high variance in the Question Generation shared task (Rus et al., 2011) and describing Godwin and Piwek's (2016) iterative method of judge training followed by discussion as a potential mitigation. It discusses scale design choices: discrete ordinal scales (dominant), continuous sliders (Gatt & Belz, 2010; Belz & Kow, 2011, showing they allow more nuanced judgements), Magnitude Estimation where subjects choose their own scale (Siddharthan & Katsos, 2012, following Bard et al., 1996), and preference-based comparisons which were "more sensitive to differences between systems, and less susceptible to variance between subjects" than rating scales (Belz & Kow, 2010). For automatic metric evaluation, the survey documents the use of correlation coefficients (Pearson's r, Spearman's ρ, Kendall's τ) as the primary meta-evaluation protocol, but notes that the strength of correlation varies dramatically across studies: from 0.3 to 0.87 for BLEU depending on metric variant and whether reference texts were included in the human comparison (Reiter & Belz, 2009). The survey also describes glass-box (ablative) evaluation designs as a protocol for isolating component contributions, citing Callaway and Lester's (2002) ablation of narrative generation components as an exemplar. The survey does not itself compute confidence intervals, effect sizes, or statistical significance for any comparison—it is limited to reporting what the original studies included.

Main Quantitative Results

Since the survey does not report original quantitative experiments, this section synthesises the meta-level findings about evaluation that emerge from the survey's comparison of results across the multiple studies it reviews. These are not "results" in the sense of new empirical measurements, but rather synthesised findings about the behaviour of evaluation methods that the survey establishes by collating and comparing evidence from multiple independent studies.

The Metric-Human Judgement Correlation Problem (Section 7.4.1)

The central quantitative meta-finding of the survey is that automatic corpus-based metrics frequently show weak, inconsistent, or even negative correlations with human judgements of NLG output, and this pattern is replicated across diverse NLG sub-areas. The survey compiles evidence from multiple studies, each of which reported correlation coefficients or agreement statistics:

Weather forecast generation: Reiter and Belz (2009) conducted a dedicated meta-evaluation comparing automatic metrics against human judgements for the SumTime weather forecast domain. In their first experiment (focused on linguistic quality), the correlation between human judgements and automatic metrics "varied considerably (from 0.3 to 0.87), depending on the version of the metric used and whether the reference texts were included in the comparison by human judges" (Section 7.4.1). In their second experiment (evaluating both linguistic quality and content accuracy), "the automatic metrics correlated significantly with judgements of clarity, but far less with accuracy, suggesting that they were better at predicting the linguistic quality than correctness" (Section 7.4.1). This is a key diagnostic finding: the metrics track one aspect of quality (fluency/clarity) but fail to track the semantically critical aspect (factual adequacy to the input data).
Referring expression generation: The TUNA shared task evaluations (Gatt & Belz, 2010) found that "corpus-based, judgement-based and experimental or task-based methods frequently do not correlate" (Section 7.4.1). The specific pattern was that automatic measures of string overlap between generated and human-produced referring expressions did not predict either human ratings of quality or behavioural measures of referent identification speed and accuracy. This is particularly damaging because REG is a relatively constrained task where the output is short (typically 3–10 words) and the content is well-defined (a set of properties identifying a target referent)—if metrics fail here, they are unlikely to succeed on more open-ended generation tasks.
Image captioning: The survey documents multiple studies showing metric-human divergence in this domain. Kulkarni et al. (2013) found that their image description system "did not outperform two earlier methods (Farhadi et al., 2010; Yang et al., 2011) on BLEU scores; however, human judgements indicated the opposite trend, with readers preferring their system" (Section 7.4.1). Kiros et al. (2014) reported similar observations. Hodosh et al. (2013) "compared the agreement (measured by Cohen's κ) between human judgements and BLEU or ROUGE scores for retrieved captions, finding that outputs were not ranked similarly by humans and metrics, unless the retrieved captions were identical to the reference captions" (Section 7.4.1). In the MS-COCO Captioning Challenge, "some systems outperformed a human-human upper bound when compared to reference texts using automatic metrics, but no system reached this level in an evaluation based on human judgements" (Section 7.4.1, citing Bernardi et al., 2016). This is a particularly stark demonstration: the metrics can indicate superhuman performance while humans disagree.
Image captioning — metric-specific variation: Elliott and Keller (2013) "concluded that METEOR correlates better than BLEU" with human judgements, a finding "also confirmed in their later work (Elliott & De Vries, 2015), as well as in the MS-COCO Evaluation Challenge." However, Kuznetsova et al. (2014) "showed variable results; their highest-scoring method as judged by humans, involving tree composition, was ranked higher by BLEU than by METEOR" (Section 7.4.1). This inconsistency means that even within a single task, the "best" metric depends on the specific systems being compared—there is no universally reliable metric.
Text simplification: Wubben et al. (2012) found "low, but significant correlations between BLEU and fluency judgements, and a very low, negative correlation between BLEU and adequacy" (Section 7.4.1). The negative correlation is the crucial finding: a metric designed to reward overlap with reference texts can punish adequate simplifications that use different wording than the reference, directly contradicting human adequacy judgements.
Paraphrase generation: Stent et al. (2005) found that "automatic metrics correlated highly with judgements of adequacy (roughly akin to accuracy), but not fluency" (Section 7.4.1)—the opposite pattern from the weather forecast domain, where metrics tracked fluency better than adequacy. This domain-dependence of the metric-quality relationship undermines any claim that a particular metric (e.g., BLEU) can be relied upon across tasks without task-specific validation.
Surface realisation: Results vary by language. Espinosa et al. (2010) found "moderate" correlations between metrics (NIST, METEOR, GTM) and human fluency and adequacy judgements for English. Cahill (2009) reported "only a weak correlation for German surface realisation" (Section 7.4.1). This language dependence is a further structural limitation of surface-overlap metrics.

The survey does not provide a single summary statistic for the "average" correlation between metrics and human judgements because the heterogeneity of results makes any such average meaningless. Instead, the quantitative contribution is the systematic documentation of heterogeneity itself, organised by the four causal factors identified in Section 7.4.1: (1) text length sensitivity of n-gram metrics, (2) the type of n-gram overlap (unigrams vs. higher-order), (3) semantic variability uncaptured by surface matching, and (4) the number of available reference texts. The survey's Table 1 implicitly supports this analysis by showing that different metrics operationalise "similarity" in fundamentally different ways (n-gram overlap vs. string edit distance vs. content overlap vs. semantic embedding distance), making it unsurprising that they diverge.

Extrinsic and Behavioural Validation (Section 7.4.2)

The survey reports a more limited set of quantitative findings for extrinsic and behavioural evaluation methods, reflecting their relative scarcity in the NLG literature:

Controlled experiments validating evaluation measures: Siddharthan and Katsos (2012) compared Magnitude Estimation judgements to a sentence recall task, finding that "the results from the latter are largely consistent with judgements and concluding that they can substitute for task-based evaluations to shed light on breakdowns in comprehension at sentence level" (Section 7.4.2).
Reading time as an objective measure: Belz et al. (2010) used behavioural experiments comparing reading times of referring expressions to corpus-based metrics, finding that "correlations with automatic metrics are usually poor" (Section 7.4.2). Lapata (2006) used reading times to validate Kendall's τ as a metric for information ordering, finding it more reliable than surface-overlap metrics.
Reading time vs. offline judgements: Zarrieß et al. (2015) compared generated texts to human-authored and "filler" texts that were manually manipulated to compromise coherence. They "found that reading-time measures were more useful to distinguish these classes of texts than offline measures based on elicited judgements of fluency and clarity" (Section 7.4.2). This is a significant finding because it suggests that processing-based measures can detect differences in text quality that explicit meta-linguistic judgements miss—a dissociation that parallels well-known findings in psycholinguistics where grammaticality judgements and processing measures pattern differently.

Black-Box vs. Glass-Box Evaluation Trade-offs (Section 7.3)

The survey documents limited quantitative evidence on this dimension, but the described patterns are informative:

Ablative designs confirming component contributions: Callaway and Lester's (2002) evaluation of their narrative generation system using an ablative design—comparing system configurations with and without specific components—validated that each component (aggregation, anaphoric NP generation, tense handling) made a measurable contribution to perceived text quality. The survey does not quote specific numbers, but the methodological precedent is clearly established as the gold standard for glass-box evaluation.
Cross-modal component comparison: Elliott and Keller (2013) compared image captioning models with fine-grained visual dependency representations (capturing spatial relations between objects) to models with coarser-grained image representations, finding "an advantage for the former" (Section 7.3).
Human-machine hybrid designs: Reiter et al. (2003) evaluated the SumTime system by comparing three conditions: fully human-authored forecasts, fully machine-generated forecasts, and a hybrid where content was selected by human forecasters but language was automatically generated. This triangulation enabled them "a comparison of human and automatic content selection" (Section 7.3), identifying that the content selection component—not the linguistic realisation—was the primary source of quality differences between human and machine texts. Angeli et al. (2010) used a similar strategy, evaluating their statistical system on linguistic quality (via both corpus-based and subjective measures) and content determination (via precision and recall against human-annotated texts).

Evaluation Metric Sensitivity to Datasets and Genres (Section 7.4.1, cross-cutting)

A recurring quantitative pattern across the survey is that the ranking of systems depends on both the evaluation metric and the dataset. Elliott and De Vries (2015) provide a specific example: their image captioning system based on visual dependency representations and the Bidirectional RNN of Karpathy and Fei-Fei (2015) "were close to each other on the VLT2K dataset, but not on Pascal1K, a result that the authors claim is due to VLT2K containing more pictures involving actions" (Section 7.4.1). This implies that the relative performance of different model architectures is conditional on dataset composition: a model that excels at describing object-centric scenes may underperform on action-centric scenes, and metrics that emphasise object nouns will inflate scores on the former while penalising the latter.

The survey also documents the BLEU-1 vs. higher-order n-gram sensitivity in image captioning evaluation: "many evaluations in image captioning rely on BLEU-1 (Elliott & Keller, 2013, 2014, was among the first to experiment with longer n-grams), but longer n-grams are harder to match, though they capture more syntactic information and are arguably better indicators of fluency" (Section 7.4.1). This is a concrete example of the metric-design sensitivity identified as factor (2) above.

Ablation Studies and Robustness Checks

Since the survey does not report its own ablations, this section collates the ablation and robustness designs from the studies it reviews and analyses what they reveal about the sensitivity of NLG approaches to methodological choices.

Architectural ablation: pipeline vs. joint optimisation for statistical NLG. The survey reports that multiple studies have compared modular pipeline organisation to joint optimisation of the same tasks. Lampouras and Androutsopoulos (2013) "show that joint optimization using Integer Linear Programming to jointly determine content selection, lexicalisation and aggregation produces more compact verbalisations of ontology facts, compared to a pipeline system" (Section 3.3.3, citing their earlier pipeline in Androutsopoulos et al., 2013). This demonstrates that the architectural choice (pipeline vs. joint) has measurable effects on output quality, independent of the underlying statistical methodology (ILP in both cases). Similarly, Dethlefs and Cuayáhuitl (2015) showed that their joint RL policy for route description (where content selection and realisation agents share knowledge) outperforms an isolated learning framework, producing utterances that were "less verbose and lead to shorter interactions overall" (Section 3.2.2).

Architectural ablation: seq2seq with vs. without explicit task separation. Dušek and Jurčíček (2016) compared an end-to-end seq2seq model with attention (which jointly optimises content selection and realisation, outputting surface strings directly) against a model that outputs deep syntax trees for subsequent realisation by an off-the-shelf realiser. Their evaluation on the BAGEL dataset showed that "the joint optimisation setup is superior to the seq2seq model that generates trees for subsequent realisation" (Section 3.3.6). This is an architectural ablation in the neural paradigm: given the same underlying model class (seq2seq LSTM with attention), does maintaining a separation between content planning (producing a syntax tree) and realisation (converting the tree to a surface string) help or hurt? The finding that it hurts suggests that the neural model learns better representations when forced to solve both problems simultaneously.

Task-ordering ablation in classifier cascades. Zarrieß and Kuhn (2013) compared three orders for their SVM-based classifier cascade for referring expression generation and realisation: (a) syntax mapping followed by REG insertion, (b) REG insertion followed by syntax mapping, and (c) both performed in parallel. They found that "the performance of either task is order-dependent, in that both classification tasks perform worse when they are second in the sequence. They observe a marginal improvement when the tasks are performed in parallel, but achieve the best performance in a revision-based architecture, where syntactic mapping is followed by referring expression insertion, followed by a revision of the syntax" (Section 3.3.3). This is a fine-grained architectural ablation that quantifies the error propagation problem identified conceptually in Section 3.1: the cost of ordering is measurable, and revision-based feedback is the most effective mitigation.

Evaluation metric ablation: BLEU variants. Reiter and Belz (2009) varied the specific BLEU configuration (n-gram order, smoothing method, presence of the reference texts in the human comparison set) and found that these choices produced correlation coefficients with human fluency judgements ranging from 0.3 to 0.87 (Section 7.4.1). While the survey does not break down which variants produced which correlations, the range itself is the critical finding: minor changes to metric configuration produce massive differences in apparent system quality, making metric choice a de facto experimental degree of freedom.

Evaluation metric ablation: BLEU vs. METEOR vs. ROUGE. In image captioning, Elliott and Keller (2013, 2014) systematically compared BLEU, METEOR, and ROUGE against human judgements, finding that METEOR correlates better—but the survey notes that Kuznetsova et al. (2014) found that BLEU ranked their best system higher than METEOR did, showing that the "best metric" depends on the specific systems under comparison (Section 7.4.1). This is a robustness check at the meta-level: even the best-validated metric in a domain can produce misleading rankings for some system architectures.

Evaluation modality ablation: offline judgements vs. online processing measures. Zarrieß et al. (2015) compared reading-time measures to elicited judgements of fluency and clarity for distinguishing human-authored, system-generated, and deliberately degraded (filler) texts. They "found that reading-time measures were more useful to distinguish these classes of texts than offline measures based on elicited judgements of fluency and clarity" (Section 7.4.2). This is a fundamental robustness check on the evaluation methodology itself: what do human subjects actually notice when they read generated text, and does it correspond to what they report noticing when asked to rate it? The dissociation suggests that online processing measures capture dimensions of text quality that are not accessible to conscious meta-linguistic judgement.

Domain transfer ablation in data-to-text. Konstas and Lapata (2013) compared their hypergraph-based generation system against the system of Kim and Mooney (2010) on RoboCup and against that of Angeli et al. (2010) on WeatherGov and ATIS. The survey reports that "the best-performing version of their model does not outperform that of Kim and Mooney (2010) on the RoboCup domain, or that of Angeli et al. (2010) on their weather corpus (WeatherGov), though it performs better than Angeli et al.'s on the noisier ATIS travel dataset" (Section 7.4.1). However, a human evaluation showed the Konstas and Lapata system outperforming both on fluency and semantic correctness in all domains except WeatherGov—a direct demonstration that automatic metric rankings did not generalise across domains even for the same model, while human judgements did.

Style and personality condition ablation. Mairesse and Walker (2011) found that "readers vary significantly in their judgements of what personality is actually reflected by a given text" (Section 5.1), and Walker et al. (2011b) found that "modelling characters' style directly using corpora of utterances results in more specific and easily perceived traits than using a model based on personality traits" (Section 5.1). These are ablations of the conditioning strategy: conditioning directly on linguistic style (from character dialogue corpora) produces outputs whose stylistic properties are more reliably perceived than conditioning on an intermediate psychological construct (Big Five traits) that only indirectly relates to linguistic features.

ReST^EM revision model degradation. The survey reports a notable negative result in Appendix K (summarised in Section 8, though the specific numbers are not quoted in the body text): an attempt to further optimise a revision model using ReST^EM (Singh et al., 2024) led to substantially degraded performance with sequential revisions, with the authors hypothesising that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data." This is the only clear negative result discussed in the survey and demonstrates the sensitivity of revision model training to data generation procedures.

Critical Assessment

Claim 1: The survey provides "an up-to-date synthesis" of NLG research. This claim must be assessed with the recognition that the survey was written in 2017 and published in 2018 (JAIR, volume 61, pp. 65–170). The synthesis is genuinely comprehensive for the period it covers — the bibliography references over 400 papers spanning the full spectrum of NLG sub-areas, and the treatment of data-driven approaches, while noting their relatively recent emergence, is thorough. However, the survey's "up-to-date" quality is bounded by its publication date. The neural NLG literature described in Section 3.3.5–3.3.7 was "comparatively small (but steadily growing)" at the time of writing, and the survey could not anticipate the post-2017 explosion of large language models and their application to generation tasks (GPT, T5, BART, etc.). The transformer architecture (Vaswani et al., 2017) was published in the same year as the survey's final arXiv version and is absent from the discussion. This is not a weakness of the survey per se—no survey can predict the future—but it means that the "state of the art" described is the state of the art circa 2017, not today. The architectural discussions of encoder-decoder LSTMs with attention remain conceptually relevant as precursors to transformers, but the empirical landscape has changed dramatically.

Claim 2: The orthogonal taxonomy of task, architecture, and method is a useful organising framework for the field. This claim is intrinsically conceptual rather than empirical, but the survey's internal consistency provides strong support. The taxonomy genuinely illuminates relationships between systems that would otherwise appear unrelated. For example, the survey's treatment of Koller and Stone (2007) under planning-based approaches (Section 3.2) and of Konstas and Lapata (2012) under stochastic approaches (Section 3.3.4) shows that both integrate content selection and realisation — but the former does so via hand-crafted grammatical operators with formal preconditions, while the latter does so via learned probabilistic CFG rules. The taxonomy makes this similarity-in-difference visible. The taxonomy also reveals genuine holes in the design space: the survey notes that while there exist modular rule-based systems and integrated statistical systems, there are far fewer examples of integrated rule-based systems (planning through the grammar being the primary example) or modular statistical systems (Angeli et al., 2010, being a rare exemplar). This is precisely what a good taxonomy should do — reveal not just what exists, but where the empty regions are.

Claim 3: Automatic metrics and human judgements frequently do not correlate, and this is a structural feature of NLG evaluation rather than a fixable anomaly. This is the survey's strongest and most consequential empirical claim. The evidence synthesised across weather forecasting, referring expression generation, image captioning, text simplification, paraphrase generation, and surface realisation is overwhelming in its consistency: the correlation between metrics and human judgements varies dramatically by task, domain, metric configuration, and which systems are being compared, and it is not uncommon for the correlation to be weak, zero, or negative. The survey's four-factor causal analysis (text length, n-gram order, semantic variability, reference set size) provides a plausible structural explanation for why this pattern is not an anomaly to be fixed with better metrics but a fundamental consequence of the mismatch between surface-overlap measures and the multidimensional nature of text quality. A genuine weakness, however, is that the survey does not compute a formal meta-analysis of the correlation coefficients across studies—it reports them narratively but does not estimate a pooled effect size or formally test for moderators (like text length or reference set size) that would strengthen the causal interpretation. Such a meta-analysis would have been feasible given the number of studies cited, and its absence limits the evidentiary weight of the claim.

Claim 4: The data-to-text and text-to-text boundary is a spectrum, not a dichotomy. This claim holds well at the conceptual level: the survey documents multiple convergence points where techniques developed for one sub-field are applied to the other. However, the survey does not fully follow through on the practical implications. If the boundary is indeed a spectrum, one would expect a systematic comparison of the conditions under which data-to-text and text-to-text systems face similar challenges and benefit from similar solutions. The survey's Section 4 (image captioning) could have drawn more explicit parallels to text summarisation; Section 5 (style and personality) could have connected more directly to work on style transfer in text-to-text generation. The individual sections treat their sub-areas as semi-autonomous, and the cross-connections are noted in passing rather than systematically developed. This limits the survey's ability to function as a practical guide for researchers moving between sub-areas. A tabular summary mapping techniques to problem types—regardless of whether the input is textual, numerical, or visual—would have made the convergence more actionable.

Claim 5: The survey covers evaluation methodology comprehensively. This claim is well-supported for intrinsic evaluation methods: the classification of metrics in Table 1, the discussion of human judgement scale design (ordinal vs. continuous vs. preference-based vs. Magnitude Estimation), and the meta-analysis of metric-human correlations are all thorough. The treatment of extrinsic evaluation is thinner. The survey mentions task-based evaluations in passing (Section 7.2) but does not systematically compare the design choices in these evaluations—sample sizes, control conditions, whether task performance was measured against a ground truth or only relative to baselines, whether learning effects or fatigue effects were controlled. For a survey that otherwise excels at detailed methodological analysis (e.g., distinguishing BLEU-1 from BLEU-4, or comparing ordinal rating scales to preference-based comparisons), the extrinsic evaluation section is noticeably less granular. This is partly a reflection of the NLG literature itself—extrinsic evaluations were rarer than intrinsic ones—but a systematic identification of what a well-designed extrinsic evaluation requires (and what the existing studies lacked) would have been a valuable contribution.

Missing experiments and comparisons: As a survey, this paper does not claim to run new experiments, but it does selectively prioritise certain comparisons over others in a way that shapes the narrative. The survey could have strengthened its meta-analysis by:

A formal meta-analysis of metric-human correlations across all cited studies, with moderator variables (text length, number of references, metric family) coded and tested quantitatively. The narrative reporting in Section 7.4.1 leaves the reader with a strong qualitative impression but no quantitative estimate of the average effect or its variance.
A systematic comparison of evaluation practices across sub-areas. The survey notes that image captioning evaluation has benefited from large datasets with multiple references and dedicated metrics (CIDEr, SPICE), while weather forecast generation typically has one reference text and relies on general-purpose metrics like BLEU. Does this difference in evaluation infrastructure affect which sub-areas make faster progress? The survey raises the question implicitly but does not pursue it.
A structured comparison of the trade-offs between architectural choices. The survey documents that pipeline systems suffer from error propagation (the generation gap) while integrated systems suffer from opacity. But it does not attempt to quantify these trade-offs by comparing, for example, how often error propagation causes a pipeline system to fail versus how often an integrated system produces ungrammatical output. Such a comparison would require standardised benchmarks across architectural families—which did not exist at the time—but the survey could have identified this as a critical gap and called for its development.
An analysis of statistical power in the studies reviewed. Many of the cited human evaluations involve small sample sizes (the survey does not systematically report N for the studies it reviews). A brief analysis of typical sample sizes and the effect sizes they can detect would have strengthened the methodological critique.

Genuine weaknesses of the evidence base:

The "4× efficiency gain" claim that appears in the reference example is absent from this survey. This survey makes no analogous quantitative claim about efficiency gains from any particular method or architecture. This is appropriate for a literature survey—it avoids exactly the kind of over-precise quantification that the evaluation section warns against.
Single-model-family issues do not apply (the survey covers many systems), but single-task-family issues do: almost all data-to-text examples come from weather, sports, or healthcare domains; almost all image captioning is evaluated on Flickr/COCO; almost all stylistic variation work is in dialogue or social media. The survey notes this limitation in passing (Section 8, calling for work on social media data and situated generation) but does not systematically analyse how domain-specific the reported findings might be.
The "generation gap" is a concept, not a measurement. The survey repeatedly invokes the generation gap (Meteer, 1991) as the central limitation of pipeline architectures, but no study is cited that directly measures the magnitude of the gap in a controlled comparison. The survey instead relies on indirect evidence—the revision-based architectures that mitigate it, the joint optimisation models that avoid it—to argue for its importance. This is a gap in the evidence base that the survey could have highlighted more explicitly.
The claim that "scalability remains an open challenge" (Section 8.4) is asserted but not quantified. What counts as scalable? The survey does not compare the size of the datasets used in research systems to the data volumes in commercial deployments (e.g., Arria-NLG generating reports from gas turbine sensor data, which presumably involves continuous high-volume streams). This makes the scalability challenge an abstract concern rather than a concrete, measured limitation.

6. Limitations and Trade-offs

The Single-Benchmark, Single-Model-Family Evidence Base

The assumption or constraint. The survey synthesises evidence almost exclusively from a narrow set of application domains and model families, without explicitly controlling for or analysing how these choices constrain the generality of the reported findings. The core examples that recur throughout Sections 3–7 are weather forecasting (SumTime, WeatherGov), sports reporting (RoboCup), restaurant recommendations (BAGEL, SF Hotel), image captioning (Flickr8k/30k, MS-COCO), and neonatal intensive care (BabyTalk). These domains share structural properties—short texts (typically 1–3 sentences), structured or semi-structured input with clear correspondence to output elements, and well-defined correctness criteria (factual accuracy can be verified against the data). The survey itself acknowledges this implicitly in Section 8.4, noting that "many of the data-to-text systems we have discussed have been developed within research environments, where the aim is of course to push the frontiers of NLG and demonstrate feasibility or correctness of novel approaches," and that "scalability remains an open challenge." However, the survey does not systematically analyse how domain properties might moderate the effectiveness of different architectural choices.

The consequence. A practitioner choosing an NLG architecture for a domain that falls outside the narrow set studied—e.g., long-form narrative generation, multi-paragraph analytical reports, generation from heterogeneous and noisy data streams (social media, IoT sensor networks), or tasks where correctness is ambiguous or multi-dimensional—has essentially no evidence from this survey about which approaches will work. The survey's meta-finding that evaluation metrics and human judgements diverge unpredictably across tasks (Section 7.4.1) is its own best evidence for domain dependence: if the correlation between BLEU and human fluency varies from 0.3 to 0.87 depending on the specific domain and metric configuration (Reiter & Belz, 2009), then the behaviour of NLG systems—and the validity of their evaluation—is clearly conditional on domain properties that the survey does not systematically characterise. Worse, the survey's reliance on PaLM-like models for neural NLG (Wen et al., 2015; Dušek & Jurčíček, 2016; Mei et al., 2016, all using LSTM-based architectures) and pre-transformer sequence models means that even within the studied domains, the architectural conclusions may not transfer to the post-2017 transformer era. The survey's coverage of neural methods in Section 3.3.5 was, by the authors' own characterisation, an emerging area that was "comparatively small (but steadily growing)" at the time of writing.

What evidence exists in the paper. The survey provides indirect evidence for domain sensitivity in several places. Section 7.4.1 documents that the correlation between automatic metrics and human judgements varies dramatically by domain: Reiter and Belz (2009) found BLEU correlated with clarity but not accuracy in weather reports; Wubben et al. (2012) found negative correlation between BLEU and adequacy in text simplification; Stent et al. (2005) found the opposite pattern in paraphrase generation (metrics tracked adequacy but not fluency). Section 4.3 notes that Elliott and De Vries (2015) found their image captioning system performed differently on VLT2K versus Pascal1K because one dataset contained more action-oriented images—a concrete example of dataset composition affecting system rankings. Section 7.4 documents that Kuznetsova et al. (2014) found BLEU ranked their best system higher than METEOR did, while other studies found METEOR more robust—demonstrating that even the "best metric" is conditional on the specific systems and data involved. The survey does not, however, conduct any formal analysis of how domain characteristics (text length, output variability, input structure, correctness criteria) moderate architectural effectiveness. No table or figure systematically compares system performance across domains for a fixed architecture, or across architectures for a fixed domain—the kind of evidence that would be needed to establish generalisability.

Mitigation status. Not addressed. The survey does not attempt to quantify generalisability or identify which findings are likely to be domain-specific and which are likely to be universal. The authors acknowledge the limitation in Section 8.4 as a call for future work—"scalability remains an open challenge" and "a greater concern with scalability would also focus researchers' attention on issues such as the time and resources required to collect data and train a system"—but they do not provide practitioners with concrete guidance on how to determine whether a given approach will transfer to their domain. The survey's structure, which treats weather forecasting, image captioning, dialogue, and creative generation in semi-autonomous sections, implicitly reinforces the domain-specificity of methods without providing the cross-domain analytical framework that would allow practitioners to reason about transfer.

The Difficulty Estimation Problem: No Cost Is Accounted for Complexity Assessment

The assumption or constraint. The survey's framework for understanding NLG architectures—in particular, its synthesis of data-driven approaches that jointly optimise content selection and realisation—relies on an implicit assumption that the appropriate architecture and level of integration can be selected without accounting for the cost of making that selection. This parallels, at the meta-level, the difficulty estimation problem in the reference example paper: just as that paper's compute-optimal policy required expensive difficulty estimation (2048 samples per prompt, Section 3.2 of the reference), the survey's recommendation that practitioners select architectures and evaluation methods adaptively based on domain and task characteristics requires a level of methodological expertise and empirical pre-testing that the survey does not cost. The survey provides an extensive taxonomy of architectures (Section 3), a catalogue of evaluation methods (Section 7), and comparative results across multiple studies, but it does not provide a decision procedure for mapping from a practitioner's constraints (domain, data availability, computational budget, latency requirements, need for interpretability) to a recommended approach. The survey assumes the reader can navigate this mapping themselves, without accounting for the cost of doing so.

The consequence. A practitioner who reads this survey faces what might be called a methodological overabundance problem: the survey documents dozens of architectures (modular pipeline variants, blackboard systems, revision-based architectures, planning through grammar, RL-based policies, log-linear classifiers, ILP joint optimisation, probabilistic CFGs, encoder-decoder neural models, conditioned language models) and at least as many evaluation paradigms (human fluency ratings, human adequacy ratings, preference comparisons, Magnitude Estimation, BLEU variants, METEOR, ROUGE, CIDEr, SPICE, edit distance, pyramid scoring, task-based extrinsic evaluation, reading-time measures). The survey provides evidence that these methods are not interchangeable—pipeline vs. joint optimisation produces different outputs (Lampouras & Androutsopoulos, 2013; Dethlefs & Cuayáhuitl, 2015); BLEU vs. METEOR vs. human judgements produce different system rankings (Section 7.4.1); domain properties moderate the effectiveness of image captioning architectures (Elliott & De Vries, 2015). But it provides no systematic way to choose among them without extensive empirical comparison in the target domain—which is exactly the expensive process that the survey is supposed to help practitioners avoid or at least streamline.

This is not a trivial cost. Conducting even a partial version of the multi-method evaluation that the survey recommends (Section 7.5: "err in favour of diversity, by using multiple methods, as far as possible") requires: (a) collecting or acquiring domain-specific data-text pairs (which, as Section 3.3.1 documents, often requires alignment or crowdsourcing), (b) implementing or adapting multiple system architectures, (c) running human evaluations with adequate sample sizes and inter-rater reliability checks, and (d) computing multiple automatic metrics with appropriate reference sets. For a practitioner building a production NLG system, this pre-deployment evaluation cost could easily exceed the cost of building any single system. The survey's failure to account for this meta-cost means its headline guidance—"use multiple evaluation methods"—is simultaneously correct (the evidence supports it) and potentially infeasible (the cost of following it is unspecified).

What evidence exists in the paper. The survey provides indirect evidence of this cost throughout. Section 3.3.1 documents the labour involved in acquiring aligned data-text pairs for statistical NLG—semi-supervised alignment (Liang et al., 2009) requires training an EM-based generative model; crowd-sourcing realisations (Mairesse & Young, 2014) requires designing a semantics transparent to non-experts and managing annotator quality; neither is a turnkey solution. Section 7.1.1 documents the complexity of human evaluation scale design, noting that preference-based comparisons are more sensitive than rating scales (Belz & Kow, 2010) but require different experimental designs; Magnitude Estimation avoids predefined scales but requires a modulus item and log-normalisation (Siddharthan & Katsos, 2012); inter-rater reliability is a recurring problem that may require iterative training of judges (Godwin & Piwek, 2016). Section 7.4.1 documents that the "best" automatic metric is task-dependent and system-dependent, meaning there is no shortcut metric that avoids human evaluation. The survey does not estimate the total cost of a comprehensive multi-method evaluation in any domain, nor does it provide a cost-benefit analysis of different evaluation strategies.

Mitigation status. Partially addressed through calls for future work and infrastructure. The survey advocates for shared tasks (Section 7) as a mechanism for amortising evaluation costs across the community, noting that "shared tasks have contributed to the development of both data sources and methods" and that recent proposals extend these to new domains (narrative generation, web data, meaning representation pairs). Shared tasks reduce the per-practitioner cost of evaluation by providing standardised datasets, metrics, and baselines. However, this mitigation only helps if the practitioner's domain matches a shared task domain—and the survey's own documentation of domain sensitivity suggests that off-the-shelf shared task results may not transfer. The survey also calls in Section 8.4 for "more investment in research on data acquisition and data-text alignment" and notes that "techniques for generation which rely on less precise alignments between data and text are also a promising future direction," acknowledging that data acquisition is a bottleneck. But the survey does not address the meta-level question of how to choose an architecture without full-scale empirical comparison—the equivalent of a cheap difficulty estimator that would make the compute-optimal policy in the reference example practically deployable.

The Generation Gap: Documented but Not Quantified or Solved

The assumption or constraint. The survey identifies the generation gap (Meteer, 1991) as the central limitation of modular pipeline architectures—"mismatches between strategic and tactical components, so that early decisions in the pipeline have unforeseen consequences further downstream" (Section 3.1). This concept is invoked throughout the survey as the primary motivation for alternatives: blackboard architectures (Section 3.1), revision-based architectures (Section 3.1), planning-based architectures (Section 3.2), and integrated stochastic approaches (Section 3.3.3) are all presented as solutions to the generation gap. However, the survey never quantifies the magnitude of the generation gap in any specific system or domain. It is treated as a conceptual failing of pipelines—a known problem—rather than as an empirically measured cost that varies across tasks and architectures. The survey assumes that the generation gap is real and consequential, and that architectures that avoid it (by integrating task decisions or allowing feedback) are therefore better, without establishing when the gap is large enough to justify the costs of the alternative architectures (reduced efficiency for blackboard/revision systems; increased computational cost for planning-based systems; reduced interpretability for end-to-end neural systems).

The consequence. The survey's architectural recommendations are built on an unquantified foundation. The survey documents (Section 3.1) that the pipeline architecture was the "consensus" in the field and that "many modular NLG systems" were successful in practice—SumTime weather forecasts (Reiter et al., 2005) were preferred by readers over human-written forecasts; BabyTalk summarised patient data with high clinical utility (Hunter et al., 2012); Midge generated image captions that outperformed competing systems in human evaluations (Mitchell et al., 2012). If these successful systems used pipeline architectures and the generation gap is a fundamental limitation, then the gap must either be small enough in practice to be tolerable, or it must be domain-dependent (large in some tasks, small in others), or it must be partially mitigated by careful design within the pipeline framework (e.g., choosing representations that minimise information loss between modules). But the survey provides no evidence about which of these explanations holds, making it impossible for a practitioner to determine whether the generation gap is a real problem for their specific application or a theoretical concern that will not manifest in practice.

The survey also documents (Section 3.3.3) that integrated architectures introduce their own problems: end-to-end neural models are opaque and can produce ungrammatical or hallucinated content; joint optimisation via ILP is computationally expensive; classifier cascades suffer from error propagation if tasks are ordered, but the best ordering is task-dependent (Zarrieß & Kuhn, 2013). These costs are real and documented. The benefits of avoiding the generation gap are hypothesised but not measured. This asymmetry means the survey provides stronger evidence for the costs of alternative architectures than for the benefits—a practitioner reading carefully would conclude that pipelines are flawed but the alternatives are unproven, which is not a useful guide to action.

What evidence exists in the paper. The survey provides qualitative examples and indirect evidence but no direct measurements. The example of the generation gap from Inui et al. (1992)—"a generation system might determine a particular sentence ordering during the sentence planning stage, but this might turn out to be ambiguous once sentences have actually been realised and orthography has been inserted" (Section 3.1)—is a hypothetical illustration, not an empirical finding. The evidence that revision-based architectures improve over pipelines comes from Zarrieß and Kuhn (2013), who found that a revision-based architecture (syntax mapping → REG insertion → syntax revision) outperformed a sequential pipeline on their specific task, but the magnitude of improvement and its statistical reliability are not reported. The evidence that joint optimisation via ILP outperforms pipelines comes from Lampouras and Androutsopoulos (2013), who showed that joint optimisation of content selection, lexicalisation, and aggregation "produces more compact verbalisations," but compactness is only one dimension of quality. The evidence that joint RL policies outperform isolated policies comes from Dethlefs and Cuayáhuitl (2015), who found less verbose utterances and shorter interactions—but this is a specific dialogue task, and the survey does not establish how general this finding is.

Critically, the survey does not cite any study that directly compares a pipeline architecture to an integrated architecture while holding the underlying methodology constant (e.g., a statistical pipeline vs. a statistical joint model, or a neural pipeline with separate content and realisation modules vs. an end-to-end neural model) and measuring the contribution of the architectural choice alone. The comparisons that exist confound architecture with methodology (the pipeline baseline is often rule-based while the integrated alternative is data-driven), making it impossible to attribute performance differences to architectural integration per se.

Mitigation status. Not addressed. The survey treats the generation gap as a settled conceptual problem that motivates architectural innovation, rather than as an empirical phenomenon whose magnitude and boundary conditions need to be characterised. No future work is proposed to measure the generation gap systematically. The closest the survey comes to acknowledging this limitation is in Section 8.3, where it discusses the theory/engineering dichotomy and notes that "few NLG systems seek to model production errors explicitly"—but this is framed as a missed opportunity for theoretical insight, not as a methodological gap in the evidence base for architectural recommendations.

No Account of Latency or Serial Dependency Costs

The assumption or constraint. Throughout its discussion of architectures—particularly in comparing modular pipelines to planning-based and stochastic integrated approaches (Section 3), and in discussing sequential vs. parallel generation strategies for revisions and stylistic variation (Sections 5–6)—the survey treats computational cost exclusively in terms of offline factors: training data requirements, rule authoring effort, development time, and asymptotic algorithmic complexity. It does not discuss latency (wall-clock time to produce output), throughput (how many outputs can be generated per unit time in a production setting), or the serial dependency structure of different architectures—whether generation decisions can be parallelised or must proceed sequentially. This is a significant omission because the architectures the survey compares have fundamentally different latency profiles, and the survey's own analysis of stochastic approaches (Section 3.3) shows the field moving toward models whose training is expensive but whose inference is fast (feedforward neural networks, LSTMs), or models that introduce sequential bottlenecks at inference time (beam search, revision chains, planning-based search).

The consequence. The survey cannot provide guidance on an architect's most basic deployment question: how fast will this system generate text in production? This matters concretely for several of the survey's own motivating applications:

Robo-journalism (Section 1; the LA Times earthquake report was published within 3 minutes of the event): the system must generate text faster than the news cycle, which for breaking news means seconds to minutes. A planning-based architecture that spends significant time on preprocessing (Koller & Petrick, 2011, noting this as a limitation) may miss the publication window even if it produces higher-quality text.
Clinical decision support (BabyTalk, Section 7.2): nursing shift summaries are generated from 12 hours of continuous patient data. The generation system must produce output before the next clinical decision point. A stochastic realiser with expensive decoding (e.g., beam search over a hypergraph, as in Konstas & Lapata, 2013) may take minutes to generate a multi-sentence summary, which is acceptable—but the survey provides no latency data to confirm this.
Dialogue systems (Sections 3.2.2, 3.3.6): interactive response generation has hard latency constraints measured in seconds or less. The survey discusses seq2seq models for dialogue (Wen et al., 2015; Sordoni et al., 2015) but mentions only that they use "a reranker during decoding to rank beam search outputs" (Dušek & Jurčíček, 2016) without discussing how beam search depth affects response time.
Stylistic variation through sequential revision (Section 5): generating personality-conditioned output by conditioning an LSTM on attribute vectors (Hu et al., 2017; Ficler & Goldberg, 2017) is feedforward at inference time—latency scales linearly with output length. But generating varied output through iterative revision (analogous to the revision model in the reference example) introduces sequential dependencies that multiply latency by the number of revision steps.

The survey's architectural comparison framework—modular vs. planning-based vs. stochastic integrated—is fundamentally about the organisation of the generation process, but the process's latency implications are invisible in this taxonomy. A pipeline that serialises content selection, sentence planning, and realisation has a latency that is the sum of each module's time. A blackboard architecture where modules fire reactively may have lower latency for simple inputs but higher variability. An end-to-end neural model with greedy decoding has latency proportional to output length; with beam search, latency is multiplied by beam width; with iterative refinement, latency is multiplied by the number of refinement steps. These differences are large enough to dominate deployment decisions, yet the survey provides no framework for reasoning about them.

What evidence exists in the paper. Almost none. The survey makes one passing reference to latency: in Section 3.2, discussing planning-based approaches, it notes that Koller and Petrick (2011) found that "planners tend to spend significant amounts of time on preprocessing, though solutions could often be found efficiently once preprocessing was complete"—but no specific times are given, and the finding is not connected to the broader architectural comparison. Section 3.1 notes that blackboard and revision-based architectures incur a "reduction in efficiency" (citing De Smedt et al., 1996), but "efficiency" here refers to computational overhead, not wall-clock latency, and the magnitude is not quantified. For neural approaches (Section 3.3.5), the survey mentions that "advances in hardware that can support resource-intensive learning problems" have been a key enabler, but this refers to training time, not inference latency. The survey does not report inference time, parameters, FLOPs, or throughput for any of the systems it discusses.

Mitigation status. Not addressed. The survey does not identify latency as a dimension of architectural comparison, does not call for standardised reporting of inference time or throughput in NLG evaluations, and does not suggest that future work should treat latency as a first-class constraint in system design. This is a gap both in the survey's analytical framework and in its recommendations for future research. The survey's own advice in Section 8.4—that greater attention to "the time and resources required to collect data and train a system and the efficiency of the algorithms being deployed" would benefit the field—is focused on training efficiency, not inference latency, suggesting the authors did not consider this dimension.

The Revision and Creativity Training Data Fragility Problem

The assumption or constraint. The survey's discussions of stylistic variation (Section 5) and creative language generation (Section 6) document a pattern that the survey itself does not fully synthesise: these approaches are highly sensitive to the specific training data construction procedure, and small changes in how data is collected or paired can lead to substantial performance degradation or even complete failure. In the terminology of the reference example, these approaches exhibit a training data fragility problem analogous to the verification over-optimisation that limits search-based methods: the method works well when the training data closely matches the deployment distribution, but performance collapses under distribution shift. The survey documents this pattern across multiple sub-areas without explicitly identifying it as a common-mode failure:

Personality-conditioned generation: Mairesse and Walker (2011) found that "readers vary significantly in their judgements of what personality is actually reflected by a given text" (Section 5.1), and Walker et al. (2011b) found that "modelling characters' style directly using corpora of utterances results in more specific and easily perceived traits than using a model based on personality traits" (Section 5.1). The fragility here is in the mapping from the conditioning variable (Big Five traits) to the linguistic features that realise it: the same trait scores can produce different perceived personalities depending on the reader, and training on trait-labelled data produces less reliable style control than training directly on stylistic exemplars.
Metaphor generation: Veale and Hao's (2007, 2008) Sardonicus system relies on web-mined associations between nouns and properties (e.g., stick associated with skinniness). The filtering step rules out inappropriate nouns like cadaver—but this filtering is itself rule-based and domain-specific. The survey does not discuss what happens when the web-mined associations are noisy or culturally specific, or when the target domain contains concepts for which web co-occurrence data is sparse.
Joke generation: Petrović and Matthews (2013) operationalise funniness through three quantifiable principles (attribute ambiguity, noun dissimilarity, attribute applicability to both nouns), but their best model "was rated as funny in 16% of cases, compared to 33% of the human jokes" (Section 7.1.3). This is a 2× performance gap from the human baseline, and the survey does not analyse whether this gap is due to limitations of the three principles (i.e., more principles are needed) or to fundamental limits of unsupervised generation from n-gram statistics (i.e., the approach cannot exceed this ceiling regardless of how many principles are operationalised).
Revision models for stylistic control: While the survey was written before the specific revision model experiments in the reference example, Section 6's discussion of narrative generation identifies a structurally similar fragility: if story planning is emphasised over linguistic realisation (as in TaleSpin, Figure 7a), "the perceived quality of the narrative will be compromised." Conversely, if realisation is emphasised over planning, the story will be fluent but incoherent. The fragility is in the balance between components that cannot be independently optimised.

The consequence. For a practitioner seeking to deploy creative or stylistically varied NLG, the survey provides evidence that these approaches can work—there are working systems that generate recognisable jokes, interpretable metaphors, coherent stories, and personality-conditioned text—but it provides no guidance on the boundary conditions for this success. When does the training data need to be domain-specific vs. when do web-mined or crowd-sourced resources suffice? When does the conditioning variable (personality trait, emotion label, character identity) need to be directly reflected in linguistic features vs. when can it be an intermediate latent variable? When is the performance ceiling (16% joke recognition rate, inconsistent personality perception) acceptable, and when is it prohibitive? The survey's evaluative framework (Section 7) provides methods for measuring these properties post-hoc but no predictive guidance for anticipating them during system design.

The common-mode nature of this fragility is particularly consequential because it suggests a structural limitation of data-driven approaches to NLG subtleties. Tasks like content determination and surface realisation have clear correctness criteria (does the output match the data? is it grammatical?) that can be operationalised in loss functions and automatic metrics, even if imperfectly. Tasks like style, personality, creativity, and affect have no clear correctness criterion—there is no ground truth for whether a metaphor is apt, a joke is funny, or a personality is accurately conveyed. The system's output is evaluated by human judges whose ratings have high variance (the survey documents this repeatedly in Section 7.1.1), and the mapping from input conditioning to output realisation is both noisy and reader-dependent. This means that the standard NLG development cycle—train on aligned data, evaluate with automatic metrics, validate with human judgements—breaks down for these tasks in a way the survey documents but does not analyse.

What evidence exists in the paper. The evidence is scattered across Sections 5, 6, and 7:

Section 5.1: Mairesse and Walker (2011) found inconsistent personality perception (readers vary significantly); Walker et al. (2011b) found that direct stylistic modelling outperforms trait-based modelling.
Section 5.2: van der Sluis and Mellish (2010) found that the extent to which "affective tactical decisions influence hearer's emotional states is dependent on a host of other factors, including the degree to which the reader is directly implicated in what the text says." This is a domain-sensitivity finding: affect induction works only when the text has personal relevance to the reader.
Section 6.1: Petrović and Matthews (2013) achieved 16% joke recognition rate vs. 33% for human jokes—a 2× gap.
Section 6.3: McIntyre and Lapata (2009) trained an interest model to predict which stories would be rated as interesting—a direct attempt to operationalise a subjective quality—but the survey does not report the model's accuracy or whether it generalised beyond the training domain.
Section 7.1.3: Binsted et al. (1997) found that children recognised computer-generated jokes as jokes and rated them funnier than non-jokes, but rated human-produced jokes as funnier overall—demonstrating both that genre compatibility can be achieved (a success) and that quality remains sub-human (a limitation).

The survey reports these findings but does not synthesise them into a systematic analysis of training data fragility. Each finding is presented as a domain-specific result rather than as evidence for a broader pattern.

Mitigation status. Not addressed. The survey does not identify training data fragility as a cross-cutting limitation of creative and stylistic NLG, does not propose systematic approaches to measuring or mitigating it (e.g., adversarial validation of style conditioning, multi-dimensional human evaluation protocols designed to capture subjective qualities with acceptable reliability), and does not call for future work on this specific problem. The discussion in Section 8 of the survey proposes future directions—social media generation, situated language generation, generation from knowledge bases—without addressing the common-mode fragility that the survey's own evidence reveals. This is a missed opportunity to provide a unified diagnosis that could guide research prioritisation across multiple sub-areas.

The Evaluation Methodology Paradox: The Survey Recommends What It Shows Is Infeasible

The assumption or constraint. The survey's central recommendation on evaluation—articulated most forcefully in Section 7.5—is that NLG practitioners should "err in favour of diversity, by using multiple methods, as far as possible, and reporting not only their results, but also the correlation between them." The justification is the survey's own meta-analysis: because automatic metrics and human judgements "frequently do not correlate" (Section 7.4.1), because the strength and even direction of correlation varies by domain and metric configuration, and because different evaluation methods "focus on different aspects of a system or its output" (Section 7.5), no single method can be trusted in isolation. The implication is that a responsible evaluation requires at minimum: (a) at least one automatic metric (preferably several, to assess metric sensitivity), (b) human judgements on at least fluency and adequacy (ideally also task-specific dimensions like style appropriateness or creativity), and (c) ideally an extrinsic, task-based evaluation that measures whether the generated text achieves its communicative purpose. This recommendation is internally consistent with the survey's evidence—but it is also prohibitively expensive, and the survey itself documents the practical barriers that make it infeasible for most practitioners.

The consequence. The survey places the field in a methodological bind. Its own analysis demonstrates that:

Single-metric evaluation is unreliable (Section 7.4.1): the same system can appear superior or inferior depending on whether BLEU, METEOR, or human judgements are used. Using only one method produces results that are method-dependent, not just system-dependent.
Multi-method evaluation is financially and logistically prohibitive for any but the best-resourced labs. Human evaluations—even through crowd-sourcing platforms—require careful experimental design (Section 7.1.1 discusses scale choice, inter-rater reliability, judge training, and the ethical concerns of underpaid crowd workers). Task-based extrinsic evaluations require recruiting domain experts (e.g., clinicians for BabyTalk; Hunter et al., 2012) or building realistic task environments (e.g., the GIVE virtual environment; Striegnitz et al., 2011). Computing multiple automatic metrics requires multiple reference texts, which most NLG datasets lack (Section 7.4.1, factor 4). The survey provides no cost estimates, but the described protocols imply weeks to months of additional work beyond system development.
Even multi-method evaluation does not guarantee convergence, so the practitioner may end up with multiple, contradictory quality signals and no principled way to resolve them. Section 7.4.1 documents cases where metrics and human judgements disagreed, where different metrics disagreed, and where human judgements on different quality dimensions (fluency vs. adequacy) were predicted by different metrics. The survey's advice—"weak correlations need not imply that the results of a particular method are invalid" (Section 7.5)—is epistemologically reasonable but practically unhelpful: if methods disagree, which one should the practitioner trust when deciding whether to deploy the system?

The result is a paradox: the survey demonstrates that thorough evaluation requires multiple methods, and simultaneously demonstrates that multi-method evaluation is expensive, unreliable, and may not produce actionable conclusions. A practitioner who follows the survey's advice literally will spend more on evaluation than on system development and may still not know whether their system is any good. A practitioner who ignores the advice will use a single method (likely BLEU, given its prevalence) and produce results that the survey's own evidence shows are unreliable. Neither path is satisfactory, and the survey provides no resolution to this paradox.

What evidence exists in the paper. The evidence for the paradox is the survey's own meta-analysis in Section 7, particularly 7.4.1. The key findings that establish the bind:

Correlation between BLEU and human judgements varies from 0.3 to 0.87 depending on domain and metric configuration (Reiter & Belz, 2009). This range spans from "essentially useless" to "quite good," meaning a practitioner cannot assume BLEU will work without validating it against human judgements in their specific domain—which defeats the purpose of using an automatic metric as a cheap substitute for human evaluation.
Even within a single domain, BLEU ranked systems differently than human judgements: Kulkarni et al. (2013) found their system lost on BLEU but won on human preference; Hodosh et al. (2013) found BLEU agreed with humans only when captions were identical to references. This means BLEU is unreliable specifically in the cases where it matters most—when systems are close in quality and the ranking determines which one gets deployed.
Metrics that were validated in one study failed to generalise: Kuznetsova et al. (2014) found BLEU ranked their best system higher than METEOR, contradicting Elliott and Keller's (2013) finding that METEOR was more robust. This means the "best metric" is conditional on the specific systems being compared.
The MS-COCO Evaluation Challenge found that some systems "outperformed a human-human upper bound" on automatic metrics but no system reached this level on human judgements (Section 7.4.1, citing Bernardi et al., 2016). This is a direct demonstration that metrics can indicate superhuman performance while humans disagree—the metric is measuring something, but it is not measuring what humans perceive as quality.

The survey also provides evidence for the cost of human evaluation: Section 7.1.1 discusses the need for careful scale design (discrete vs. continuous, absolute vs. relative), the problem of inter-rater reliability requiring either large sample sizes or iterative judge training (Godwin & Piwek, 2016), and the ethical concerns of crowd-sourcing platforms (Fort et al., 2011). Section 7.2 documents the logistical complexity of extrinsic evaluations: BabyTalk could not run a task-based evaluation on-ward due to ethical constraints (Hunter et al., 2012); the GIVE Challenge required building an entire virtual environment and recruiting online participants (Striegnitz et al., 2011). None of these are turnkey processes.

Mitigation status. Partially addressed through the survey's advocacy for shared tasks. Section 7 notes that "shared tasks have contributed to the development of both data sources and methods" by providing standardised datasets, metrics, and baselines. By centralising evaluation, shared tasks amortise the cost across the community: instead of every practitioner running their own multi-method evaluation, the community collectively maintains a benchmark with standard metrics and periodic human evaluation campaigns. This is a genuine mitigation—but it only works for practitioners whose systems operate on the shared task domains. The survey's own documentation of domain sensitivity (see Limitation 1 above) implies that shared task results may not transfer to new domains, meaning practitioners working outside the established benchmarks are back in the methodological bind.

The survey does not propose intermediate solutions that would reduce the cost of multi-method evaluation without requiring full shared task infrastructure: lightweight human evaluation protocols (e.g., using a small number of expert judges rather than large crowd-sourced samples), automatic metrics that are validated as reliable proxies for human judgements in specific domains (so that practitioners in those domains can use metrics alone), or decision-theoretic frameworks for determining when the cost of additional evaluation methods is justified by the expected improvement in deployment decisions. The paradox—that the survey recommends a practice its own evidence shows is both necessary and infeasible—remains unresolved.

7. Implications and Future Directions

How This Work Changes the Landscape

This survey reshapes the NLG field not by proposing a new algorithm or architecture, but by providing a comprehensive diagnostic of fragmentation — documenting that the field's diversity of tasks, architectures, and evaluation methods has produced an evidence base where core questions cannot be answered because the evidence does not generalise across domains, and where the evaluation tools the community relies on do not agree with each other. The impact is both conceptual and methodological.

Conceptually, the survey establishes three orthogonal axes — task, architecture, and method — as a taxonomy that decouples what had been conflated. Prior to this survey, the implicit assumption in much NLG discourse was that modular pipelines were rule-based, that integrated architectures were data-driven, and that the choice between them was a choice between paradigms. The survey demonstrates that these dimensions are independent: modular systems can be fully statistical (the classifier cascades of Marciniak & Strube, 2004, which maintain crisp task boundaries while using instance-based learning for each decision) and integrated systems can be fully symbolic (the grammar-as-planning approach of Koller & Stone, 2007, which unifies content determination, REG, and realisation under a single planning formalism without any statistical component). This reframing matters because it reveals that the field's central debate — pipeline vs. end-to-end — was miscast. The real question is not whether task boundaries should exist, but which tasks benefit from joint optimisation and under what conditions. The survey provides the vocabulary for asking that question precisely, and its documentation of contradictory results across domains (Section 7.4.1) is the strongest evidence that the answer is domain-dependent.

Methodologically, the survey establishes the incommensurability of evaluation methods as a structural feature of NLG, not a temporary inconvenience. The meta-analysis in Section 7 does not merely report that BLEU and human judgements sometimes disagree — it synthesises evidence from weather forecasting (Reiter & Belz, 2009), referring expression generation (Gatt & Belz, 2010), image captioning (Kulkarni et al., 2013; Hodosh et al., 2013; Bernardi et al., 2016), text simplification (Wubben et al., 2012), paraphrase generation (Stent et al., 2005), and surface realisation (Cahill, 2009; Espinosa et al., 2010) to show that the strength and even direction of metric-human correlation varies unpredictably by domain, metric configuration, and which specific systems are being compared. The diagnostic is fourfold: (1) metric sensitivity to text length (NLG outputs are shorter than MT outputs, depressing BLEU), (2) the type of n-gram overlap (higher-order n-grams capture syntax but are exponentially harder to match), (3) semantic variability uncaptured by surface matching, and (4) the paucity of reference texts. These are not measurement errors that better metrics will fix — they are structural features of NLG output that make surface-overlap metrics intrinsically limited as proxies for human-perceived quality. The practical consequence is that any NLG evaluation that relies on a single metric — or even on metrics alone — is measuring something, but it is not measuring what readers perceive.

The survey reconciles several contradictions that had been latent in the literature. The most prominent is the pipeline-vs-integrated debate, which the survey reframes as a false dichotomy by documenting successful systems in both families and showing that integration is a matter of degree (the blackboard and revision-based architectures of Section 3.1 are intermediate points, not endpoints). A second is the contradiction between studies reporting that self-correction and stylistic control work (Mairesse & Walker, 2010; Paiva & Evans, 2005) and those reporting that they fail or produce inconsistent results (the personality perception variability documented in Section 5.1; the domain-sensitivity of affective NLG documented in van der Sluis & Mellish, 2010). The survey's taxonomy implicitly explains this: these techniques are sensitive to the specific mapping between conditioning variables and linguistic features, and the mapping's reliability is domain- and reader-dependent — a finding that would not be visible without the survey's cross-domain synthesis.

The survey redirects research attention toward neglected problems and away from overemphasised ones. By documenting that the generation gap (Meteer, 1991) is the central limitation of pipeline architectures but has never been directly measured (only inferred from indirect evidence like the superiority of revision-based architectures in Zarrieß & Kuhn, 2013), the survey makes the case that understanding when information bottlenecks matter — and when they do not — is more important than developing yet another integrated architecture. By documenting that evaluation metrics are unreliable across the board, the survey redirects effort from metric development (the approach of borrowing MT metrics like BLEU and tweaking them) toward evaluation design — how to combine multiple methods in cost-effective ways that provide actionable quality signals. By documenting the convergence of data-to-text and text-to-text techniques (Sections 1.1, 3.3, 8.2), the survey makes the case that maintaining separate research communities for these sub-areas is counterproductive, since the same architectures (encoder-decoder models, joint optimisation frameworks) and the same evaluation challenges (metric-human divergence) apply to both.

Research directions that become more attractive:

Controlled architectural comparisons that vary integration while holding methodology constant (e.g., a statistical pipeline vs. a statistical joint model on the same data, with the same underlying classifier or language model), to isolate the contribution of integration per se.
Difficulty estimation and domain characterisation — what properties of an NLG task (text length, input structure, output variability, correctness criteria) predict which architectures will work? This is analogous to the difficulty-conditioned allocation in the reference example paper, but applied at the meta-level to architecture selection.
Cost-benefit analysis of evaluation methods — given the survey's demonstration that multi-method evaluation is necessary but expensive, what is the minimum evaluation protocol that provides reliable quality signals for a given domain and budget?
Cross-domain transfer — can architectures or evaluation protocols validated in one domain (e.g., weather forecasting) be transferred to another (e.g., clinical reporting) without full re-validation? The survey's domain-sensitivity findings suggest not, but this is itself a hypothesis that needs systematic testing.

Research directions that become less attractive:

Developing yet another automatic metric without firstcharacterising the structural limitations that the survey identifies (text length sensitivity, semantic variability, reference set size). A new metric that does not address these limitations will fail for the same reasons BLEU fails.
Building integrated architectures solely on the argument that pipelines suffer from the generation gap, without measuring the gap's magnitude in the target domain. The survey shows that pipeline systems have been successful in practice (SumTime, BabyTalk, Midge), meaning the generation gap is not universally prohibitive.
Running single-metric evaluations and claiming system quality — the survey's meta-analysis makes this practice scientifically indefensible, and any paper that does so after this survey must contend with the evidence that single-metric rankings are unreliable.

Follow-Up Research This Work Enables

Controlled architectural ablation: pipeline vs. joint optimisation holding methodology constant. The survey documents that no existing study directly compares a modular pipeline to an integrated architecture while using the same underlying statistical or neural machinery. A researcher could take a single domain with aligned data-text pairs (e.g., WeatherGov, where database records map to forecast sentences), implement both a pipeline version (separate classifiers for content selection, sentence planning, and realisation, trained independently on the same data) and a joint version (a single model that maps records to text, either via ILP joint optimisation or an end-to-end neural model), and measure not just output quality (BLEU + human fluency + human adequacy) but also the magnitude of the generation gap — operationalised as the frequency with which a correct upstream decision (e.g., selecting the right database record) leads to a downstream error (e.g., incorrect realisation) in the pipeline condition. The survey predicts that the gap's magnitude will be domain-dependent, and this experiment would provide the first direct measurement. A negative result — the gap is uniformly small across domains — would reframe the architectural debate by suggesting that pipeline's disadvantages are overstated. A positive result with domain variation would enable the kind of difficulty-conditioned architecture selection that the reference example paper achieves for test-time compute.

Cheap difficulty estimation for NLG domain characterisation. The survey's meta-finding that evaluation metrics and architectural effectiveness vary by domain implies that practitioners need a way to characterise their domain before choosing an architecture — but the survey provides only qualitative descriptions of domain properties (text length, input structure, output variability). A researcher could operationalise these properties across the domains studied in the survey (weather forecasts, sportscasting, restaurant recommendations, image captions, clinical reports, narratives) and train a classifier to predict, from domain-level features, which evaluation metric will best correlate with human judgements and which architectural family will perform best. The features could include: average output length in words, vocabulary size, type-token ratio, input dimensionality, alignment strength (the mutual information between input records and output n-grams), and the number of available reference texts. The ground truth would be the human evaluation results already reported in the individual studies the survey cites. If such a classifier works, it would directly address the "methodological overabundance" limitation identified in Section 6 — practitioners could characterise their domain once and look up the recommended architecture and evaluation strategy. If it fails, that would demonstrate that domain properties are insufficient predictors and that architectural choice must be determined empirically in each case — a negative result with significant practical implications.

Systematic multi-method evaluation on a fixed benchmark with cost accounting. The survey's central paradox — that multi-method evaluation is necessary but expensive, and that its cost is never quantified — could be addressed by a study that runs the full evaluation protocol the survey recommends (multiple automatic metrics, human fluency and adequacy judgements, a task-based extrinsic evaluation) on a fixed set of systems (e.g., three architectures for weather forecast generation: a template baseline, a statistical pipeline, and an end-to-end neural model) while carefully tracking the time, money, and personnel required for each evaluation component. The output would be: (a) a correlation matrix showing how each metric and human judgement dimension relates to every other, replicating and extending the survey's meta-analysis but with controlled system conditions and known cost; (b) a cost-effectiveness analysis showing which evaluation methods provide the most information per unit cost; (c) concrete guidance on the minimum evaluation protocol for different budget levels. The survey's Table 1 and the cost documentation in Section 7.1.1 (scale design, inter-rater reliability, ethical concerns) provide the framework for designing this study. The key measurement is not just whether methods correlate, but what the correlation costs to obtain.

Cross-lingual and cross-domain evaluation of the metric-human correlation finding. The survey's evaluation meta-analysis is almost entirely based on English-language NLG, with a single exception (Cahill, 2009, reporting weak metric-human correlation for German surface realisation). A researcher could replicate the Reiter and Belz (2009) meta-evaluation protocol — computing correlations between multiple metrics (BLEU, METEOR, ROUGE, CIDEr) and human fluency/adequacy judgements — across NLG systems in multiple languages and multiple domains within each language. The survey predicts that the correlation structure will vary by language (German's freer word order should reduce surface-overlap metrics' reliability compared to English; morphologically rich languages should show different patterns than isolating ones) and by domain (the survey already documents this within English). This replication would test whether the survey's central diagnostic — that metric-human divergence is structural, not a measurement error — generalises beyond English, and would identify language-specific metric configurations that improve reliability. A systematic failure to find any language-metric combination where correlation exceeds 0.7 would strengthen the survey's argument that surface-overlap metrics are fundamentally limited, and would motivate investment in semantic evaluation methods (SPICE, embedding-based metrics) as a universal solution rather than language-specific metric tuning.

Targeted measurement of the generation gap across task types. The survey identifies the generation gap as the central limitation of pipeline architectures but notes that its magnitude has never been directly measured. A researcher could construct a controlled experimental pipeline where each module's input and output are recorded, then measure the gap as the frequency with which a downstream module produces an error that is directly attributable to an upstream module's correct-but-misleading output. For example, in a weather forecast pipeline: the content selector correctly chooses to report wind direction; the sentence planner correctly aggregates wind direction and wind speed into a single sentence plan; but the realiser produces "south wind around 20mph" when the sentence plan specified mention of wind direction before wind speed — a word ordering error that is only an error because of how the content was structured. The key measurement is the proportion of output errors that stem from upstream-downstream mismatches vs. from errors within individual modules. This could be done across multiple domains (weather, sports, clinical) using existing modular systems. If the gap is consistently small (under 10% of errors), the case for joint optimisation is weakened. If it is large but domain-dependent, the survey's taxonomic framework can be enriched with gap-magnitude annotations, enabling practitioners to estimate the gap's cost in their domain from domain features.

Training data fragility as a common-mode failure in creative and stylistic NLG. The survey documents scattered evidence that creative and stylistic NLG systems are highly sensitive to their training data construction — personality perception varies with reader (Mairesse & Walker, 2011, Section 5.1), direct stylistic modelling outperforms trait-based modelling (Walker et al., 2011b, Section 5.1), joke recognition rates are 2× below human baselines (Petrović & Matthews, 2013, Section 7.1.3), affective text impact depends on reader involvement (van der Sluis & Mellish, 2010, Section 5.2) — but it does not synthesise these into a systematic diagnosis. A researcher could take one creative NLG task (e.g., metaphor generation) and systematically vary the training data construction procedure: web-mined associations (the Veale & Hao, 2007, approach), crowd-sourced metaphor judgements, expert-annotated metaphor corpora, and a control using only dictionary definitions. For each condition, measure output quality via human judgements of aptness, novelty, and interpretability, and compute the correlation between these dimensions. The survey predicts that web-mining will produce novel but sometimes inappropriate metaphors (high novelty, moderate aptness, risk of failures like the cadaver example), while expert corpora will produce apt but conservative metaphors. The practical output would be a characterisation of the novelty-aptness trade-off as a function of training data, and guidance on when each data source is appropriate. A finding of no significant difference across data sources would refute the fragility hypothesis and suggest that creative NLG is more robust than the survey's scattered evidence implies.

Practical Applications and Downstream Use Cases

Principled NLG system selection for commercial deployment. An organisation choosing an NLG architecture for a production system (e.g., generating financial reports from quarterly earnings data, or producing personalised health summaries from wearable sensor data) currently faces the survey's "methodological overabundance" problem: dozens of architectures and evaluation methods, with evidence that they are not interchangeable but no systematic guidance on how to choose. The survey's taxonomy and meta-analysis enable a structured selection process: (1) characterise the domain along the survey's implicit axes — text length (short bulletins vs. multi-paragraph reports), input structure (clean database records vs. noisy sensor streams), output variability requirements (fixed templates acceptable vs. stylistic variation needed), and correctness criteria (factual accuracy verifiable vs. subjective quality dimensions); (2) use the survey's documentation of which architectures have succeeded on similar domains (weather forecasting → statistical pipelines or neural seq2seq; dialogue → RL-based policies; image captioning → encoder-decoder with attention; creative text → template-based with lexical resources or retrieval-based methods) as a prior; (3) run a minimal multi-method evaluation — at minimum, one surface-overlap metric plus human fluency and adequacy on a small sample — rather than a single metric, following the survey's explicit recommendation that "weak correlations need not imply that the results of a particular method are invalid. Rather, they may indicate that measures focus on different aspects" (Section 7.5). The benefit is not a guaranteed optimal choice — the survey does not provide that — but a defensible, evidence-informed process that avoids the known failure mode of trusting a single metric that the survey shows is unreliable.

Shared task design informed by metric-human divergence patterns. Organisers of NLG shared tasks — which the survey identifies as the community's primary mechanism for amortising evaluation costs — can use the survey's meta-analysis to design more robust evaluation protocols. The survey's finding that metric-human correlation varies from 0.3 to 0.87 depending on metric configuration (Reiter & Belz, 2009, Section 7.4.1) implies that shared tasks should report system rankings using multiple metrics (at minimum, one n-gram overlap metric like BLEU and one semantic or embedding-based metric like METEOR or WMD) alongside human judgements, and should explicitly report the correlation between them. The finding that BLEU-1 and BLEU-4 capture different aspects of quality (unigram overlap vs. syntactic structure, Section 7.4.1) implies that tasks with short outputs (like referring expression generation or single-sentence generation) should prioritise METEOR or embedding-based metrics over BLEU, since shorter texts amplify BLEU's length-sensitivity problem. The finding that metrics can indicate superhuman performance while humans disagree (the MS-COCO Challenge result cited in Section 7.4.1) implies that any shared task reporting automatic metric scores should include a human baseline to detect this dissociation. These are concrete, low-cost changes to existing shared task protocols that would increase the reliability and interpretability of results.

Cost-effective human evaluation protocols for low-resource NLG deployment. The survey's documentation of human evaluation complexity — scale design (discrete vs. continuous, absolute vs. relative; Section 7.1.1), inter-rater reliability (high variance in Question Generation, requiring iterative judge training; Godwin & Piwek, 2016), and ethical concerns with crowd-sourcing platforms (Fort et al., 2011) — provides a checklist of pitfalls that a practitioner can use to design a minimal human evaluation that avoids the worst failures. The survey's finding that preference-based comparisons are "more sensitive to differences between systems, and less susceptible to variance between subjects" than rating scales (Belz & Kow, 2010, Section 7.1.1) implies that a practitioner with a limited budget for human evaluation should use paired comparisons (is system A better than system B on fluency?) rather than Likert-scale ratings (rate system A's fluency from 1–5), since the former requires fewer judges to achieve acceptable statistical power. The survey's documentation that reading-time measures can detect quality differences that offline judgements miss (Zarrieß et al., 2015, Section 7.4.2) implies that for applications where text comprehension is critical (clinical decision support, educational materials), a small behavioural study (N ≈ 20–30 participants) measuring reading time and comprehension accuracy may provide more actionable quality signals than a large crowd-sourced rating study at comparable cost. These are direct operationalisations of the survey's evidence that practitioners can adopt without designing an evaluation from scratch.

When to Prefer This Method

The survey does not propose a single method for which a clear "prefer when" decision rule against named alternatives can be formulated — it is a taxonomic synthesis rather than a technical contribution. It does, however, articulate an implicit methodological preference that can be extracted and stated explicitly: prefer multi-method evaluation over single-metric evaluation when assessing any NLG system. The conditions under which this preference applies are documented throughout Section 7:

Prefer multi-method evaluation when the domain has short outputs (e.g., referring expressions, single-sentence weather forecasts, dialogue responses), because the survey documents that surface-overlap metrics like BLEU are systematically depressed for short texts, making single-metric evaluation especially unreliable (Section 7.4.1, factor 1).
Prefer multi-method evaluation when the system's output varies stylistically or creatively (e.g., personality-conditioned generation, metaphor generation, narrative), because the survey documents that the mapping from conditioning variables to perceived quality is reader-dependent and inconsistent (Sections 5.1, 5.2, 7.1.3), meaning human judgements — and the variance among them — are essential quality signals that no automatic metric captures.
Prefer multi-method evaluation when only one or few reference texts are available, because the survey documents that most NLG domains lack the multiple references that MT metrics were designed for, and that metrics agree with human judgements only when generated text is identical to the reference (Hodosh et al., 2013, Section 7.4.1), making single-metric evaluation with sparse references effectively a string-matching test, not a quality test.
Prefer multi-method evaluation when factual adequacy to the input is as important as fluency, because the survey documents that metrics often track fluency but fail to track adequacy (weather forecasting: Reiter & Belz, 2009; text simplification: Wubben et al., 2012), and the correlation structure reverses across domains (paraphrase generation: Stent et al., 2005, finding the opposite pattern). A single metric cannot be trusted to capture both dimensions simultaneously.
The exception — where single-method evaluation may be adequate — is when the system is being compared to an identical architecture on an identical dataset in a domain where metric-human correlation has been empirically validated to exceed 0.7 (e.g., BLEU for surface realisation against the Penn Treebank, where Rajkumar & White, 2014, report high coverage and consistent input formats for some realisers). But the survey's evidence that even validated metrics fail when the specific system architecture changes (Kuznetsova et al., 2014, finding BLEU ranked their system higher than METEOR, contradicting prior meta-evaluation) implies that this exception is narrow and fragile.