PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models

ArXiv: 2605.20873

🎯 Pitch

Even frontier LLMs fail to produce globally consistent plans under coupled constraints—satisfying local conditions most of the time while fatally violating the overall solution. This paper shows that training on synthesized planning problems with well-specified optimal solutions, rather than instances with many acceptable answers, provides the clear reward signals that unlock generalization to unseen planning and instruction-following tasks.

1. Executive Summary

This paper introduces PlanningBench, a framework for generating scalable, diverse, and verifiable planning data by abstracting real-world planning scenarios into a structured taxonomy and synthesizing instances through a constraint-driven synthesis pipeline with adaptive difficulty control, automatic verification, and a preference for determinate or well-specified optimal solutions (instances where the correct plan has a unique or clearly defined optimum rather than admitting many locally acceptable answers). Evaluating open-source and closed-source frontier LLMs on the generated 467-instance suite reveals that even the strongest model, GPT-5.4-xhigh, achieves only 63.17% All-pass, with a substantial gap between All-pass and Avg-pass (e.g., GPT-5.4-medium reaches 90.03% Avg-pass but only 58.09% All-pass) indicating that models frequently satisfy local constraints while failing to produce globally consistent plans. Beyond evaluation, GRPO-based reinforcement learning on verified PlanningBench data improves transfer to unseen planning benchmarks—boosting TravelPlanner All-pass from 28.85% to 46.86%—and to broader instruction-following tasks, with the Collie score rising from 38.33% to 53.17%, establishing that planning-oriented training with determinate reward signals generalizes beyond the training distribution only when the verification signal provides clear directional feedback rather than permissive partial credit.

2. Context and Motivation

The Core Problem: We Need Scalable Planning Data, Not Just Fixed Benchmarks

Planning is not just another NLP task — it represents a fundamentally different class of reasoning challenge. When an LLM generates a plan, it must coordinate goals, constraints, resources, and long-term consequences into a coherent whole. A decision that looks locally reasonable (assigning a doctor to this shift, routing a truck down this road, allocating this batch of materials) can invalidate the entire plan if it violates a global constraint that only becomes apparent later. As the paper notes:

"a locally reasonable decision may invalidate the global plan, and a feasible plan may still be far from optimal"

This is what distinguishes planning from tasks where we evaluate individual outputs or intermediate reasoning steps in isolation. In many reasoning benchmarks, you can score a model by checking whether each step is logical or whether the final answer matches. In planning, you must verify that the entire solution satisfies a web of coupled constraints — temporal, spatial, capacity-based, priority-based, dependency-based — and that it achieves the stated objectives.

The problem this paper addresses is deceptively simple to state: we lack scalable mechanisms for generating planning data with high diversity, controlled difficulty, and automatic verification. Existing planning benchmarks treat planning data as fixed collections of instances rather than as controllable generation targets. This matters because:

First, planning benchmarks are dominated by narrow domain coverage. As Table 1 reveals, most existing benchmarks focus on a single task type — overwhelmingly travel planning (TravelPlanner, TravelBench, TRIP-Bench, WorldTravel, TripTailor, ChinaTravel, TripCraft, and others). Only a handful cover two or four task types (DeepPlanning with 2, ReliabilityBench with 4). PlanningBench covers 30+ task types across six structural families. This isn't just about having more examples — it's about whether a benchmark can test planning ability or merely tests whether a model has memorized travel-planning heuristics.

Second, difficulty control in existing benchmarks is surface-level rather than structural. When benchmarks want harder problems, they typically increase prompt length or the number of requirements. But planning difficulty doesn't come primarily from information volume — it comes from the structure of the constraints: how tightly they couple, whether they conflict, how many levels of dependency exist, and whether the problem admits a unique optimum or a diffuse space of acceptable solutions. A 10-constraint problem where all constraints are independent may be easier than a 3-constraint problem where the constraints interact. Surface-level proxies fail to capture this.

Third, most benchmarks cannot support planning-oriented training. For a benchmark to provide useful training signals, it needs automatic verification. If you're going to use reinforcement learning to improve planning ability, you need reward signals that can be computed programmatically at scale — you can't rely on human evaluation for every training instance. Existing benchmarks vary in their verification support (Table 1), and critically, none of the prior benchmarks that the paper compares against are validated for training use. The "Training validated" column in Table 1 is blank for every prior benchmark.

Fourth, current benchmarks don't distinguish between partial and complete planning success. A model might satisfy 9 out of 10 checklist items while producing a globally invalid plan because the one missed constraint is the keystone that everything else depends on. Standard accuracy metrics (did the plan use the right format? did it respect time windows?) can mask this failure mode. PlanningBench explicitly separates All-pass (all checklist items satisfied) from Avg-pass (fraction of items satisfied) to expose this gap.

Why This Matters: The Verification-Signal Bottleneck

The stakes extend beyond academic benchmarking into how we train the next generation of LLMs. The paper's training experiments reveal a subtle but crucial dynamic: the quality of the verification signal determines whether planning-oriented training succeeds or backfires.

When the paper trained a model on an early batch of ~1k synthetic planning examples without emphasizing determinate optimal solutions, "GRPO training did not yield clear gains and degraded performance on general instruction-following benchmarks." This is a consequential negative result. It means that simply generating "planning data" and using loose verification — checking whether constraints are "mostly satisfied" — can actually make models worse at general instruction following. The paper hypothesizes why:

"Loose answer spaces can allow many outputs to be partially feasible. If verification rewards such outputs too generously, a model may learn local constraint satisfaction without learning to coordinate time, resources, dependencies, and objectives at the global level."

In other words, if the reward model says "good job" when the model satisfies 80% of constraints, the model learns to satisfy 80% of constraints — which is a different skill from producing a globally consistent plan. The training signal becomes diffuse rather than directional. This is why the paper emphasizes determinate optimal solutions: when the task admits exactly one correct answer (or a well-specified optimum), the verification system can say "wrong" or "right" unambiguously, providing a clear gradient for the model to follow.

This has implications for the entire field of RL-based LLM training. The current paradigm — "collect diverse data, train a reward model, optimize against it" — assumes that the reward signal is meaningful even when imperfect. PlanningBench's results suggest that for complex constrained-generation tasks, rewards that are merely "mostly correct" may be worse than useless — they can actively mislead the optimization process. Only rewards that are directional (pointing toward the unique correct solution) produce stable training dynamics and transferable skills.

Prior Approaches and Their Limitations

The paper situates itself against a substantial body of prior work on LLM planning, which it groups into three categories:

Planning benchmarks as fixed test collections. TravelPlanner (Xie et al., 2024) pioneered the use of realistic constraint-based evaluation for travel planning, requiring models to construct valid itineraries under hard constraints. Subsequent benchmarks extended this to multi-turn interactions (TravelBench, Cheng et al., 2025), tool-augmented planning (TRIP-Bench, Shen et al., 2026; COMPASS, Qin et al., 2025), personalized preferences (TripTailor, Wang et al., 2025; TripCraft, Chaudhuri et al., 2025), and multimodal or web-based environments (WorldTravel, Wang et al., 2026; Flex-TravelPlanner, Oh et al., 2025). All of these are valuable testbeds, but they share a common architecture: someone manually designs a set of planning tasks, and then models are evaluated on that fixed set. There's no mechanism for scaling beyond the initial collection or for systematically varying difficulty along structural dimensions.

Other benchmarks extend beyond travel: CostBench (Liu et al., 2025b) evaluates cost-optimal tool sequencing; DeepPlanning (Zhang et al., 2026) covers long-horizon agentic planning with verifiable constraints; Retail (Deng et al., 2025) introduces retail and service scenarios; τ-bench and τ2-bench (Yao et al., 2024; Barres et al., 2025) evaluate tool-agent-user interaction; and ReliabilityBench (Gupta, 2026) tests agent reliability under stress conditions. But again, these are fixed collections, and their difficulty is controlled through proxies — number of turns, tool count, interaction depth — rather than through manipulation of the underlying planning structure.

Planning methods that focus on stronger agents. A parallel line of work improves LLM planning through better agent architectures: task decomposition (Triflow, Chen et al., 2025a), multi-agent collaboration (Vaiage, Liu et al., 2025a; Atlas, Choi et al., 2025), constraint repair (PlangenLLMs, Wei et al., 2025), reinforcement learning (DeepTravel, Ning et al., 2025), human-in-the-loop workflows (Roamify, Udandarao et al., 2025), and synthesized experience (Scaling Agent Learning, Chen et al., 2025b). These methods are orthogonal to PlanningBench: they focus on how to design better planners given a planning task, while PlanningBench focuses on how to generate better planning tasks. The paper explicitly notes that its Generator-Responder-Critic loop is "for data construction, not a new agent architecture."

RL-based training with verifiable rewards. The paper builds on recent work showing that reinforcement learning with verifiable rewards — where a programmatic checker determines correctness — can improve LLM reasoning (DeepSeek-R1, Guo et al., 2025; DeepSeekMath, Shao et al., 2024b). These approaches have largely been applied to math and coding, where ground-truth verification is straightforward (does the answer match? do the tests pass?). PlanningBench extends this paradigm to planning, where verification requires checking a set of coupled constraints simultaneously.

How PlanningBench Positions Itself

PlanningBench is not "another planning benchmark." It is a planning data generation framework that treats data construction as a controllable synthesis problem rather than a one-time collection effort. The key design choices that distinguish it:

1. Real-scenario grounding with taxonomic abstraction. Rather than starting from scratch or from a single domain, PlanningBench abstracts reusable planning schemas from real planning scenarios across six structural families (Scheduling and Timetabling, Allocation and Matching, Shift and Workforce Scheduling, Routing and Travel, Project and Production Operations, Emergency Response and Public Service). Twenty professional annotators with planning experience, working with algorithm researchers, identified stable problem structures — planning targets, candidate actions, resources, time windows, dependencies — that recur across domains. These aren't just domain labels; they're structural categories that capture different sources of planning complexity (temporal conflicts vs. resource assignment vs. route selection vs. milestone dependencies).

2. Constraint-driven synthesis with difficulty control. Each task type is paired with a pool of constraints organized into three difficulty tiers (basic, medium, hard). The synthesis pipeline samples task-constraint configurations and instantiates them as self-contained problems, using a closed-loop feedback mechanism (Generator → Responder → Critic) that increases difficulty when the current Responder model fully solves a generated instance. This means difficulty is adaptive: the pipeline actively searches for problem configurations at the frontier of what the Responder can handle.

3. Verification-first design. Every generated instance includes a verification checklist that specifies exactly what a correct plan must satisfy. This isn't a post-hoc evaluation layer; it's built into the data generation process from the start. The same verification structure serves double duty: it enables automatic evaluation (the Critic checks model outputs against the checklist) and it provides reward signals for RL training.

4. Determinate optima as a design principle. PlanningBench deliberately favors problem configurations with unique or well-specified optimal solutions. This isn't a philosophical preference; it's an empirical finding from early experiments showing that loose answer spaces produce loose reward signals, and loose reward signals fail to improve planning ability. The paper is transparent about this — it reports the negative result with Syn-NotDetOptimal data (where synthetic data doesn't emphasize determinate optimality) showing only +0.75 points average improvement on general benchmarks versus +7.06 points for determinate-optimal data (Table 5).

The paper's relationship to prior work is therefore complementary rather than competitive. It doesn't claim to be a better travel planner or a better agent architecture. It claims to provide a controllable source of planning data that existing methods — and future ones — can use for both evaluation and training. As the authors state:

"PlanningBench complements prior benchmarks by providing a scalable source of diverse and verifiable planning data rather than another fixed evaluation set."

This positioning is important because it shifts the conversation from "which benchmark is harder?" to "how do we generate planning data that can actually improve models?" — a question that the field has not systematically addressed.

3. Technical Approach

3.1 Reader Orientation

The system being built is a synthetic data generation pipeline that produces planning problems and their answer-checking rubrics in a controllable, automated way. It solves the problem that existing planning benchmarks are fixed collections of hand-designed instances—you cannot easily scale them, control their difficulty, or use them for training—by instead defining a reusable design space of planning tasks and constraints, then synthesizing new instances through a closed-loop process that adapts difficulty based on how well a reference model solves the current problems.

3.2 Big-Picture Architecture (Diagram in Words)

The PlanningBench system has two macro stages and three interacting components within the second stage:

Macro Stage 1: Taxonomy Construction. Twenty professional annotators with planning experience and algorithm researchers analyze representative real-world planning scenarios. They abstract stable problem structures—planning targets, candidate actions, resources, time windows, dependencies, required plan conditions—and organize them into two taxonomies: a task taxonomy (what type of plan to construct) and a constraint taxonomy (feasibility, optimization, and state-dependent requirements that make plans verifiable). The output is a structured specification linking each task type to its reusable constraint pools, organized into three difficulty tiers (basic, medium, hard) and an optional stateful constraint layer.

Macro Stage 2: Constraint-Driven Synthesis and Difficulty Enhancement. Three components operate in a closed loop:

1. The Generator samples a task type, subtask variant, and constraint configuration from the taxonomies, then instantiates a self-contained planning problem (x) with a paired verification checklist (c). The checklist converts the prompt-level constraints into instance-specific, machine-checkable rubric items.

2. The Responder (a configurable LLM) attempts to solve the generated problem, outputting a candidate plan (ŷ).

3. The Critic evaluates the candidate plan against the checklist using a prompt-based verifier, producing a verification score (ρ ∈ [0, 1]) and a binary all-pass indicator (u = 1 if all constraints are satisfied and final verification passes, 0 otherwise).

The loop uses u as feedback: if the Responder fully solves the instance (u = 1), the pipeline shifts the constraint sampling distribution toward harder constraints before generating the next instance. If the Responder fails (u = 0), the instance remains challenging under the current configuration and is retained. After synthesis, instances undergo human quality-control auditing and revision before entering a verified data pool that supports both model evaluation and planning-oriented training.

3.3 Roadmap for the Deep Dive

• First, I'll explain the definition and scope of a PlanningBench instance—what exactly is being generated and verified, and why the paired (problem, checklist) representation is fundamental.
• Second, I'll walk through the task and constraint taxonomies in detail: how they're constructed, the six task families and what structural complexity each captures, the three constraint categories (general, task-specific, specialized stateful), and the three difficulty tiers.
• Third, I'll unpack the constraint-driven synthesis loop: how the Generator samples task-constraint configurations, how the Responder produces candidate solutions, how the Critic evaluates them, and how the closed-loop difficulty update function works.
• Fourth, I'll explain the automatic verification design—how the checklist is structured, how it enables both evaluation and training, and the human quality-control audit process that filters and revises instances.
• Fifth, I'll discuss the preference for determinate optimal solutions: what it means, why the early training experiments showed it was essential, and how later data batches implement it.
• Sixth, I'll cover the training setup that uses PlanningBench data: the GRPO configuration, the four comparison settings (Base, Human-Authored, Syn-PlanningBench, Syn-NotDetOptimal), and how this connects the data generation architecture to the downstream training experiments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a data generation framework paper whose core idea is that planning data should be treated as a controllable synthesis target rather than a one-time collection, and that the quality of verification signals—specifically their determinacy—determines whether planning-oriented training succeeds.

PlanningBench Instance Definition and Scope

Every instance PlanningBench generates is a pair (x_i, c_i), where x_i is a self-contained planning task and c_i is a verification checklist. This pairing is not an afterthought; it is the atomic unit of the entire framework. The paper defines the scope explicitly: PlanningBench targets "text-based planning problems in which all information needed to construct and verify a plan is provided in the input." This means no external tools, no database lookups, no environment interaction—the model receives a text prompt and must produce a text plan that can be verified solely against the prompt and the checklist.

The checklist c_i specifies "the requirements a model-generated plan must satisfy, including relevant context, constraints, output conditions, and task objectives." When an instance specifies a determinate optimum, the checklist additionally checks whether the plan reaches that optimum. This dual-purpose design—same verification structure for evaluation and training—is deliberate. For evaluation, the checklist enables automatic pass/fail scoring. For training, the same checklist items provide reward signals during reinforcement learning: a model that satisfies more checklist items earns higher reward, and a model that satisfies all items earns the maximum reward.

A crucial design choice is that PlanningBench does not ask "can the model produce a reasonable suggestion?" It asks "can the model integrate all provided information into a concrete, executable, and verifiable plan?" This distinguishes planning from open-ended generation. The paper notes: "When an instance specifies a determinate optimum, verification further checks whether the plan reaches that optimum." This means the framework tests not just feasibility (does this plan work?) but also optimality (is this the best plan under the stated objectives?).

Task Taxonomy: Six Structural Families

The task taxonomy is not organized by surface application labels (e.g., "healthcare," "logistics," "education") but by planning structure—the underlying mathematical or logical form of the constraint satisfaction and optimization problem. The authors argue that this structural organization captures genuine sources of planning difficulty, rather than obscuring them behind domain jargon. Twenty professional annotators with planning experience, working alongside algorithm researchers, reviewed representative scenarios from real applications and identified recurring problem structures.

The taxonomy defines six families, each capturing a different source of structural planning complexity:

1. Scheduling and Timetabling covers problems where the core challenge is temporal coordination under conflicts and time-window matching. The structural complexity comes from the need to assign activities to time slots while respecting: time windows (each activity can only occur within a specified interval), mutual exclusion (no resource can be in two places at once), sequencing (activities must occur in a specific order), and execution duration. The paper lists concrete tasks in this family: Schedule Planning (personal agendas), Meeting Planning (coordinating shared slots across participants), Course Timetabling (classrooms, teachers, classes without conflicts), Exam Scheduling (rooms, invigilators, student load), Email Scheduling (audience timing, frequency, campaign ordering), Content Scheduling (social media calendars, production dependencies), Study Planning (subject allocation under daily limits and fixed commitments), Fitness Planning (training schedules under goals and restrictions), and Team-Building Event Planning (venues, logistics, budget, participant changes).

2. Allocation and Matching covers problems where the core challenge is resource assignment under compatibility and capacity constraints. Unlike scheduling, the primary difficulty isn't temporal (although time may be a factor) but rather the combinatorial matching problem: given limited resources and multiple demanders with different requirements, how do you assign resources to maximize some objective while respecting constraints? The tasks include Procurement Planning (replenishment timing and suppliers under lead time, inventory, and budget), Material Allocation (scarce supplies under fairness, priority, and minimum-guarantee rules), Educational Resource Planning (teachers, classrooms, equipment across classes under shared constraints), and Student Grouping (forming balanced groups under size, compatibility, and diversity requirements).

3. Shift and Workforce Scheduling covers problems where the core challenge is coverage, rotation fairness, and workforce availability. The structural complexity comes from the need to ensure continuous coverage (someone must always be on duty) while respecting individual constraints (availability, qualifications, rest requirements) and fairness criteria (no one should always get the undesirable shifts). Tasks include Volunteer Scheduling (deploying volunteers to posts under skills, training, peak demand, and rotation constraints), Recruitment Planning (multi-stage hiring pipelines under interviewer availability, urgency, and budget), Shift and Duty Scheduling (nurse rosters, customer-service duty schedules with coverage, rest, qualification, and holiday rules), and Work Assignment Scheduling (task allocation under skills, deadlines, dependencies, and workload balance).

4. Routing and Travel covers problems where the core challenge is route selection, spatiotemporal coordination, and multi-leg transfers. The structural complexity comes from the need to construct sequences of locations that satisfy ordering, time, capacity, and cost constraints—this is the classic vehicle routing / traveling salesman family of problems. Tasks include Commuting Planning (route selection under arrival-time, cost, transfer, and contingency constraints), Urban Traffic Planning (routing and diversion under road capacity, peaks, and event-driven demand), Travel Itinerary Planning (multi-day itineraries under budget, reservations, preferences, mobility limits, and disruptions), Logistics Delivery Planning (multi-stop delivery routing under vehicle capacity, customer time windows, and driver-hour limits), and Transportation Dispatch Planning (vehicle assignment to transport jobs under fleet limits, sequencing, and deadhead minimization).

5. Project and Production Operations covers problems where the core challenge is milestones, dependencies, capacity limits, and execution continuity. The structural complexity comes from the need to sequence interdependent tasks under shared resource constraints—this is the project scheduling / job-shop scheduling family. Tasks include Project Planning (milestone-driven projects under staffing, budget, dependencies, and deadlines), Financial Budgeting (budget distribution across competing needs under caps, priorities, and reserve requirements), Production Planning (production line scheduling under materials, changeovers, and urgency), Warehouse Planning (storage layout and inbound sequencing under zoning, capacity, accessibility, and turnover constraints), Power Dispatch Planning (power supply allocation under demand peaks, generation limits, and critical-load protection), Resource Scheduling (shared equipment across concurrent tasks under cost or makespan objectives), and IT System Maintenance Planning (maintenance windows, rollback-ready changes, and service dependency constraints).

6. Emergency Response and Public Service covers problems where the core challenge is timeliness, priority-based allocation, and plan restructuring under disruption. The structural complexity comes from the need to make rapid, high-stakes allocation decisions under severe resource constraints, often with the requirement that the plan remain robust to cascading failures. Tasks include Emergency Planning (response workflows under deadlines, roles, materials, and concurrent incidents), Medical Resource Planning (doctor scheduling, room allocation, patient flow under urgency, capacity, and waiting-time targets), and Disaster Evacuation Planning (safe evacuation routes under exit capacity, vulnerable groups, and route failures).

Each family contains concrete tasks, and each task is divided into 5–10 subtasks on average to capture variants of the same planning problem type. The subtask level is where PlanningBench encodes specific parameterizations—for example, under Meeting Planning, one subtask might be "cross-team meeting coordination" while another might be "interview panel scheduling." This granularity matters because different subtasks within the same task family can stress-test different sub-capabilities (e.g., handling many participants with sparse availability vs. handling few participants with tight priority constraints).

The taxonomy is provided in full in Appendix A (Table 6), which lists each main task, illustrative subtask variants, and "what the task mainly tests." The testing column reveals the paper's design philosophy: each task is constructed to evaluate specific planning sub-skills (arranging under time windows and priorities, resolving resource conflicts, building fair rosters, coordinating routing under capacity constraints, etc.), not general "does the model output something that looks like a plan?"

Constraint Taxonomy: Three Categories, Three Difficulty Tiers

Constraints are what make planning problems verifiable and controllable. The paper argues that constraints should be modeled together with task types, not added as post-hoc annotations, because "task and constraint structures are closely coupled." This produces what the paper calls a structured task-constraint specification: each task is associated with reusable constraint elements, and the synthesis pipeline samples from these elements to compose instances.

The constraint taxonomy organizes reusable constraints into three categories:

General constraints are shared across all tasks and capture universal planning requirements. The paper lists the full set in Appendix B (Table 7), with six basic-level, six medium-level, and six hard-level constraints. Representative basic general constraints include: "Explicit primary objective" (the task must state a clear, single, verifiable objective, not just "make a reasonable plan"), "Complete inputs" (key inputs including entities, resources, time ranges, capacity bounds, and constraints must be explicit), "Exclusive resource usage" (the same resource cannot be assigned to multiple mutually exclusive tasks simultaneously), "Time-window constraints" (tasks may be scheduled only within specified windows), "Capacity limits" (quantities must respect hard upper bounds), and "Structured output" (the output should use a structured format such as tables, YAML, JSON, or time-blocked lists). Medium general constraints introduce optimization and fairness requirements: "Hard vs. soft constraints" (explicit separation with hard constraints taking precedence), "Multi-objective optimization" (a primary objective plus 1–3 secondary objectives with explicit priority ordering), "Fairness and balance" (a concrete balance metric such as maximum gap or variance cap), "Buffer time" (minimum transition/rest time between tasks), "Precedence dependencies" (explicit ordering requirements), and "Conflict validation" (the plan must include an explicit post-plan feasibility check). Hard general constraints introduce advanced planning challenges: "Infeasibility detection" (when input conditions are impossible, the plan should identify infeasibility rather than produce a superficially complete plan), "Minimum-change replanning" (when revising an existing plan, minimize the scope of changes), "Robustness design" (the plan should remain stable under small perturbations through redundancy, backups, or elastic capacity), "Exception recovery strategy" (provide backup plans for common abnormal scenarios such as absence, equipment failure, or delays), "Local information incompleteness" (when input contains ambiguity, adopt conservative arrangements and flag assumptions), and "Explainability" (give brief reasons for key arrangements focused on feasibility, optimization goals, and constraint tradeoffs).

Task-specific constraints are tied to a particular task structure and define requirements that arise from that task's specific operational context. Appendix B (Table 8) provides concrete examples across five tasks: IT System Maintenance Planning, Meeting Planning, Medical Resource Planning, Work Assignment Scheduling, Logistics Delivery Planning, and Material Allocation. Each task has its own matrix of constraints at basic, medium, and hard levels. For example, Meeting Planning basic constraints include "Key Attendee Availability" (only available people may be scheduled), "Meeting Room Capacity and Equipment Match" (the room must fit and have needed equipment), "Valid Duration" (meetings must respect time boundaries), "No Time Conflict" (a person cannot be in two meetings at once), "Complete Attendance Role Coverage" (all required roles must be filled), "Equipment-Demand Matching" (required equipment must be available), "Minimum Meeting Duration" (meetings cannot be too short), and "Daily Meeting Count Limit" (a hard cap on meetings per day). Meeting Planning medium constraints introduce optimization: "Reduce Schedule Fragmentation" (avoid scattered free time), "Cross-Time-Zone Fairness" (balance meeting times across time zones), "Meeting Priority" (encode and enforce priority rules), "Consecutive-Meeting Control" (limit back-to-back meetings), "Time-Zone Fairness" (a separate balance check), "Consecutive-Meeting Buffer" (enforce gaps between meetings), "Priority Ranking Rule" (explicit ordering), and "Meeting-Chain Order" (respect dependencies between related meetings). Meeting Planning hard constraints introduce disruption handling: "Minimum Disruption from Rescheduling," "Backup Time Slots," "Local Information Incompleteness," "Global Coordination across Multiple Meetings," "Recovery from Key-Person Changes," "Temporary Meeting-Room Failure," "Agenda Splitting and Merging," and "Confidentiality-Level Layering." Each constraint is matched with a specific assessment criterion and a constraint note explaining how to operationalize it.

Specialized stateful constraints depend on previously executed actions, accumulated state, or triggered conditions. These are maintained as a separate optional layer and sampled independently when needed. Unlike general and task-specific constraints, stateful constraints are not bound to a particular difficulty tier and can be added to any instance where the task template requires state-dependent behavior. The paper doesn't provide an exhaustive catalog of stateful constraints but characterizes them as constraints that require tracking an evolving solution state—for example, "if resource X is used in step 3, it cannot be used again until step 7 unless condition Y is met," or "when the cumulative load exceeds threshold Z, switch to backup protocol W."

The three difficulty tiers are defined structurally, not superficially:

• Basic constraints cover fundamental feasibility: complete inputs, resource boundaries, time windows, capacity limits, exclusive usage, and executability. These are the "hard" constraints that make a plan valid or invalid; violating any basic constraint means the plan is infeasible.
• Medium constraints introduce optimization objectives: fairness, balance, multi-objective trade-offs, load balancing, soft constraint satisfaction, priority-based ordering, and dependency chains. These go beyond "does the plan work?" to "is the plan good?"—they create the need for tradeoff reasoning.
• Hard constraints capture demanding structural requirements: infeasibility recognition, exception recovery, robustness design, coordination under conflicting objectives, minimum-change replanning, and explainability. These test whether the model can handle the meta-level of planning—not just constructing a plan, but reasoning about plan quality, plan failure, and plan adaptation.

The difficulty tiers are not arbitrary labels. They are operationalized in the synthesis pipeline: the initial constraint-count sampling prior assigns specific probabilities to different constraint counts within each tier, and the closed-loop difficulty enhancement mechanism shifts these probabilities toward harder tiers when the Responder consistently solves generated instances. The paper provides the exact priors:

• For basic constraints (Nb): drawn from {1: 0.2, 2: 0.6, 3: 0.2}—so each instance has 1–3 basic constraints, with 2 being most common.
• For medium constraints (Nm): drawn from {0: 0.25, 1: 0.55, 2: 0.2}—so instances sometimes have no medium constraints, usually have 1, and occasionally have 2.
• For hard constraints (Nh): drawn from {0: 0.7, 1: 0.3}—so hard constraints are sparse, appearing in only 30% of initial instances.

These priors ensure that early generated instances are manageable—they always have at least one basic constraint for feasibility, typically include some optimization (medium) requirements, and only occasionally introduce the hardest planning challenges. The closed-loop mechanism then adjusts these probabilities upward for subsequent generations when the Responder demonstrates mastery.

The Constraint-Driven Synthesis Loop

Given the constructed taxonomies, the pipeline synthesizes instances through a three-component closed loop. This is the operational core of PlanningBench—it's what turns static taxonomies into a scalable data generation engine.

Step 1: Generator instantiates a problem and checklist. For a sampled task τ and subtask variant σ, the Generator draws constraints from the task-specific constraint pools Cb(τ), Cm(τ), and Ch(τ). The initial constraint counts are sampled from the categorical priors described above, which the paper explicitly encodes as:

Nb ∈ {1, 2, 3}
Nm ∈ {0, 1, 2}
Nh ∈ {0, 1}

After drawing the counts, the Generator samples without replacement from the corresponding pools:

Sb ⊂ Cb(τ),  Sm ⊂ Cm(τ),  Sh ⊂ Ch(τ)

where |Sb| = Nb, |Sm| = Nm, and |Sh| = Nh. These subsets are checked against task-template compatibility rules to avoid duplicate, contradictory, or underspecified requirements. The Generator also samples a random variable z controlling "wording, numerical instantiation, and contextual details"—this ensures that two instances with the same constraint configuration don't produce identical text.

The full generation specification is:

r = (τ, σ, Sb, Sm, Sh; z)

The Generator then produces a problem and checklist pair (x, c) = G(r). The problem x is a self-contained text prompt containing all necessary information for planning; the checklist c is a structured set of verifiable conditions derived from the sampled constraints.

The exact prompt template used to instruct the Generator is provided in Appendix G.1. It is a detailed prompt that instructs the model (acting as the Generator) to design a planning task and checklist based on provided evaluation points, scenario context, and external reference material. The prompt explicitly requires that the task "must require the test-taker to genuinely perform planning / scheduling / allocation / dispatching / arrangement, rather than merely giving advice, describing ideas, or discussing the topic in general terms." It also specifies output format requirements, mandating structured input information covering "object sets, resource sets, time ranges, capacity limits, budget limits, conflict conditions, dependency relations, priorities, and exceptional situations."

Step 2: Responder attempts a solution. Given the generated problem x, the Responder (a configurable LLM selected during pipeline operation) outputs a candidate plan ŷ. The Responder is not fixed—the paper notes that during evaluation data construction, it used Qwen-A3B-30B as the Responder, while during training data construction, the specific model can vary. The important point is that the Responder's capability determines the difficulty frontier: the closed-loop mechanism increases difficulty when the Responder solves instances, so a stronger Responder leads to harder generated instances.

Step 3: Critic evaluates the solution. The Critic assesses ŷ against the checklist c using a prompt-based verifier (the paper uses GPT-oss-120b as the Critic during evaluation data construction). The Critic produces two outputs:

• A verification score ρ ∈ [0, 1], representing the overall quality of the response on a continuous scale.
• A binary all-pass indicator u = I{ŷ satisfies all constraints and passes final verification}, which is 1 only if every checklist item is satisfied and 0 otherwise.

The Critic prompt template is provided in Appendix G.2. It instructs the evaluator model to follow a rigorous grading procedure: first list all explicit and implicit requirements from the checklist, then check the student answer against each requirement, then conduct a self-reflection step (completeness, strictness, consistency, and objectivity checks), and finally output a scoring rationale, a requirement satisfaction status list (a binary vector where each element indicates whether one checklist item is satisfied), and an overall score from 0 to 10.

Step 4: Difficulty update. The binary indicator u serves as feedback for the next iteration of synthesis. When u = 1, the current Responder fully solves the instance—meaning the problem is too easy relative to the target difficulty—so the pipeline keeps the same task type but shifts the constraint sampling distribution toward harder constraints for subsequent generations. When u = 0, the instance remains challenging under the current configuration, so the pipeline retains it without further difficulty escalation.

This is the paper's core difficulty adaptation mechanism, formalized as an exponential update on the sampling probabilities. Let:

p(k) = (p(k)_b, p(k)_m, p(k)_h)

denote the sampling probabilities over basic, medium, and hard constraints at iteration k. When a harder variant is requested (because u = 1 in the previous iteration), the next distribution is updated by:

where ⊙ denotes element-wise (Hadamard) multiplication, and η, α, β, γ > 0 are hyperparameters controlling the update magnitude and direction. The vector [−α, β, γ] means:

• The basic constraint weight is multiplied by exp(−ηα), which is less than 1, so basic constraints become less likely to be sampled.
• The medium constraint weight is multiplied by exp(ηβ), which is greater than 1, so medium constraints become more likely.
• The hard constraint weight is multiplied by exp(ηγ), which is greater than 1, so hard constraints become more likely.

The Normalize function then rescales the resulting vector so the three probabilities sum to 1, maintaining a valid categorical distribution.

What it computes: Given current sampling probabilities over the three difficulty tiers and a signal that the Responder solved the previous instance, this update produces new sampling probabilities that reduce the probability of drawing basic constraints and increase the probability of drawing medium and hard constraints. The result is a vector p^{(k+1)} of three non-negative numbers summing to 1.

Why this form: The exponential update is multiplicative rather than additive, meaning the relative odds between difficulty tiers are scaled by constant factors rather than by constant amounts. This is important because the probabilities live on the simplex (they sum to 1), so additive updates with a fixed step size could push probabilities outside [0,1] or cause oscillatory behavior. The multiplicative form ensures the update is proportional to the current probabilities—if basic constraints already have low probability, the reduction is small; if they have high probability, the reduction is large. The separate α, β, and γ parameters allow asymmetric adjustment: the decrease in basic constraints (controlled by α) can be slower or faster than the increase in medium (β) and hard (γ) constraints. This asymmetry matters because the pipeline shouldn't jump from all-basic to all-hard in one step; it should gradually shift the distribution.

After the probability update, the continuous probabilities are converted back to discrete constraint counts. Given the next-turn total constraint number M^{(k+1)}, the expected allocation is:

E[n^{(k+1)}] = M^{(k+1)} p^{(k+1)}

This expected allocation is then projected onto the admissible count space:

S = {1, 2, 3} × {0, 1, 2} × {0, 1}

through a projection operator Π_S(·), yielding discrete counts (N^{(k+1)}_b, N^{(k+1)}_m, N^{(k+1)}_h) that respect the allowed ranges for each tier. This projection step is necessary because the expected allocation may contain fractional values (e.g., 1.7 basic constraints), and the pipeline must select an integer number of constraints from each pool. The paper notes that this projection "preserves the allowed count ranges for basic, medium, and hard constraints while reflecting the updated difficulty distribution."

The overall effect is an adaptive search over structured planning configurations. The pipeline doesn't randomly generate problems at varying difficulty; it actively seeks out the difficulty frontier where the Responder begins to struggle, producing instances that are challenging but not impossible. This is analogous to curriculum learning in reverse: rather than training a model on progressively harder problems, the pipeline generates problems at progressively higher difficulty to find the boundary of a reference model's capability.

It's worth noting what the pipeline does not do. It does not guarantee that every generated instance is perfectly calibrated or that the difficulty progression is monotonic. The Critic's evaluation is itself model-based (GPT-oss-120b) and may have its own errors or biases. The feedback loop shifts sampling probabilities but does not guarantee that every subsequent instance is genuinely harder—it's a stochastic process that trends toward harder configurations on average. This is why human quality control (described next) is an essential filtering step.

Automatic Verification and Quality Control

Every generated instance includes a verification checklist c_i that converts the prompt-level constraints and objectives into instance-level, machine-checkable items. The checklist is not a separate artifact produced after the fact; it is generated simultaneously with the problem x_i by the same Generator model, ensuring tight coupling between what the problem asks and what the verification checks.

The paper describes the checklist as covering "input conditions, resource and time constraints, output format, and determinate objectives when available." This means the checklist items are operationalized versions of the abstract constraints sampled during synthesis. For example, if the Meeting Planning task samples the "Key Attendee Availability" basic constraint, the checklist will contain an item like "Check whether all scheduled meeting times fall within the availability windows of the required attendees as specified in the input." If the task samples the "Meeting Room Capacity and Equipment Match" constraint, the checklist will contain an item like "Verify that each assigned meeting room has capacity ≥ the number of required attendees and has all specified equipment."

The checklist design follows a structured format intended to be directly verifiable. As described in Appendix G.1, the prompt template instructs the Generator to design a "0/1 scoring standard: assign 1 only if all conditions are satisfied; assign 0 if any key condition is not satisfied." Each condition must be "directly verifiable" and include "a clear verification method." The conditions should refer to "concrete elements in the task, such as specific time slots, specific resource names, specific headcount limits, specific budget values, specific output fields, or specific conflict rules." This design ensures that the verification can be performed programmatically or by a judge model without ambiguity about what constitutes a pass or fail.

The dual-purpose nature of this verification structure is central to PlanningBench's value proposition. For evaluation, the checklist enables the All-pass and Avg-pass metrics reported in Section 4.1. The Critic model (GPT-oss-120b in the evaluation experiments) reads the model's output, checks it against each checklist item, and determines whether all items are satisfied (All-pass = 1) or what fraction are satisfied (Avg-pass). For training, the same checklist provides reward signals during GRPO-based reinforcement learning (Section 4.2). A model output that satisfies more checklist items earns higher reward, and the binary all-pass indicator provides a particularly clean signal: 1 for a completely correct plan, 0 otherwise.

Human quality-control audit. After automatic generation and verification, PlanningBench applies a human audit to ensure data usability. The paper does not present this as optional—it's a necessary step because automated synthesis, even with a Critic, can produce instances with subtle flaws. Twenty professional annotators (the same pool that constructed the taxonomies) review synthesized samples and assign each to one of four categories:

1. No modification needed: The prompt, checklist, and reference answer are clear, consistent, and directly usable as-is.
2. Minor revision with usable source data: The instance is generally valid but minor edits improve wording, formatting, checklist coverage, or answer clarity. The underlying source information is correct.
3. Minor revision with source data not directly usable: The instance can be retained only after correcting or completing source information, such as missing rules, incomplete boundary conditions, or underspecified constraints. The instance is recoverable but requires more substantial editing.
4. Discard: The instance contains irrecoverable ambiguity, inconsistency, or verification failure and is removed.

The paper reports statistics on the audited batch: 86.15% of synthesized samples require no revision or only minor revision (categories 1 and 2), while 13.85% require additional source correction before retention (category 3). Notably, no sample in this batch is directly assigned to the discard category. This suggests the synthesis pipeline usually produces recoverable instances, but that roughly one in seven requires non-trivial editing to fix issues with the underlying source data.

The common revision types cited include "missing checklist items, incomplete constraint checks, ambiguous or underspecified prompts, unclear references, failures to follow required output structures, missing rules or boundary conditions, and occasional logical inconsistencies between the prompt and answer." These are not random errors—they reflect systematic failure modes of the Generator model, which can sometimes produce instances where the constraints in the prompt don't fully support the verification items in the checklist, or where the reference answer (if generated alongside the instance) doesn't actually satisfy all stated constraints.

This quality-control step is important for interpreting PlanningBench's results. The evaluation set of 467 instances passed through human review, so the scores in Table 2 reflect performance on curated, corrected instances. This means the benchmark's difficulty is genuine—models aren't failing because of buggy or ambiguous prompts. The training data also passes through the same quality-control process, ensuring that the reward signals during GRPO training are based on well-formed problem-checklist pairs.

Preference for Determinate Optimal Solutions

This design principle is not an obvious requirement for a planning benchmark—many real-world planning problems don't have unique optimal solutions—but the paper makes a strong empirical case for it. The argument proceeds in three steps.

Step 1: The problem with loose answer spaces. In planning tasks where many outputs can be partially or even fully feasible, verification becomes ambiguous. If a task asks to "create a reasonable schedule" and the model produces something that satisfies 8 out of 10 constraints, is that a correct answer? A permissive verification system might say "yes" (because the output is certainly reasonable) or assign partial credit. But the paper argues this produces "positive but diffuse" reward signals during training:

"If verification rewards such outputs too generously, a model may learn local constraint satisfaction without learning to coordinate time, resources, dependencies, and objectives at the global level. The resulting reward signal can be positive but diffuse. It may indicate how to obtain partial credit, but not how to construct a globally consistent plan."

This is the key insight: partial credit for planning teaches partial planning. A model that learns to satisfy 80% of constraints has not learned to plan—it has learned to optimize a surrogate metric that correlates with but does not equal planning success. The gradient from such a signal points toward "satisfy more checklist items," not toward "construct a coherent global plan."

Step 2: Empirical evidence from early experiments. The paper reports a concrete negative result that motivated this design choice. In an early batch of about 1,000 synthesized examples that did not emphasize determinate optimality:

"GRPO training did not yield clear gains and degraded performance on general instruction-following benchmarks."

This is a striking failure mode. The model didn't simply fail to improve—it got worse at general instruction following. The paper's hypothesis is that loose verification signals produced a gradient that encouraged the model to pattern-match toward partial constraint satisfaction, which interfered with its ability to follow instructions holistically. This is consistent with the broader RLHF literature on reward hacking: when the reward function is an imperfect proxy for the true objective, optimization can exploit the proxy in ways that diverge from the intended behavior.

Step 3: Operationalizing determinate optimality. Later batches of PlanningBench data strengthen the preference for determinate or well-specified optimal solutions "whenever possible." The paper defines determinate optimal solutions as instances where the correct plan has a unique or clearly defined optimum rather than admitting many locally acceptable answers. This doesn't mean every instance is a single correct answer—some tasks have multiple acceptable solutions that achieve the same objective value. But it does mean that the verification checklist can unambiguously distinguish correct from incorrect plans.

The effect is visible in the training dynamics comparison (Section 4.2.4, Figure 4). Data constructed with determinate optimality (Syn-PlanningBench) shows: (a) a solve-none ratio that decreases fastest and converges to the lowest level, (b) a solve-all ratio that increases more consistently, and (c) a smoother critic reward curve suggesting more stable optimization signals. In contrast, data without determinate optimality (Syn-NotDetOptimal) shows "weaker dynamics and maintains a low solve-all ratio, suggesting that partial checklist satisfaction does not reliably translate into complete solutions."

The paper is appropriately cautious about claiming causality, noting that "more systematic experiments are still needed" to isolate the effect of determinate optimality from other differences between data batches. But the empirical pattern—negative results with loose optima, positive results with determinate optima, and more favorable training dynamics—is consistent with the theoretical argument.

This design principle has implications beyond PlanningBench. It suggests that for any domain where RL-based training with verifiable rewards is applied, the quality of the verification signal matters enormously. A verification system that says "this is mostly correct" is fundamentally different from one that says "this is exactly correct," and the former may actually be worse than useless for training. This parallels findings in the mathematical reasoning literature (e.g., DeepSeek-R1) where exact-match verification provides cleaner training signals than approximate scoring, but extends it to the domain of structured planning where verification is inherently more complex.

Training Setup: GRPO on Verified PlanningBench Data

The final component of the technical approach is how PlanningBench data is used for training. While Section 4.2 reports the results, the training methodology itself is part of the system design because it determines what PlanningBench data must provide.

Base model and algorithm. The paper uses Qwen-A3B-30B as the base model for all training experiments. GRPO (Group Relative Policy Optimization) is the reinforcement learning algorithm, following the approach introduced by Shao et al. (2024b) in DeepSeekMath. GRPO is a variant of PPO that eliminates the need for a separate value function by using group-relative advantages, making it more memory-efficient for training on verifiable-reward tasks.

Training hyperparameters. The paper reports the following configuration (Appendix F):

• Batch size: 128
• Training epochs: 50
• Actor learning rate: 2 × 10⁻⁶
• KL coefficient: 0.001 (controls how far the policy can deviate from the reference model)
• Rollout sampling: 8-way (the model generates 8 candidate responses per prompt during training, and advantages are computed relative to the group)

During inference, the paper uses temperature = 0.7, top-p = 0.6, and top-k = 20.

Training data settings. The paper compares four configurations, all using Qwen-A3B-30B as the starting model:

1. Base Model: The original model without any reinforcement learning. This is the control condition.

2. Syn-PlanningBench: 300 PlanningBench training instances constructed with "verification-oriented constraints and a preference for determinate or well-specified optimal solutions." This is the primary experimental condition—data generated through the full pipeline with the determinate-optimality design preference.

3. Syn-NotDetOptimal: 300 synthetic planning instances that "does not emphasize determinate optimality to the same extent." This is the ablation condition testing whether determinate optimality matters. The paper notes that this data still comes from the synthesis pipeline but with looser constraints on the solution space.

4. Human-Authored: A comparison baseline of 300 planning instances "independently written from scratch by the same pool of twenty professional annotators, without using the PlanningBench taxonomy, constraint-driven synthesis pipeline, or generated instances." This tests whether the structured synthesis approach provides benefits over human-crafted data of comparable size.

All three RL datasets are controlled to be approximately the same size (300 instances). This is important because it rules out the possibility that gains from Syn-PlanningBench are simply due to having more data than Human-Authored.

What the training data provides. Each of the 300 instances is a (x_i, c_i) pair—a self-contained planning problem and its verification checklist. During GRPO training, the model generates a candidate plan ŷ for each problem x_i, and the Critic (or a programmatic checker) evaluates ŷ against c_i to compute a reward. The reward signal is derived from how many checklist items are satisfied, with the binary all-pass indicator serving as a particularly clean signal for complete planning success.

The critical design choice is that the same verification structure serves for both evaluation and training. There's no separate "training rubric" and "evaluation rubric"—the checklist used to score the model during RL is the same checklist used to assess it at test time. This means the training signal is directly aligned with the evaluation metric, which is the ideal condition for RL (no reward-evaluation mismatch).

Why GRPO with this setup? The paper's choice of GRPO rather than standard PPO or DPO is motivated by the verification-first design. GRPO is specifically designed for tasks with verifiable rewards—it computes advantages within a group of sampled responses rather than requiring a learned value function, which is appropriate when the reward comes from a deterministic checker rather than a learned reward model. The 8-way rollout sampling means the model generates 8 candidate plans per problem and receives rewards for each, allowing the group-relative advantage to capture which plans are better or worse relative to the model's own generation distribution.

The three RL datasets test different hypotheses about what makes planning data useful for training. Syn-PlanningBench tests whether taxonomy-guided, verification-oriented synthesis with determinate optima produces data that generalizes. Syn-NotDetOptimal tests whether the determinate-optimality component is necessary (the paper's hypothesis is yes). Human-Authored tests whether structured synthesis is better than human-crafted data of the same size—if Human-Authored performs similarly to Syn-PlanningBench, then the synthesis pipeline is just automating what humans can do; if Syn-PlanningBench significantly outperforms Human-Authored, then the taxonomy-guided synthesis is producing structurally better training data.

4. Key Insights and Innovations

Innovation 1: Difficulty-Controlled, Diagnostic Evaluation via All-Pass / Avg-Pass Decoupling

The paper's most conceptually distinctive evaluation move is the deliberate decoupling of All-pass (100% checklist satisfaction) from Avg-pass (fraction of items satisfied), and using the gap between them as a diagnostic signal rather than a simple multi-metric reporting convention. This is not an incremental refinement of existing evaluation practice — it's a targeted reframing of what it means for a model to "fail" at planning.

Prior benchmarks overwhelmingly report aggregate success rates or per-requirement accuracy, which conflates two fundamentally different failure modes: (1) the model doesn't understand the task at all (All-pass = 0, Avg-pass ≈ 0), and (2) the model understands and satisfies many local constraints but produces a globally inconsistent plan (All-pass = 0, Avg-pass >> 0). Standard accuracy metrics collapse these into the same "failure" bin, making it impossible to distinguish genuine capability gaps from systematic integration failures.

PlanningBench's evaluation suite makes this distinction legible. The results in Table 2 are stark: GPT-5.4-medium reaches 90.03% Avg-pass but only 58.09% All-pass — a 31.94 percentage point gap. Seed-2.0-pro-high reaches 84.02% Avg-pass but only 44.33% All-pass. These gaps are not noise; they are the signal. They tell you that these models reliably satisfy most local requirements — they respect time windows, obey capacity limits, follow output formats — but they fail to integrate these locally correct decisions into a globally valid plan because of a small number of critical errors that cascade.

The significance is that this gap provides a fine-grained diagnostic axis that prior benchmarks cannot offer. A benchmark that only reports a single score (e.g., "42% accuracy on TravelPlanner") tells you how often the model succeeds but not why it fails when it does. Is the model completely lost? Is it mostly right but messes up one constraint? Is it systematically bad at a particular constraint type? The All-pass/Avg-pass gap answers: a large gap implies the model is broadly competent but struggles with the coordination overhead of multi-constraint planning; a small gap implies the model is either consistently successful or consistently failing.

The error analysis in Table 3 corroborates this diagnostic value. Across all models, "Wrong Calculation / Assignment" dominates at 60.9%–83.5% of semantic failures, while "Format / Structure" accounts for only 0.7%–3.4%. This tells us that the bottleneck is not output formatting compliance (which many instruction-following benchmarks primarily test) but the ability to make correct numerical, temporal, scheduling, and allocation decisions under coupled constraints. The paper's evaluation thus reveals what kind of planning failure dominates, not just how often failure occurs.

The comparison to prior work is instructive. TravelPlanner (Xie et al., 2024) uses delivery rate and commonsense constraint pass rates, which provide some disaggregation but don't structurally separate complete-plan success from partial-constraint satisfaction in the same way. ChinaTravel (Shao et al., 2024a) reports multiple metrics across difficulty tiers but similarly lacks a clean binary all-or-nothing pass indicator at the instance level. PlanningBench's All-pass metric is conceptually closer to the "strict accuracy" used in mathematical reasoning benchmarks (where the answer either matches or doesn't), adapted to the planning domain where verification requires checking multiple coupled conditions simultaneously.

This is a fundamental diagnostic advance rather than an incremental metric tweak because it changes what question evaluation answers. A standard benchmark asks "how good is this model at planning?" PlanningBench's design asks "what is the nature of this model's planning failures, and where should we direct improvement efforts?" The answer — that models struggle with global constraint integration even when they satisfy local requirements — has direct implications for training priorities (focus on coordination, not format following) that a single aggregate score would not provide.

Innovation 2: The Verification-Signal Quality Hypothesis — Determinate Optima as a Prerequisite for Transferable Planning Training

The most consequential conceptual contribution of this paper may be its negative result and the hypothesis it generates: that for planning-oriented reinforcement learning to produce transferable skills, the verification signal must provide clear directional gradients toward a unique correct solution, and that permissive or partially-correct rewards from loose answer spaces can actively degrade general capabilities.

This is not a small refinement of existing RL training methodology. It's a claim about the structure of training signals required for complex constrained-generation tasks — a claim that challenges the prevailing assumption in the LLM RL literature that "more diverse training data with approximately correct rewards" is always beneficial.

Prior assumption in the field. The dominant paradigm in RL-based LLM training — from RLHF to GRPO to DPO — assumes that reward signals need not be perfect to be useful. The literature on reward hacking and over-optimization (Gao et al., 2022; Casper et al., 2023) has established that imperfect rewards can be exploited, but the standard remedy is to constrain optimization (KL penalties, early stopping, reward model ensembles) rather than to insist on near-perfect reward determinacy in the training data itself. In the specific domain of verifiable rewards for reasoning, DeepSeek-R1 (Guo et al., 2025) demonstrated that exact-match verification on math and coding problems produces strong reasoning improvements, but this was interpreted as "verifiable rewards work" rather than "only determinate verifiable rewards work."

What PlanningBench discovered. The paper reports two direct comparisons that isolate the effect of reward determinacy:

1. The Syn-NotDetOptimal ablation (Table 5): Training on 300 synthetic planning instances without determinate optimality produced only a marginal average gain of +0.75 points across Multi-Challenge, Inverse IFEval, and Collie, and slightly decreased performance on Inverse IFEval (-0.70 points). In contrast, Syn-PlanningBench (same size, same pipeline, but with determinate optimality) produced +7.06 points average improvement, including a dramatic +14.84 point gain on Collie.

2. The early-training negative result (Section 3.6): An early batch of approximately 1,000 synthesized examples that did not emphasize determinate optimality caused GRPO training to actually degrade general instruction-following performance. The paper's language is unusually direct about this failure: "GRPO training did not yield clear gains and degraded performance on general instruction-following benchmarks."

These are not subtle effects. The difference between +0.75 and +7.06 average points (a nearly 10× gap in improvement magnitude) when the only systematic difference is the determinacy of the optimal solution is evidence of a fundamental mechanism, not a hyperparameter tuning artifact.

The conceptual mechanism proposed. The paper hypothesizes that the gradient structure of the reward landscape matters more than the reward's average correctness. When a planning task admits many partially-feasible solutions (loose optima), the reward function is flat in many directions: small perturbations to the plan don't clearly change the reward because many variations are equally "mostly correct." The model receives positive but diffuse signals — it learns that satisfying 80% of constraints earns high reward, and it optimizes for that proxy rather than for complete global consistency. The gradient from such a signal points toward "satisfy more checklist items individually," not toward "construct a coherent global plan where every decision is consistent with every other decision."

When the task has a determinate optimum, the reward function is sharp: only one solution (or a narrow equivalence class) earns maximum reward. The gradient points directly toward that solution, and any deviation from the optimum — even a "mostly correct but one constraint wrong" plan — produces a clear negative signal. The model cannot succeed by satisfying 80% of constraints; it must satisfy 100%. This forces the model to learn the coordination and integration skills that distinguish planning from simpler constrained generation.

Training dynamics evidence. Figure 4 provides independent corroboration of this mechanism. The Syn-PlanningBench training curve shows: (a) the solve-none ratio decreases fastest and converges to the lowest level (the model learns to stop producing completely failed plans), (b) the solve-all ratio increases more consistently (the model learns to produce fully correct plans, not just partially correct ones), and (c) the critic reward curve is smoother. The Syn-NotDetOptimal curve shows the opposite: a persistently low solve-all ratio and weaker dynamics. The fact that Syn-NotDetOptimal's solve-all ratio remains low despite the model receiving positive rewards for partial solutions confirms the paper's interpretation: the model learned to optimize the partial-credit proxy, which did not translate into complete solution capability.

Why this is fundamental, not incremental. This finding reframes the design space for RL-based training on complex constrained tasks. The field's current conversation around "verifiable rewards" assumes that verifiability is a binary property — either you can check correctness or you can't — and that verifiable rewards are always better than learned reward models. PlanningBench suggests that verifiability is a spectrum: verification that produces a sharp, determinate signal (exact-match, all-constraints-satisfied) is qualitatively different from verification that produces a diffuse, permissive signal (partial-credit scoring, approximate matching). The former can produce transferable skills; the latter may produce models that are worse than before training.

This has immediate implications for practitioners building RL training pipelines. If you're generating synthetic data for planning-oriented training, the paper suggests you should deliberately constrain the problem space toward determinate optima — make the tasks harder in the sense of having unambiguous correctness criteria — rather than generating diverse problems with loose answer spaces. This is somewhat counterintuitive: the instinct in synthetic data generation is often to maximize diversity, but PlanningBench suggests that diversity without determinacy may be counterproductive.

The paper appropriately caveats this finding ("more systematic experiments are still needed"), and it's important to note that the mechanism is hypothesized rather than proven — the paper demonstrates correlation between determinate optimality and training success but doesn't isolate it as the sole causal factor. Nevertheless, the empirical pattern is strong enough to warrant treating reward-signal determinacy as a first-class design consideration in planning-oriented training, which is a conceptual contribution that extends beyond PlanningBench itself.

Innovation 3: Scalable, Verifiable Planning Data as a Generation Target Rather Than a Collection Target

The most foundational conceptual move in this paper is its reframing of planning data construction from benchmark collection to controllable synthesis. This is not a methodological contribution (the specific synthesis pipeline is a mechanism) but a conceptual one: it changes what the field considers to be a planning dataset and opens a design space that didn't previously exist.

The prior paradigm: benchmark-as-collection. Every planning benchmark the paper cites — TravelPlanner, ChinaTravel, DeepPlanning, Compass, CostBench, and the rest — treats a planning dataset as a fixed, hand-designed collection of instances. The benchmark creator writes a set of planning problems, designs evaluation metrics, and releases the result. Scaling the benchmark means writing more problems. Controlling difficulty means sorting problems by some proxy (prompt length, number of constraints) and labeling them easy/medium/hard. Covering new domains means writing problems for each domain individually. In this paradigm, the benchmark is the data, and "more data" requires "more human effort."

The proposed paradigm: data-as-generative-output. PlanningBench treats planning instances as the output of a structured generative process parameterized by a taxonomy of tasks and constraints. The data is not a fixed set; it's a sample from a designed distribution over the task-constraint space. Scaling means increasing the sample size. Controlling difficulty means adjusting the sampling distribution over constraint tiers (as the update formula in Equation 1 operationalizes). Covering new domains means adding new task types and constraint pools to the taxonomy — all existing synthesis machinery then applies to the new domain without additional per-instance engineering.

What makes this a conceptual contribution rather than an engineering one. The shift from collection to synthesis is deceptively simple to state but requires reconceptualizing several aspects of planning data:

• Difficulty becomes a controllable parameter, not an observed property. In a collection paradigm, difficulty is something you measure after creating instances — you sort by length, by number of constraints, by observed model accuracy. In a synthesis paradigm, difficulty is something you specify before creating instances — you choose constraint compositions that encode specific difficulty profiles. The closed-loop mechanism (where the Responder's success triggers difficulty escalation) makes difficulty adaptive to a reference model's capability frontier, which is impossible in a collection paradigm because the instances are fixed before any model sees them.

• Verification becomes a design-time property, not a post-hoc annotation. In a collection paradigm, verification rules are created alongside evaluation instances, often as software checkers or human-evaluation rubrics. In a synthesis paradigm, verification is generated simultaneously with the problem as part of the same synthesis process — the checklist c_i is an output of the Generator, not a separate annotation step. This means the verification is inherently aligned with the problem's constraints because they share a common origin in the sampled constraint configuration.

• Taxonomic coverage becomes a measurable property, not an aspirational goal. In a collection paradigm, benchmark creators try to cover "diverse scenarios" by writing instances across multiple domains. But coverage is ad-hoc — you cover the domains you thought of. In a synthesis paradigm, the taxonomy defines the coverage space: the six task families, 30+ task types, and their associated constraint pools constitute a formal specification of what planning structures are included. You can measure how many task-constraint configurations have been sampled, identify gaps, and adjust the generation process accordingly.

Why this matters beyond PlanningBench. The paper's reframing opens a path toward planning data that scales with compute rather than with human effort. Just as the field moved from hand-crafted NLP datasets (e.g., MUC for information extraction, early translation benchmarks) to programmatically generated ones (e.g., SQuAD via Wikipedia parsing, synthetic math datasets), PlanningBench suggests that planning data can be systematically generated from a structured design space. The 467-instance evaluation set and 300-instance training set in this paper are initial samples from that space; the framework can produce orders of magnitude more data without additional human annotation effort (though the paper currently applies human quality control, which is a bottleneck — see Innovation 4 below on that tension).

The contrast with Human-Authored data in Tables 4 and 5 provides evidence that this conceptual shift has practical consequences. Human-Authored instances (independently written by the same annotators without using the taxonomy or synthesis pipeline) produce smaller training gains than Syn-PlanningBench instances: on TravelPlanner, Human-Authored achieves 33.86% average All-pass versus 46.86% for Syn-PlanningBench; on the general benchmarks, Human-Authored achieves 41.81% versus 45.80%. These are instances of comparable size (300 each) written by the same pool of annotators. The only systematic difference is that Syn-PlanningBench instances are constructed through the taxonomy-guided, constraint-composition approach, while Human-Authored instances are hand-crafted without this structure. The results suggest that structural composition — sampling from a designed constraint space — produces data with different (and arguably more transferable) properties than human intuition alone.

This is a fundamental shift in how the field thinks about planning data, not an incremental improvement in benchmark coverage. It's comparable to the shift from collecting human-written math problems to generating them procedurally (as in GSM8K vs. synthetic math datasets), but applied to a much more structurally complex domain where constraint coupling makes naive template-based generation inadequate.

Innovation 4: Taxonomy-Guided Structural Difficulty Control Replacing Surface-Level Proxies

Existing planning benchmarks control difficulty through surface-level proxies: prompt length, number of requirements, number of conversation turns, or interaction depth. PlanningBench replaces these with taxonomy-guided structural difficulty — difficulty tiers defined by the kind of constraint (basic feasibility vs. optimization tradeoffs vs. meta-planning requirements like robustness and infeasibility detection) rather than the quantity of constraints.

This is a conceptual innovation in how difficulty is defined and controlled for planning tasks. The three difficulty tiers (basic, medium, hard) in PlanningBench correspond to qualitatively different planning challenges: basic constraints test whether the model can produce any feasible plan; medium constraints test whether the model can optimize and balance competing objectives; hard constraints test whether the model can reason about its plan (recognizing infeasibility, designing robustness, recovering from disruptions). This is not a quantity gradient — it's a capability gradient.

The difference from prior work. In TravelPlanner (Xie et al., 2024), harder instances involve more constraints or more complex itineraries, but the type of planning reasoning required doesn't fundamentally change — it's always "construct a valid itinerary under constraints." In ChinaTravel (Shao et al., 2024a), the Easy/Medium/Human subsets primarily reflect territory coverage and constraint density. In TripCraft (Chaudhuri et al., 2025), difficulty relates to spatiotemporal granularity. None of these benchmarks test whether a model can recognize that a problem is infeasible (a hard-level constraint in PlanningBench) as distinct from whether it can solve a feasible problem with many constraints. None test whether a model can design a plan that remains robust under perturbations (another hard-level constraint). These are qualitatively different planning skills, not just quantitatively harder instances of the same skill.

The evidence that this tier structure captures genuine difficulty variation — rather than just labeling harder instances as "hard" — comes from the evaluation results in Figure 3 and the performance breakdown by task type, prompt length, and rubric count. These analyses show that PlanningBench difficulty correlates with (but is not determined by) surface-level factors like prompt length and constraint count. Some short-prompt, low-rubric-count instances are hard because their constraints are tightly coupled, while some long-prompt, high-rubric-count instances are easier because their constraints are largely independent. The paper's ability to distinguish these cases demonstrates that the taxonomic structure captures something that surface-level proxies miss.

Why this matters for diagnosis. A benchmark that controls difficulty structurally rather than superficially can answer different questions about model capability. If a model fails on "hard" instances in a surface-level benchmark, you know it struggles with high information load — but not whether it struggles with constraint coupling, optimization under conflict, robustness reasoning, or infeasibility detection. PlanningBench's tier structure allows this disaggregation: if a model succeeds on basic and medium instances but fails on hard ones, the failure is specifically in meta-planning capabilities (robustness, recovery, infeasibility detection) rather than in constraint satisfaction generally.

The operationalization of this concept. The exponential update mechanism in Equation 1 — which shifts the sampling distribution from basic toward medium and hard constraints when the Responder solves instances — is the mechanism that makes this structural difficulty control adaptive. But the conceptual innovation is in the design of the tiers themselves: the recognition that planning difficulty decomposes into feasibility, optimization, and meta-reasoning, and that these can be independently sampled and composed. The paper doesn't just say "here are easy, medium, and hard problems." It says "here are three orthogonal sources of planning difficulty, and we can control the mixture."

This is a fundamental contribution to planning evaluation methodology, not an incremental improvement in benchmark construction. It provides a language for talking about why planning is hard — constraint coupling, objective conflict, robustness requirements — that is grounded in a taxonomy rather than in post-hoc observation of model failure patterns.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The evaluation set consists of 467 planning instances generated through the PlanningBench pipeline and audited through human quality control. The paper constructs this set by using "strong models as the Generator to inform the human annotation process," with Qwen-A3B-30B serving as the Responder and GPT-oss-120b as the Critic during synthesis. The 467-instance set spans six task families and 30+ task types, with each instance including a self-contained planning problem and a verification checklist.

• Base model(s). The paper evaluates 18 models spanning both closed-source and open-source families. Closed-source models include GPT-5.4-xhigh, GPT-5.4-high, GPT-5.4-medium, Gemini-3-1-pro, and Gemini-2.5-pro. Open-source models include Seed-2.0-pro-high, DeepSeek-V3.2-thinking, Hy3-Preview, Qwen3.5-plus-thinking, DeepSeek-R1, Seed-2.0-pro-medium, DeepSeek-V3.2-exp, Hy2.1, Seed-1.8, Qwen3-30b-moe, Qwen3-32b, Qwen3-14b, and Qwen3-8b. For training experiments, the base model is Qwen-A3B-30B. The paper argues that the training model choice reflects a capability level where planning improvements are both measurable and practically meaningful.

• Metrics. The paper reports two complementary metrics. All-pass measures the fraction of instances where a model's output satisfies every item in the verification checklist — a binary complete-solution success rate under coupled constraints. Avg-pass measures the average fraction of checklist items satisfied across all instances, capturing partial progress toward valid plans. The gap between these two metrics is itself treated as a diagnostic signal: a large All-pass/Avg-pass gap indicates that models satisfy many local constraints while failing to produce globally consistent plans. The verification is performed by GPT-oss-120b as a judge model, which assesses each model output against its instance-specific checklist following the rubric-based evaluation protocol described in Appendix G.2.

• Baselines. For the evaluation experiments, the paper does not use external baseline methods — it directly compares 18 models against the PlanningBench evaluation suite. For training experiments (Section 4.2), four configurations are compared: Base Model (Qwen-A3B-30B without any reinforcement learning), Syn-PlanningBench (GRPO training on 300 PlanningBench instances constructed with determinate optimality), Syn-NotDetOptimal (GRPO training on 300 synthetic planning instances without determinate optimality emphasis), and Human-Authored (GRPO training on 300 planning instances independently written from scratch by the same annotator pool without using the PlanningBench taxonomy or synthesis pipeline). Trained models are evaluated on external benchmarks: ChinaTravel (Shao et al., 2024a), TravelPlanner (Xie et al., 2024), Multi-Challenge (Deshpande et al., 2025), Inverse IFEval (Zhang et al., 2025), and Collie (Yao et al., 2023).

• Generation budget / compute accounting. The paper does not measure test-time compute in terms of FLOPs or generation count during evaluation — all models are queried with default inference parameters and evaluated on their single-pass outputs without search, revision, or sampling strategies. In the training experiments, compute is accounted for through GRPO configuration: 8-way rollout sampling per training instance, batch size 128, 50 epochs, actor learning rate 2 × 10⁻⁶, and KL coefficient 0.001. During inference after training, models use temperature = 0.7, top-p = 0.6, and top-k = 20.

• Cross-validation / statistical protocol. For training experiments, the paper reports statistical significance testing at p < 0.05 (indicated by † in Tables 4 and 5), though the specific test used is not detailed. The evaluation experiments do not report confidence intervals or cross-validation, as the 467-instance evaluation set is treated as a fixed benchmark. The training datasets (300 instances each) are controlled for size across the three RL conditions to ensure comparability.

Main Quantitative Results

PlanningBench as an Evaluation Suite: Model Performance

The headline finding from Table 2 is that PlanningBench remains challenging even for the strongest evaluated models. GPT-5.4-xhigh achieves the best All-pass at 63.17%, meaning it fails to satisfy the full verification checklist on more than one-third of the evaluation set. The second-best model, GPT-5.4-high, reaches only 58.56% All-pass. This is a benchmark that is far from saturated even at the frontier of LLM capability.

The All-pass rankings reveal a clear capability hierarchy. Closed-source models dominate the top positions: GPT-5.4-xhigh (63.17%), GPT-5.4-high (58.56%), GPT-5.4-medium (58.09%), Gemini-3-1-pro (53.25%), and Gemini-2.5-pro (31.70%) occupy positions 1–4 and 10, respectively. The strongest open-source models — Seed-2.0-pro-high (44.33%), DeepSeek-V3.2-thinking (37.13%), Hy3-Preview (36.17%), and Qwen3.5-plus-thinking (34.03%) — form a clear middle tier, with a substantial gap of roughly 9–19 points below the GPT-5.4 variants.

At the lower end, the capability drop-off is sharp. Qwen3-30b-moe achieves only 12.15% All-pass. Qwen3-32b drops to 0.27%. Qwen3-14b and Qwen3-8b both achieve exactly 0.00% All-pass, meaning they cannot produce a single instance that satisfies every checklist item across the 467-instance evaluation set. These models are not entirely failing — their Avg-pass scores of 30.11%, 25.69%, and 22.79% respectively show they satisfy some local requirements — but they are incapable of producing a globally consistent plan under the All-pass criterion.

The All-pass/Avg-pass gap is the paper's primary diagnostic tool, and Table 2 reveals its diagnostic value. GPT-5.4-medium reaches 90.03% Avg-pass but only 58.09% All-pass — a gap of 31.94 percentage points. Seed-2.0-pro-high reaches 84.02% Avg-pass but only 44.33% All-pass — a 39.69 point gap. GPT-5.4-xhigh achieves the highest All-pass (63.17%) and the highest Avg-pass (92.35%), giving it the smallest gap (29.18 points) among top models, but the gap is still nearly 30 points. These are not marginal effects — they indicate that even the best models systematically fail to convert high local-constraint satisfaction rates into complete planning success. The paper's interpretation is that "models often satisfy many checklist items while still failing to produce a globally valid plan because of a small number of critical errors," and the All-pass/Avg-pass gap quantifies exactly how often this occurs.

A subtle pattern in the rankings: Avg-pass does not perfectly predict All-pass. GPT-5.4-medium scores 90.03% Avg-pass (higher than GPT-5.4-high's 84.60%) but achieves a slightly lower All-pass (58.09% vs. 58.56%). Gemini-3-1-pro scores 88.36% Avg-pass but 53.25% All-pass — a much larger gap (35.11 points) than GPT-5.4-xhigh's 29.18-point gap, suggesting that Gemini-3-1-pro is relatively better at satisfying individual constraints but relatively worse at integrating them into a globally consistent plan. This inversion — where a higher Avg-pass model scores lower on All-pass — demonstrates that the two metrics capture genuinely different aspects of planning capability, not just different thresholds on the same underlying ability.

For weaker models, the pattern shifts. Qwen3-32b, Qwen3-14b, and Qwen3-8b show All-pass scores near or at zero but Avg-pass scores of 30.11%, 25.69%, and 22.79% — indicating that they still satisfy roughly one-quarter to one-third of checklist items. However, the sharp drop-off suggests a threshold effect: below some capability level, models cannot coordinate constraints well enough to produce even a single complete success, even though they can handle individual requirements in isolation.

Task-Specific Performance Analysis

Figure 3a breaks down All-pass performance across five representative task types — Study Plan, Power Dispatch Planning, Hiring Plan, Material Allocation, and Student Grouping — for five models at different capability levels. The results reveal substantial task-specific variation in model performance that is not captured by aggregate scores.

The cross-model spread is large within each task type. On Hiring Plan, performance ranges from GPT-5.4-xhigh at 80.0% down to 6.7% — a 73.3-point gap. On Power Dispatch Planning, the range is 53.9% to 7.1% (46.8 points). Even on Student Grouping, where the spread is narrower, it spans 72.7% to 45.5% — a 27.2-point gap. This wide variation confirms that PlanningBench captures task-dependent capability differences: a model's aggregate score reflects performance across structurally distinct planning challenges, not a single generalized planning ability.

Models also show distinct task-specific profiles — they are not uniformly better or worse across all tasks. GPT-5.4-xhigh performs best on Hiring Plan (80.0%) and Material Allocation (69.2%) but drops to 47.4% on Study Plan. Gemini-3-1-pro achieves its highest score on Student Grouping (72.7%) but only 36.8% on Study Plan. Qwen-3.5-plus-thinking performs relatively better on Student Grouping (45.5%) than on Power Dispatch Planning (7.1%). These profile differences suggest that different model architectures or training recipes produce different patterns of planning strengths and weaknesses — a level of diagnostic granularity that single-task benchmarks cannot provide.

The task types themselves appear to have different difficulty profiles independent of model capability. Study Plan shows the lowest top-model performance (GPT-5.4-xhigh at 47.4%), while Hiring Plan shows the highest (GPT-5.4-xhigh at 80.0%). This suggests that the structural complexity of different planning families — scheduling and timetabling (Study Plan) versus shift and workforce scheduling (Hiring Plan), versus allocation and matching (Material Allocation, Student Grouping) — creates genuinely different difficulty levels even for the strongest models.

Prompt Length and Checklist Item Count Analysis

Figures 3b and 3c examine whether surface-level factors — prompt length and number of checklist items — can explain the difficulty variation observed in PlanningBench. The answer is that they partially correlate with difficulty but do not fully determine it.

Prompt length (Figure 3b) shows a general downward trend in All-pass as prompt length increases. GPT-5.4-xhigh declines from 64.0% in the Very Short group (950–1817 tokens) to 50.6% in the Very Long group (2614–5908 tokens). Gemini-3-1-pro drops from 61.5% to 43.8% across the same range. Seed-2.0-pro-high declines from 46.7% to 34.0%. However, the trend is not strictly monotonic: Gemini-3-1-pro scores 61.5% on Very Short but 55.0% on Short, then 58.9% on Medium before dropping. DeepSeek-R1 shows a similar non-monotonic pattern. This suggests that prompt length contributes to difficulty — more information to integrate increases the chance of missing a constraint — but is not the dominant factor. Some long-prompt instances are easier than some short-prompt instances because constraint coupling, rather than information volume, is the primary source of difficulty.

Number of checklist items (Figure 3c) shows a clearer downward trend, particularly at the extremes. GPT-5.4-xhigh reaches 63.4% All-pass in the 2–6 item group but drops to 40.0% in the 15+ item group. Gemini-3-1-pro drops from 62.1% in the single-item group to 20.0% in the 15+ group. The 15+ item bucket is consistently the most challenging for every model, with scores dropping sharply relative to the 11–15 bucket (e.g., Seed-2.0-pro-high drops from 39.9% to 20.0%; DeepSeek-R1 drops from 30.9% to 10.0%). This suggests that as the number of constraints that must be jointly satisfied increases, the probability of satisfying all of them decreases faster than the probability of satisfying any individual one — which is exactly what we would expect if constraint coupling (the interdependency between constraints) drives difficulty rather than each constraint contributing independently.

However, the single-item group is also notably challenging for some models: Gemini-3-1-pro scores 62.1% on single-item instances but 58.5% on 2–6 item instances — a smaller gap than expected. Qwen-3.5-plus-thinking scores 36.8% on single-item versus 39.0% on 2–6 items, actually performing slightly better with more checklist items. This suggests that single-checklist-item instances in PlanningBench are not trivially easy — they may involve tightly coupled constraints where the single checklist item subsumes multiple requirements, making verification all-or-nothing rather than additive.

The overall pattern from these analyses supports the paper's claim that PlanningBench difficulty "reflects both information load and the need to jointly satisfy multiple constraints while preserving global coherence" — but with the crucial caveat that surface-level proxies (prompt length, item count) are correlates rather than determinants of difficulty. The structural composition of constraints — how they couple, conflict, and interact — creates difficulty variation that simple counting metrics cannot fully capture.

Error Analysis

Table 3 provides a semantic categorization of why models fail on instances where they do not achieve All-pass, excluding exact refusals and blank outputs. The dominant error type across every model is Wrong Calculation / Assignment, accounting for 60.9%–83.5% of semantic failures. This means that when models fail to produce a fully correct plan, the failure is overwhelmingly due to making incorrect numerical, temporal, scheduling, allocation, or logical decisions under constraints — not due to failing to follow the output format or failing to include required elements.

The magnitude of this dominance is striking. For the strongest model, GPT-5.4-xhigh, 65.5% of failures are Wrong Calculation / Assignment. For the weakest model analyzed, Qwen3-32b, 83.5% of failures fall into this category. The intermediate models cluster in the 67–75% range. This consistency across capability levels suggests that constrained reasoning — the ability to make correct decisions when multiple constraints interact — is the fundamental bottleneck, not a problem that stronger models have solved while weaker ones haven't.

Constraint Omitted is the second most common error type, accounting for 4.2%–19.9% of failures. Gemini-3-1-pro has the highest share at 19.9%, meaning roughly one-fifth of its failures involve missing required constraints or details. GPT-5.4-xhigh and Seed-2.0-pro-high show moderate rates at 15.1% and 14.8%. Qwen3-32b has the lowest at 4.2%, but this is partly because its error profile is dominated by Wrong Calculation / Assignment — it fails in more fundamental ways before getting to the point of omitting constraints.

State Tracking failures — where the model fails to preserve or update an evolving solution state — account for 5.3%–11.3% of errors. GPT-5.4-xhigh has the highest rate at 11.3%, which is notable because it suggests that even when the strongest model makes correct individual decisions, it sometimes loses track of the global plan state as it constructs the solution. This aligns with the All-pass/Avg-pass gap interpretation: the model can satisfy individual constraints at each step but fails to maintain the integrated state that ties all steps together.

Format / Structure violations are remarkably rare, accounting for only 0.7%–3.4% of failures. GPT-5.4-xhigh shows the highest rate at 3.4%, but even this is low. Most models cluster around 1–2%. This is an important negative finding: format compliance is not a meaningful bottleneck for current models on PlanningBench. The challenge is in the planning reasoning itself, not in following output schemas.

Missing Rationale errors — where required explanations, comparisons, or verification steps are absent — account for 4.6%–10.7% of failures. Seed-2.0-pro-high has the highest rate at 10.7%, suggesting it may be relatively better at constraint satisfaction but relatively worse at the meta-cognitive elements that PlanningBench's hard-level constraints require.

The model-specific patterns in Table 3 provide additional diagnostic granularity. Gemini-3-1-pro's high Constraint Omitted rate (19.9%) and relatively lower Wrong Calculation / Assignment rate (60.9%) suggest it tends to plan less completely — it makes correct decisions for the constraints it considers but misses constraints entirely. GPT-5.4-xhigh's high State Tracking rate (11.3%) suggests it attempts to handle all constraints but sometimes loses coherence across the plan's construction. Qwen3-32b's extremely concentrated error profile (83.5% Wrong Calculation / Assignment) suggests it attempts to address all constraints but simply makes incorrect decisions at a high rate.

Generalization to External Planning Benchmarks

Table 4 reports transfer performance from GRPO training on PlanningBench data to two external planning benchmarks, ChinaTravel and TravelPlanner. The headline result is that Syn-PlanningBench consistently improves over Base Model on both benchmarks, with TravelPlanner showing particularly large gains.

On ChinaTravel, Syn-PlanningBench improves average All-pass from 50.92% to 58.36% (+7.44 points) and average Avg-pass from 84.88% to 88.12% (+3.24 points). The All-pass gains are distributed across difficulty subsets: Easy improves from 14.63% to 29.12% (+14.49 points), Medium moves from 51.37% to 52.78% (+1.41 points), and the Human subset (ChinaTravel-Human) rises from 86.76% to 93.18% (+6.42 points). The substantially larger gain on Easy (+14.49) than on Medium (+1.41) suggests that PlanningBench training primarily improves the model's ability to handle problems within its capability range — it refines competence on tractable problems rather than enabling solution of previously intractable ones. This is consistent with the paper's earlier finding (from the evaluation experiments) that difficulty is strongly task-dependent.

On TravelPlanner, the transfer effect is substantially stronger. Average All-pass jumps from 28.85% to 46.86% (+18.01 points), and average Avg-pass increases from 80.84% to 91.93% (+11.09 points). The gains are largest on TP-Val: All-pass rises from 21.04% to 43.82% (+22.78 points). On TP-Train, All-pass rises from 29.80% to 48.84% (+19.04 points), and on TP-Test from 35.72% to 47.93% (+12.21 points). The fact that gains are substantial across all three TravelPlanner subsets — including the held-out TP-Test — suggests genuine generalization rather than overfitting to a particular data distribution.

The comparison with Human-Authored reveals the value of structured synthesis. Human-Authored produces smaller gains on both benchmarks: on ChinaTravel, average All-pass improvement is +1.49 points (vs. +7.44 for Syn-PlanningBench); on TravelPlanner, average All-pass reaches 33.86% (vs. 46.86% for Syn-PlanningBench). Notably, Human-Authored even shows a slight decrease on TravelPlanner TP-Test (34.91% vs. 35.72% for Base Model, a −0.81 point change) and a decrease on ChinaTravel Medium (−0.35 points). This means that 300 human-written planning instances — produced by the same annotator pool without using the taxonomy or synthesis pipeline — provide weaker and less consistent transfer than 300 taxonomy-guided synthetic instances. The paper's interpretation is that "Human-Authored provides valid training data, [but] its examples are not constructed through taxonomy-guided constraint composition or controlled difficulty progression," which limits their ability to teach transferable constraint-integration skills.

The statistical significance testing (p < 0.05, marked with †) shows that Syn-PlanningBench's improvements are significant on nearly all metrics: ChinaTravel Easy All-pass (+14.49), Medium All-pass (+1.41), Human All-pass (+6.42), and all TravelPlanner subsets. The exceptions are ChinaTravel Medium Avg-pass (+2.54, not marked significant) and ChinaTravel Human Avg-pass (+2.18, not marked) — these are relatively small gains on subsets where the base model already performs strongly (87.96% Avg-pass on Medium, 94.92% on Human).

Transfer to General Instruction-Following Benchmarks

Table 5 reports transfer performance on three general-purpose benchmarks that are not planning-specific but require instruction integration, constraint following, and multi-requirement coordination across complex responses. The headline result is that Syn-PlanningBench substantially outperforms both Base Model and Human-Authored, with a particularly dramatic gain on Collie.

On Multi-Challenge, Syn-PlanningBench achieves 33.09% (tied with Human-Authored), a +3.91 point improvement over Base Model at 29.18%. On Inverse IFEval, Syn-PlanningBench reaches 51.14% (+2.42 points over Base Model's 48.72%). On Collie, the improvement is dramatic: from 38.33% to 53.17% (+14.84 points). The average across all three benchmarks rises from 38.74% to 45.80% (+7.06 points).

The Syn-NotDetOptimal ablation is the critical comparison for the determinate-optimality hypothesis. Syn-NotDetOptimal achieves only +0.75 points average improvement over Base Model (39.49% vs. 38.74%), with a slight decrease on Inverse IFEval (−0.70 points). This is a near-null result: training on 300 synthetic planning instances without determinate optimality produces essentially no transfer to general instruction-following benchmarks. The comparison with Syn-PlanningBench's +7.06 point average gain confirms that determinate optimality — or whatever correlated property distinguishes the two data batches — is the driver of transfer, not the act of training on planning data per se.

The Human-Authored comparison shows a more nuanced pattern. Human-Authored achieves a +3.07 point average improvement, which is better than Syn-NotDetOptimal (+0.75) but substantially worse than Syn-PlanningBench (+7.06). Human-Authored's gains are concentrated on Multi-Challenge (+3.91) and Collie (+4.00), with only marginal improvement on Inverse IFEval (+1.28). This suggests that human-crafted planning data provides some transfer benefit — likely from teaching general constraint-satisfaction skills — but less than taxonomy-guided synthetic data that systematically composes constraints and controls difficulty.

The Collie benchmark deserves special attention. Collie (Yao et al., 2023) is designed to test systematic construction of constrained text generation tasks — it's the benchmark most closely aligned with PlanningBench's core competency of constraint integration and global consistency. The +14.84 point improvement on Collie is the largest single-benchmark gain in Table 5, and it dwarfs the gains on Multi-Challenge (+3.91) and Inverse IFEval (+2.42). This pattern suggests that PlanningBench training transfers most strongly to tasks that structurally resemble planning — those requiring multi-constraint coordination and global coherence — rather than providing a uniform boost to all instruction-following capabilities. The paper is appropriately measured in interpreting this: "These results do not imply that planning data improves all general capabilities uniformly. Rather, they suggest that verification-driven PlanningBench data can provide useful training signals for tasks where models must coordinate multiple requirements across a complex response."

Training Dynamics and the Role of Determinate Optima

Figure 4 presents training curves for three data conditions — Syn-PlanningBench, Human-Authored, and Syn-NotDetOptimal — across three metrics: solve-none ratio, solve-all ratio, and critic reward. These curves visualize the learning dynamics that underlie the transfer results in Tables 4 and 5.

Solve-none ratio (Figure 4a) measures the fraction of training instances where the model produces a plan that satisfies zero checklist items — a complete failure. Syn-PlanningBench shows the fastest decrease and converges to the lowest level. By the end of training, its solve-none ratio is visibly lower than both Human-Authored and Syn-NotDetOptimal. Human-Authored shows intermediate dynamics, decreasing more slowly than Syn-PlanningBench. Syn-NotDetOptimal shows the weakest improvement, with its curve staying above the other two throughout training. This means the determinate-optimal data most effectively teaches the model to avoid producing plans that are completely wrong — it learns to at least partially satisfy constraints even when it cannot achieve full success.

Solve-all ratio (Figure 4b) measures the fraction of training instances where the model produces a plan that satisfies every checklist item — complete success. This is the most important curve for the determinate-optimality hypothesis. Syn-PlanningBench's solve-all ratio increases more consistently and reaches a higher final level than either alternative. Human-Authored shows some improvement but plateaus at a lower level. Syn-NotDetOptimal maintains a "low solve-all ratio" throughout training, barely improving from its initial level. The paper interprets this as direct evidence for the mechanism proposed in Section 3.6: "partial checklist satisfaction does not reliably translate into complete solutions." When the reward signal is diffuse — rewarding partial constraint satisfaction — the model learns to satisfy individual constraints better (as reflected in the solve-none ratio decrease for Syn-NotDetOptimal) but fails to learn the coordination skills needed for complete planning success. Only data with determinate optima, where the reward function sharply distinguishes complete success from partial success, teaches the model to consistently produce globally valid plans.

Critic reward (Figure 4c) shows the overall reward signal received during training. Syn-PlanningBench's reward curve is "smoother, suggesting more stable optimization signals." Both Human-Authored and Syn-NotDetOptimal show noisier curves with more variance. Smoother reward curves in RL training typically indicate that the optimization landscape has clearer gradients — the model can reliably determine which actions improve reward and which don't — which aligns with the paper's argument that determinate optima provide "clearer reward direction for complete planning success."

The convergence pattern across all three metrics tells a consistent story: Syn-PlanningBench trains faster (solve-none decreases fastest), trains better (solve-all increases highest), and trains more stably (reward curve is smoothest). Human-Authored occupies an intermediate position. Syn-NotDetOptimal fails to meaningfully improve the solve-all ratio despite receiving positive rewards for partial solutions. This pattern directly supports the paper's central training claim: "reward determinacy is important for stable and transferable planning-oriented reinforcement learning."

Ablation Studies and Robustness Checks

Synthetic vs. human-authored training data (Tables 4, 5): The Human-Authored comparison functions as an ablation of the synthesis pipeline itself — does taxonomy-guided constraint composition matter, or is any diverse planning data sufficient? Human-Authored (300 instances written from scratch by professional annotators without using the taxonomy or pipeline) produces substantially smaller transfer gains than Syn-PlanningBench on both external planning benchmarks (+1.49 vs. +7.44 average All-pass on ChinaTravel; 33.86% vs. 46.86% average All-pass on TravelPlanner) and general benchmarks (+3.07 vs. +7.06 points average). This demonstrates that structured, taxonomy-guided synthesis produces training data with different (and more transferable) properties than human-crafted instances of comparable size — a non-obvious finding since the natural assumption would be that expert-written data should be at least as good as synthetic data.

Determinate vs. loose optima (Table 5, Figure 4): Syn-NotDetOptimal ablates the determinate-optimality design choice. The near-null transfer result (+0.75 average points on general benchmarks, with a slight decrease on Inverse IFEval) and the flat solve-all ratio in Figure 4b provide the paper's strongest evidence that reward-signal determinacy is a first-order factor in planning-oriented training. This ablation is important because it isolates a specific property of the training data (optimality determinacy) rather than the synthesis process as a whole — both Syn-PlanningBench and Syn-NotDetOptimal use the same pipeline, task taxonomy, and constraint pools; the systematic difference is in how tightly the solution space is constrained to a unique or well-specified optimum.

Task type variation (Figure 3a): The task-specific performance breakdown demonstrates that PlanningBench's difficulty is not uniform across task types — different planning structures produce genuinely different challenge levels for the same model. This ablation validates the taxonomy design: if all task types produced similar scores, it would suggest that the taxonomy captures only superficial domain variation rather than structural planning complexity. The wide score ranges within each task (e.g., 80.0% to 6.7% on Hiring Plan) confirm that different models also have different task-specific strengths, which a single-task benchmark would fail to detect.

Prompt length and checklist item count (Figures 3b, 3c): These analyses serve as ablations of surface-level difficulty proxies. If prompt length or item count perfectly predicted All-pass, then PlanningBench's structural difficulty control would be unnecessary — you could simply sort instances by length. The non-monotonic patterns in Figure 3b (e.g., Gemini-3-1-pro scoring higher on Medium-length prompts than Short-length prompts) and the counterintuitive single-item performance in Figure 3c demonstrate that structural factors — constraint coupling, dependency chains, optimization tradeoffs — drive difficulty beyond what surface-level metrics capture.

Model scale ablation (Table 2): The evaluation includes models spanning a wide parameter range within the Qwen3 family — Qwen3-30b-moe (12.15% All-pass), Qwen3-32b (0.27%), Qwen3-14b (0.00%), Qwen3-8b (0.00%). This within-family scaling comparison shows that PlanningBench discriminates capability even among architecturally similar models, and that there appears to be a sharp capability threshold below which All-pass becomes essentially zero (Qwen3-32b is very close to this threshold at 0.27%). However, this is not a controlled scaling experiment — the models differ in architecture details beyond parameter count — so claims about scaling laws should be interpreted cautiously.

Critical Assessment

Does PlanningBench Actually Test Planning Ability?

The paper's central evaluation claim is that PlanningBench "forms a challenging and discriminative evaluation suite" that tests whether models "can integrate all provided information into an executable and verifiable plan, rather than only satisfy isolated local requirements." The evidence for this claim is:

What the experiments show: The All-pass/Avg-pass gap in Table 2 demonstrates that models frequently satisfy most checklist items while failing to satisfy all of them. GPT-5.4-medium's 90.03% Avg-pass but 58.09% All-pass (a 31.94-point gap) is the clearest example. The error analysis in Table 3 shows that failures are dominated by Wrong Calculation / Assignment (60.9%–83.5%) rather than Format / Structure (0.7%–3.4%), indicating that the bottleneck is constrained reasoning rather than output compliance. The task-specific variation in Figure 3a shows that different task types produce genuinely different difficulty profiles for the same models.

What the experiments don't show: The paper doesn't demonstrate that PlanningBench's structural difficulty tiers (basic/medium/hard) predict model performance better than surface-level proxies. The analyses in Figures 3b and 3c show that prompt length and item count correlate with difficulty, but the paper doesn't quantify the incremental predictive value of structural constraint composition over these simpler metrics. A stronger claim would require showing that two instances with the same prompt length and checklist item count but different constraint compositions produce significantly different model performance — the paper implies this (from the non-monotonic patterns) but doesn't test it directly.

Additionally, the paper doesn't validate that the 467-instance evaluation set is representative of the broader task-constraint space. The synthesis pipeline generates instances from a designed distribution, and the evaluation set is a sample from that distribution after human quality control. But we don't know whether the 467 instances adequately cover the 30+ task types, their subtasks, and the constraint pools — a set of 467 instances across 30+ task types means roughly 15 instances per task type on average, which may be too sparse for per-task evaluation (the task-specific analysis in Figure 3a only examines five tasks). The paper doesn't report the number of instances per task type in the evaluation set.

Does PlanningBench Training Transfer to Unseen Planning Problems?

The paper claims that "training on PlanningBench improves performance on unseen planning benchmarks and broader instruction-following and reasoning tasks." The evidence is in Tables 4 and 5.

What the experiments show: Syn-PlanningBench produces statistically significant improvements on ChinaTravel (average All-pass +7.44 points), TravelPlanner (+18.01 points), and the three general benchmarks (+7.06 points average). The TravelPlanner improvements are particularly large and consistent across TP-Train, TP-Val, and TP-Test. The gains over Human-Authored demonstrate that structured synthesis provides benefits beyond what equally-sized human-crafted data provides.

What the experiments don't show: The paper does not test whether PlanningBench training transfers to planning benchmarks outside the travel domain. Both ChinaTravel and TravelPlanner are travel-planning benchmarks (as their names indicate). PlanningBench covers 30+ task types across six families, only one of which (Routing and Travel) is travel-related. If PlanningBench training genuinely teaches general planning skills, we should expect transfer to non-travel planning benchmarks — for example, PlanBench (Valmeekam et al., 2023), which tests planning and reasoning about change in blocks-world domains, or DeepPlanning (Zhang et al., 2026), which covers long-horizon agentic planning. The paper doesn't evaluate on any non-travel planning benchmark, so the claim of "planning-specific transfer" is supported only for transfer to travel planning specifically.

Additionally, the transfer evaluation uses Qwen-A3B-30B as the base model. We don't know whether the same training protocol would produce gains on other base models — the choice of base model could interact with the training data's effectiveness. A replication on at least one other model family would strengthen the generalizability claim.

The paper also doesn't evaluate whether PlanningBench training produces improvements above what a simpler baseline would achieve. For example, would training on 300 examples of travel-planning-specific data (sampled from ChinaTravel or TravelPlanner's own training distributions) produce larger transfer to those benchmarks than PlanningBench's cross-task training? This "direct training on the target distribution" baseline is missing. The paper demonstrates that PlanningBench training helps on external benchmarks, but not that it helps more than training on the target benchmark's own data would.

Does Determinate Optimality Actually Matter for Training?

The paper's strongest and most interesting claim is that "determinate or well-specified optimal solutions provide clearer reward signals and more stable training dynamics" and that loose answer spaces can produce diffuse rewards that fail to teach planning skills. The evidence is in Table 5 and Figure 4.

What the experiments show: Syn-NotDetOptimal produces near-null transfer (+0.75 points average) compared to Syn-PlanningBench (+7.06 points). Figure 4b shows that Syn-NotDetOptimal's solve-all ratio barely improves during training, while Syn-PlanningBench's rises consistently. This is a clear and interpretable pattern.

What the experiments don't show — and why this matters: The paper cannot fully isolate determinate optimality as the causal factor. Syn-NotDetOptimal is described as "synthetic planning data that does not emphasize determinate optimality to the same extent" — this is a relative comparison, not a clean ablation. The two data batches may differ in other systematic ways: the determinate-optimal data may also have tighter constraint coupling, fewer degrees of freedom in solution construction, or different distributions across task types and difficulty tiers. The paper acknowledges this caveat ("although more systematic experiments are still needed") but the current evidence is correlational, not causal.

A clean ablation would require: take the same set of planning problems, generate two versions of the verification checklist — one with a determinate optimality requirement (all solutions must match the unique optimum) and one with a loose verification (any solution satisfying core constraints passes) — and compare training outcomes. This would isolate the effect of verification determinacy while holding the problem distribution constant. The current experiment compares two different batches of data that differ in optimality emphasis but likely differ in other properties as well.

The paper also doesn't provide a quantitative definition of "determinate optimality." Is it a binary property (unique optimum vs. multiple optima)? Or a continuous one (how concentrated the solution space is)? The 300 Syn-PlanningBench instances are described as having a "preference" for determinate optima, but we don't know what fraction actually have unique solutions, how "well-specified" the non-unique ones are, or how this compares to the Syn-NotDetOptimal batch. Without this quantification, the claim is difficult to operationalize for practitioners who want to apply the insight to their own data generation.

Are the Training Gains Robust and Meaningful?

The training experiments use 300 instances for each condition, base model Qwen-A3B-30B, and 50 epochs of GRPO with 8-way rollout sampling. Several aspects of this setup warrant scrutiny:

Small training set size: 300 instances is a small RL dataset by contemporary standards. The fact that Syn-PlanningBench produces substantial transfer gains with only 300 instances is impressive, but it also raises questions about ceiling effects — would 600 or 1,200 instances produce proportionally larger gains, or do the gains saturate quickly? The paper's early experiment with ~1,000 instances (mentioned in Section 3.6) produced negative results, suggesting that data quality rather than quantity is the binding constraint, but no systematic data-scaling experiment is reported.

Single base model: All training experiments use Qwen-A3B-30B. We don't know whether the same training protocol on the same data would produce gains on GPT-5.4, DeepSeek, or Gemini models — or whether the determinate-optimality effect would replicate across model families. The base model's starting capability level may interact with the training data's effectiveness: a stronger base model might benefit more (because it can already produce approximately correct plans and just needs refinement) or less (because it's already near ceiling on the skills PlanningBench teaches).

No test-time compute scaling: The paper evaluates trained models with a single inference pass (temperature 0.7, top-p 0.6, top-k 20). It does not test whether trained models benefit from test-time compute strategies (best-of-N, majority voting, revision) on planning benchmarks. This is a notable omission given the paper's focus on planning-oriented training — if the training teaches better proposal distributions, we might expect the trained model to benefit more from test-time verification than the base model. Conversely, if the training primarily improves the model's ability to produce correct plans in a single pass, test-time compute might provide diminishing returns.

The 300-vs-300 comparison may not be fair in the intended way: Syn-PlanningBench uses 300 instances selected from the synthesis pipeline, presumably filtered for quality and determinate optimality. Human-Authored uses 300 instances written from scratch by annotators without the taxonomy. If the synthesis pipeline generates many instances and the best 300 are selected, while the 300 Human-Authored instances are all that were written, this introduces a selection bias in favor of the synthetic data. The paper doesn't specify whether the synthetic data underwent additional selection beyond the quality-control audit, or whether the 300 Human-Authored instances were similarly curated from a larger pool.

Evaluation Suite Limitations

Single judge model: All verification in the evaluation experiments uses GPT-oss-120b as the Critic. The paper does not report inter-annotator agreement between GPT-oss-120b and human evaluators on the checklist verification task. If the judge model has systematic biases — for example, being lenient on format errors or strict on calculation errors — the scores in Table 2 would reflect those biases rather than true planning capability. The paper acknowledges this implicitly by using a prompt-based verifier rather than programmatic checking, but doesn't quantify its reliability.

No comparison to prior planning benchmarks: The paper evaluates 18 models on PlanningBench but doesn't evaluate them on existing planning benchmarks (TravelPlanner, ChinaTravel, PlanBench, etc.) to establish convergent validity. If PlanningBench is measuring the same underlying planning ability as TravelPlanner, we would expect the model rankings to correlate strongly. Without this comparison, we can't assess whether PlanningBench provides genuinely new information or simply reproduces existing benchmark rankings.

The 0% All-pass models may reflect a threshold effect rather than zero capability: Qwen3-8b, Qwen3-14b, and Qwen3-32b score 0.00%, 0.00%, and 0.27% All-pass respectively. This could mean these models genuinely cannot produce a single fully correct plan on any of the 467 instances — or it could mean that the evaluation set is too small to detect their very low but non-zero capability. With 467 instances and a true success rate of, say, 0.1%, a model would be expected to achieve zero All-pass in a sample of this size. A larger evaluation set (or a more efficient estimator of very low success rates) would be needed to distinguish near-zero from exactly-zero capability.

Training data contamination: The paper uses the same synthesis pipeline for both evaluation data generation and training data generation. While the specific instances are presumably different, the underlying distribution — task types, constraint compositions, subtask variants — is shared. The models in Table 2 are evaluated zero-shot on PlanningBench, which is appropriate. But if any of these models had been exposed to PlanningBench-like data during pretraining (e.g., through the model providers using similar synthesis methods), the evaluation scores would be inflated. The paper doesn't discuss this possibility.

Summary of Evidence-to-Claim Mapping

• "PlanningBench is challenging even for the strongest models": Supported. GPT-5.4-xhigh's 63.17% All-pass leaves substantial headroom (Table 2). The claim would be stronger with evidence that this headroom reflects genuine planning difficulty rather than evaluation noise or judge model error.

• "The All-pass/Avg-pass gap reveals systematic integration failures": Well-supported by the consistent 30–40 point gaps across top models (Table 2) and the error analysis showing Wrong Calculation / Assignment as the dominant failure mode (Table 3). The diagnostic value of this gap is convincingly demonstrated.

• "PlanningBench training transfers to external planning benchmarks": Supported for travel planning benchmarks (ChinaTravel, TravelPlanner) on Qwen-A3B-30B (Table 4). Not tested on non-travel planning benchmarks, which limits the "planning-specific transfer" claim to travel planning specifically.

• "Determinate optimality is crucial for training effectiveness": Strongly suggested by the Syn-NotDetOptimal ablation (Table 5, Figure 4) but not causally isolated. The evidence is correlational — determinate optimality covaries with training success, but other properties of the data batches may also differ. A clean causal test would strengthen this to the paper's most important claim.

• "PlanningBench provides a scalable source of diverse and verifiable planning data": The scalability claim is demonstrated in principle (the pipeline can generate more instances) but not empirically (no experiment shows that scaling from 300 to 600 or 1,200 training instances produces monotonic gains). The diversity claim is supported by the 30+ task types and six families but not quantified beyond the task taxonomy listing.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in the Generation Pipeline

The assumption or constraint. The closed-loop difficulty enhancement mechanism in Section 3.4 requires the Responder to attempt every candidate instance and the Critic to evaluate it, producing a binary all-pass indicator u that determines whether difficulty should escalate. This is explicitly described as "adaptive search over structured planning configurations" that "turns the task and constraint taxonomies into a controllable mechanism for scalable planning data generation." However, each iteration of this loop consumes a non-trivial amount of inference compute: the Responder generates a full candidate plan, and the Critic (a strong LLM — GPT-oss-120b in the paper's instantiation) performs rubric-based evaluation of that plan against the checklist. The paper acknowledges the existence of this loop but does not quantify its computational cost or factor it into any reported efficiency metric.

The consequence. The headline result that PlanningBench "generates scalable, diverse, and verifiable planning data" omits a critical practical consideration: the generation process itself is computationally expensive. Each successfully generated instance that ends up in the final data pool may have required multiple Responder attempts (with associated Critic evaluations) before reaching the difficulty frontier where u = 0. The paper reports that the final evaluation set contains 467 instances and the training set 300 instances, but does not report how many total generation attempts (Responder + Critic invocations) were required to produce these sets. A practitioner seeking to replicate or extend PlanningBench to new domains would need to budget for this compute overhead, which could rival or exceed the cost of the downstream evaluation or training that the generated data is intended to support.

Furthermore, the difficulty escalation mechanism uses the Responder's success as the trigger for making problems harder. This means the difficulty of generated instances is defined relative to the specific Responder model used during synthesis. If a practitioner uses a weaker Responder, the pipeline will produce easier instances (because the Responder fails earlier, triggering fewer difficulty escalations). If they use a stronger Responder, the pipeline will produce harder instances. The paper uses Qwen-A3B-30B as the Responder during evaluation data construction, but the difficulty of the resulting instances is therefore calibrated to that specific model's capability frontier — a practitioner using a different Responder would produce data at a different difficulty level, making it hard to reproduce the exact difficulty characteristics of the published evaluation set.

What evidence exists in the paper. The paper describes the synthesis loop architecture in Section 3.4 and Figure 2, including the Responder and Critic components and the difficulty update formula (Equation 1). The Critic prompt template is provided in Appendix G.2. However, no experiment or analysis quantifies the number of generation iterations per retained instance, the total compute consumed by the synthesis pipeline, or the sensitivity of the resulting data to the choice of Responder model. Section 3.2 notes that the closed loop "turns the task and constraint taxonomies into a controllable mechanism for scalable planning data generation," but "scalable" here refers to the ability to generate arbitrarily many instances from the taxonomy, not to the computational efficiency of doing so.

Mitigation status. Not addressed. The paper does not discuss the compute cost of data generation, propose methods to reduce it, or suggest that future work should measure it. The synthesis pipeline is presented as a mechanism for controllability and difficulty adaptation, with efficiency treated as out of scope. The paper also does not discuss whether the Responder and Critic could be replaced with cheaper models (e.g., a smaller Responder for difficulty calibration, or a programmatic checker instead of GPT-oss-120b for verification) to reduce generation cost. Given the paper's broader argument that PlanningBench "shifts planning data construction from fixed benchmark collection to controllable generation," the unaccounted cost of that controllability is a significant omission for practitioners evaluating whether to adopt the framework.

Human Quality Control Creates a Bottleneck That Undermines Scalability

The assumption or constraint. After automatic synthesis, every generated instance undergoes human quality-control auditing by the same twenty professional annotators who constructed the taxonomies (Section 3.5, Appendix D). The audit assigns each instance to one of four categories: direct retention, retention after minor revision, retention after source correction, or discard. The paper reports that in the audited batch, 86.15% of instances require no or only minor revision, while 13.85% require additional source correction — and that "human revision further improves the clarity, consistency, and verifiability of the final data." This human step is not presented as optional; it is an integral part of the data construction pipeline (Figure 2 shows "Quality control" as a filtering step between the data pool and the final evaluation/training outputs).

The consequence. The paper's central claim is that PlanningBench provides "scalable" planning data generation — a shift from "fixed collection to controllable generation." But scalability in practice is bounded by the human quality-control throughput. Twenty professional annotators can review a finite number of instances per unit time. If the pipeline generates 10,000 instances, each must still be audited — the 13.85% requiring source correction represent manual editing work that does not scale with compute. This creates a fundamental tension in the paper's value proposition: the synthesis pipeline can generate arbitrarily many candidate instances, but the verified, usable data pool is rate-limited by human availability.

The human bottleneck also constrains the claim that PlanningBench "can support planning-oriented reinforcement learning" at scale. The training experiments use 300 instances (Section 4.2), which is a small dataset for RL. One of the paper's key motivations is that existing benchmarks "offer limited support for scalable generation, automatic verification, or planning-oriented training" (Section 1). But if the PlanningBench pipeline still requires human auditing to ensure data quality — and the paper explicitly states that 13.85% of instances need source correction, with problems like "missing checklist items, incomplete constraint checks, ambiguous or underspecified prompts... and occasional logical inconsistencies between the prompt and answer" (Appendix D) — then the framework has not fully escaped the human-bottleneck limitation it critiques in prior benchmarks. The automatic verification only works reliably after humans have fixed the automatically generated instances.

What evidence exists in the paper. The quality-control statistics are reported in Section 3.5 and Appendix D. The 86.15% / 13.85% breakdown comes from a single audited batch of unspecified size. The paper does not report whether the 13.85% revision rate is stable across task types or difficulty tiers, whether it decreases as the Generator model improves, or how many instances per hour the annotators can process. The paper also does not report inter-annotator agreement on the audit categories — if different annotators disagree on whether an instance needs revision, the reliability of the quality-control process itself is uncertain.

Mitigation status. The paper does not acknowledge this as a limitation. Section 8 (Future Directions) does not mention reducing or eliminating the human quality-control step. The synthesis pipeline is presented as producing "recoverable planning instances" (Section 3.5), with human revision as a refinement step rather than a recognized scalability bottleneck. A practitioner reading the paper might reasonably conclude that PlanningBench can generate unlimited training data, when in fact the current instantiation can only generate as much data as twenty annotators can review. Fully automatic quality control — for example, using the Critic model to filter instances without human review, or training a classifier to detect the common failure modes listed in Appendix D — is not discussed.

Transfer Is Only Demonstrated for Travel Planning, Undermining the Claim of General Planning Capability

The assumption or constraint. The paper claims that PlanningBench training "improves performance on unseen planning benchmarks" and provides evidence from two external benchmarks: ChinaTravel (Shao et al., 2024a) and TravelPlanner (Xie et al., 2024). Both are travel-planning benchmarks. PlanningBench itself covers 30+ task types across six structural families — Scheduling and Timetabling, Allocation and Matching, Shift and Workforce Scheduling, Routing and Travel, Project and Production Operations, and Emergency Response and Public Service — only one of which (Routing and Travel) is travel-related. The training data of 300 instances is drawn from the full PlanningBench taxonomy and is not restricted to travel-planning tasks.

The consequence. The evidence for planning-specific transfer is restricted to a single domain — travel planning — despite PlanningBench's training data covering a much broader range of planning structures. This creates an inferential gap: the paper concludes that PlanningBench training teaches "constraint integration, multi-step reasoning, and global consistency" that transfers to "unseen planning benchmarks," but the only unseen planning benchmarks tested share a specific domain (travel) with each other, even if they don't share it with the training data. We cannot distinguish between two competing hypotheses:

1. Domain-general planning transfer: PlanningBench training teaches general planning skills (constraint coordination, global consistency, multi-objective optimization) that transfer to any planning domain, including but not limited to travel. This is what the paper's language implies ("planning-oriented training," "generalizable planning abilities").

2. Cross-task transfer within a reasoning modality: PlanningBench training teaches structured reasoning and constraint satisfaction skills that transfer particularly well to travel planning because travel planning shares underlying reasoning structures (temporal scheduling, resource allocation under constraints, multi-leg coordination) with the PlanningBench tasks, even if the surface domain differs. Transfer to structurally dissimilar planning domains (e.g., blocks-world planning, game-theoretic planning, classical AI planning) might be weaker or non-existent.

The paper cannot adjudicate between these because it tests only travel-planning benchmarks. This is a consequential limitation because the paper's central contribution is a framework for generating planning data across 30+ task types — if training on that diverse data only transfers to travel planning, then the diversity of the taxonomy is not producing the generalizable planning ability that motivates it.

What evidence exists in the paper. The external planning benchmark results are in Table 4. Both ChinaTravel and TravelPlanner are travel-planning benchmarks. The general instruction-following benchmarks in Table 5 (Multi-Challenge, Inverse IFEval, Collie) provide some evidence of broader transfer, but these are not planning benchmarks — they test instruction integration and constraint following in general text generation contexts, which is a different capability from constructing executable plans under resource, temporal, and dependency constraints. The paper does not evaluate on any non-travel planning benchmark such as PlanBench (Valmeekam et al., 2023), which tests classical AI planning tasks, or on benchmarks from the scheduling, allocation, or routing families that PlanningBench's taxonomy explicitly covers.

Mitigation status. The paper does not acknowledge this as a limitation. The Results section (4.2.2) describes the transfer to ChinaTravel and TravelPlanner without noting that both are travel-planning benchmarks. The claim that PlanningBench "improves performance on unseen planning benchmarks" (plural, implying generalization across planning domains) is not supported for any planning domain other than travel. The paper would need to evaluate on at least one non-travel planning benchmark — ideally one from a structural family well-represented in PlanningBench's taxonomy (e.g., a scheduling benchmark, an allocation benchmark, or a project-planning benchmark) — to support the domain-general planning transfer claim.

The Evaluation Suite Is Small (467 Instances) and May Not Reliably Discriminate Model Capability at the Extremes

The assumption or constraint. The PlanningBench evaluation set contains 467 instances spanning 30+ task types across six structural families (Section 4.1.1). The paper uses this set to rank 18 models by All-pass and Avg-pass (Table 2) and to analyze performance by task type, prompt length, and checklist item count (Figure 3). The set is constructed through the synthesis pipeline with human quality control and is treated as a fixed benchmark for model comparison.

The consequence. Three separate concerns arise from the evaluation set's size and composition.

First, statistical reliability of model rankings. With 467 instances, the standard error of the All-pass rate for a model with true success probability p is approximately sqrt(p(1−p)/467). For GPT-5.4-xhigh at 63.17%, this is roughly 2.2 percentage points. The gap between GPT-5.4-high (58.56%) and GPT-5.4-medium (58.09%) is 0.47 percentage points — well within one standard error. We cannot determine whether these two models have genuinely different planning capabilities or whether their ranking would reverse on a different sample of 467 instances. The paper does not report confidence intervals or conduct pairwise significance tests between models, so the precise ranking in Table 2 should be interpreted cautiously, particularly for models with All-pass rates within a few percentage points of each other.

Second, near-zero capability discrimination. Three models score exactly 0.00% All-pass (Qwen3-14b, Qwen3-8b) and one scores 0.27% (Qwen3-32b). With 467 instances, a model with a true All-pass rate of 0.1% would be expected to score zero in a random sample more than 60% of the time. The evaluation set cannot distinguish between models that have essentially zero planning capability and models that have very low but non-zero capability. For a benchmark that claims to provide "useful discrimination across model capability levels" (Section 4.1.2), this floor effect means the bottom of the capability distribution is unresolved.

Third, per-task-type sparsity. With 30+ task types and 467 total instances, the average number of instances per task type is roughly 15 (and likely fewer for some types, given the non-uniform distribution shown in Figure 1). The task-specific analysis in Figure 3a examines only five task types and does not report how many instances each contains. If a particular task type has only 10–15 instances in the evaluation set, the per-task All-pass rates are very noisy — a single additional correct or incorrect answer can shift the rate by 7–10 percentage points. This makes it difficult to draw reliable conclusions about which task types are harder or which models have task-specific strengths, beyond the qualitative observation that performance varies across tasks.

What evidence exists in the paper. The evaluation set size (467) is stated in Section 4.1.1. Figure 1 shows the distribution of task types, revealing that the largest categories (Scheduling and Timetabling, Project and Production Operations, Routing and Travel) each contain 17–28% of the data, while the smaller categories (Shift and Workforce Scheduling, Emergency Response and Public Service) contain roughly 10–11%. The paper does not report per-task-type instance counts, confidence intervals on All-pass rates, or statistical tests between model pairs. Figure 3a provides per-task All-pass for five tasks but does not indicate sample sizes.

Mitigation status. Not addressed. The paper frames the 467-instance set as a challenging evaluation suite and does not discuss its statistical limitations. The scalability of the synthesis pipeline means larger evaluation sets could be generated — the paper could have produced 1,000 or 2,000 instances for the evaluation set with the same methodology — but chose 467 without justifying the size. The paper also does not discuss whether future work should scale the evaluation set or whether 467 is considered sufficient for the discriminative claims being made. For a framework whose primary contribution is scalable generation, the relatively small evaluation set is a missed opportunity to demonstrate that scalability in the evaluation context.

The Determinate Optimality Claim Lacks a Causal Ablation, Leaving the Central Training Insight Correlational

The assumption or constraint. Section 3.6 argues that determinate or well-specified optimal solutions provide "more stable and directional reward signals" and that this design choice is essential for planning-oriented training. The evidence comes from two sources: an early experiment where "GRPO training did not yield clear gains and degraded performance on general instruction-following benchmarks" using data without determinate optimality (Section 3.6), and the Syn-NotDetOptimal ablation in Table 5 and Figure 4, which shows near-null transfer (+0.75 points average) compared to Syn-PlanningBench (+7.06 points). The paper concludes that "reward determinacy is an important factor for stable and transferable planning-oriented training."

The consequence. The Syn-NotDetOptimal condition is not a clean causal ablation of determinate optimality. It is described as "synthetic planning data that does not emphasize determinate optimality to the same extent" (Section 4.2.1). This phrasing — "to the same extent" — indicates a relative difference, not a controlled manipulation. The two data batches (Syn-PlanningBench and Syn-NotDetOptimal) may differ in several systematic ways beyond the determinacy of their optimal solutions:

• Constraint coupling tightness: Instances designed to have determinate optima likely have tighter constraint coupling — constraints that jointly restrict the solution space to a narrow region. Instances without determinate optima may have looser coupling, where many solutions satisfy the core constraints. The training benefit may come from learning to handle tight constraint coupling, not from the optimality determinacy per se.
• Difficulty distribution: The determinate-optimal data may be systematically harder or easier, or may have a different distribution across task types and difficulty tiers. If Syn-PlanningBench is harder (because determinate optima require more precise reasoning), the training benefit may come from curriculum effects rather than reward-signal determinacy.
• Verification strictness: The determinate-optimal data uses verification that is all-or-nothing (the solution must match the unique optimum). The loose-optima data may use verification that is more permissive. The training benefit may come from the strictness of the verification criterion, not the determinacy of the optimum — a problem could have a unique optimum but lax verification (rewarding "close enough" solutions), or multiple optima but strict verification (requiring exact match to any one of them).

The paper cannot distinguish these alternatives because Syn-NotDetOptimal is a single data batch that differs from Syn-PlanningBench along multiple (uncontrolled) dimensions. The claim that "determinate optimality" specifically drives the training benefit is therefore correlational — it is associated with better training outcomes, but the causal mechanism is not isolated.

What evidence exists in the paper. The Syn-NotDetOptimal results are in Table 5 and Figure 4. The paper acknowledges the limitation explicitly in Section 3.6: "We conjecture that loose optima and permissive verification made the reward less directional. Later batches therefore strengthen the preference for determinate or otherwise well-specified optimal solutions whenever possible. Subsequent evaluations show clearer gains, although more systematic experiments are still needed." This is an honest acknowledgement, but it means that one of the paper's central claims — that determinate optimality is the key factor — rests on a conjecture supported by correlational evidence rather than a controlled experiment.

Mitigation status. Partially addressed through acknowledgement. The paper explicitly states that "more systematic experiments are still needed" (Section 3.6), recognizing that the current evidence is preliminary. However, given that this claim is presented as a key finding ("We further identify reward determinacy as an important factor for stable and transferable planning-oriented training" in the abstract and contributions), the gap between the evidence (correlational) and the claim (causal) is significant. A proper causal test would require: take a fixed set of planning problems, create two verification protocols — one requiring exact match to the unique optimum and one awarding full credit to any feasible solution — and compare training outcomes while holding the problem distribution constant. Until such an experiment is conducted, the determinate optimality claim should be treated as an empirically-motivated hypothesis rather than an established finding.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper introduces a conceptual shift in how the field thinks about planning data: from benchmark-as-collection to data-as-generative-output. Prior to this work, every planning benchmark — TravelPlanner, ChinaTravel, DeepPlanning, PlanBench, and their variants — treated a planning dataset as a fixed, hand-designed collection of instances whose coverage and difficulty were artifacts of the designer's choices. PlanningBench reframes planning data as samples from a structured design space parameterized by a task-constraint taxonomy, making coverage, difficulty, and verification into controllable generation parameters rather than post-hoc observable properties. This is not a new benchmark; it is a data generation framework that can produce arbitrary numbers of novel planning instances within a defined structural distribution.

The magnitude of this shift is comparable to what happened in mathematical reasoning when the field moved from hand-crafted problem sets to programmatic generation — but adapted to a significantly more complex domain where constraint coupling means that naïve template-based generation fails. The key distinction is that mathematical problems typically test a single reasoning chain culminating in one answer, while planning problems require simultaneous satisfaction of multiple interacting constraints at every decision point. PlanningBench's taxonomy-guided constraint composition directly addresses this structural complexity by making constraint coupling, dependency chains, and multi-objective tradeoffs into first-class sampling parameters rather than emergent properties of hand-written instances.

This work resolves a specific contradiction that has limited prior attempts to use planning data for training. The paper reports an early experiment where GRPO training on approximately 1,000 synthetic planning instances degraded general instruction-following performance (Section 3.6). The explanation — that permissive verification on loose answer spaces produces diffuse reward signals that teach partial constraint satisfaction rather than global planning consistency — reconciles why some prior efforts to train on planning data have failed while others (notably in math and code reasoning) have succeeded. Planning tasks that reward "mostly correct" solutions create gradients that point toward satisfying individual constraints, not toward constructing globally coherent plans. Only when the reward landscape has a sharp peak at the unique correct solution — as in determinate-optimal PlanningBench instances — does the optimization signal point toward the integration and coordination skills that define genuine planning ability.

The paper also clarifies the boundary between two failure modes that prior benchmarks conflate. A model that scores 0% on a standard planning benchmark may be completely lost (satisfying no constraints) or may be satisfying most constraints while failing on a critical few (satisfying, say, 80% of requirements but violating a coupled dependency that invalidates the whole plan). PlanningBench's All-pass / Avg-pass gap disambiguates these: GPT-5.4-medium's 90.03% Avg-pass but 58.09% All-pass tells you precisely that the model is roughly competent at individual constraint satisfaction but fails at global integration. This diagnostic granularity doesn't just report performance — it directs improvement efforts. The error analysis in Table 3, combined with the All-pass/Avg-pass gap, tells a practitioner that format compliance (0.7%–3.4% failure rate) is not the bottleneck; numerical, temporal, and allocation reasoning under coupled constraints (60.9%–83.5%) is. This redirects training priorities from "better output formatting" to "better constrained reasoning."

Research directions that become more attractive after this work:

1. Verification-signal quality as a first-class design parameter. The Syn-NotDetOptimal ablation (+0.75 points average transfer vs. +7.06 for Syn-PlanningBench) establishes that not all verified training data is created equal. Research on RL-based reasoning training should now consider reward determinacy — how sharply the reward function peaks at the correct answer — as a measurable property of training data, alongside traditional properties like diversity, difficulty, and size. Data with diffuse rewards may be worse than useless; data with sharp rewards produces transferable skills. This applies beyond planning to any constrained-generation task where partial correctness does not imply overall correctness.

2. Structural difficulty decomposition. The paper's three-tier difficulty taxonomy (basic feasibility, medium optimization, hard meta-reasoning) provides a language for asking what kind of planning is hard, not just how hard it is. This opens research into whether specific architectural components (e.g., chain-of-thought, tool use, retrieval) address specific difficulty tiers, enabling targeted improvements rather than uniform scaling. A model that excels on basic and medium tiers but fails on hard-tier robustness and infeasibility detection has a specific, diagnosable weakness that can be addressed with targeted interventions.

3. Synthetic data generation as a systems problem. The Generator-Responder-Critic loop in PlanningBench demonstrates that data generation itself can be an adaptive optimization process — the pipeline actively searches for instances at the difficulty frontier by using the Responder's success/failure as feedback. This generalizes: any domain where task difficulty can be parameterized (by constraint composition, problem structure, information load) can potentially use a similar closed-loop mechanism to generate data calibrated to a specific model's capability frontier, enabling personalized curricula or diagnostic test sets.

Research directions that become less attractive after this work:

1. Building yet another travel-planning benchmark. Table 1 lists 11 travel-planning benchmarks (TravelPlanner, TravelBench, TRIP-Bench, WorldTravel, TripTailor, ChinaTravel, TripCraft, and others). PlanningBench demonstrates that the bottleneck is not domain coverage within travel planning — it's structural diversity across planning families. A new benchmark that adds more travel constraints or more realistic travel data does not advance the field's understanding of LLM planning capability beyond what existing travel benchmarks already test.

2. Uniform test-time compute scaling for planning. The evaluation results show that even the strongest models (GPT-5.4-xhigh at 63.17% All-pass) leave a substantial gap to ceiling. However, the error analysis in Table 3 reveals that the bottleneck is not insufficient sampling or search — it's incorrect calculation and assignment under coupled constraints. Simply scaling test-time compute (e.g., best-of-N sampling, majority voting) is unlikely to address this failure mode because the errors are in the reasoning itself, not in the stochastic selection among correct candidates. This contrasts with domains like math problem solving, where sampling-based methods can exploit the fact that a correct solution exists somewhere in the model's generation distribution.

3. Building stronger agents for single-domain planning. The prior literature's focus on better agent architectures for travel planning (Triflow, Vaiage, Atlas, DeepTravel) implicitly assumes that the challenge is in how to construct plans given domain-specific constraints. PlanningBench's cross-domain taxonomy suggests that the deeper challenge is in the constraint integration and coordination skills that generalize across structurally similar planning families — and that training on diverse planning structures (as in the Syn-PlanningBench condition) transfers better to unseen domains than training on task-specific heuristics.

Follow-Up Research This Work Enables

A clean causal test of the determinate optimality hypothesis. The paper's most intriguing claim — that determinate optima are essential for planning-oriented training — rests on a between-batch comparison (Syn-PlanningBench vs. Syn-NotDetOptimal) rather than a controlled experiment. The critical follow-up would isolate reward determinacy from other properties of the training data. Take a fixed set of, say, 500 PlanningBench-generated instances that each admit multiple feasible solutions. Create two verification protocols: (a) a sharp protocol that awards maximum reward only if the model's plan matches a pre-specified optimal solution exactly, and (b) a diffuse protocol that awards maximum reward for any plan satisfying all core constraints. Train two models under otherwise identical GRPO conditions and compare their transfer to TravelPlanner, ChinaTravel, and the general instruction-following benchmarks. If the sharp-protocol model substantially outperforms the diffuse-protocol model, the determinate-optimality claim would be causally established. If performance is similar, the Syn-NotDetOptimal results were driven by other confounds (constraint coupling, difficulty distribution, data quality). This experiment would refine the paper's central training insight from a correlational pattern to a causal mechanism, with direct implications for data construction in any domain where verification determinacy can be engineered.

Non-travel planning transfer evaluation. The paper demonstrates transfer from PlanningBench training to two travel-planning benchmarks (ChinaTravel, TravelPlanner), but PlanningBench's training data spans six structural families, only one of which (Routing and Travel) is travel-related. The claim of general planning transfer remains untested for non-travel domains. A strong follow-up would evaluate Syn-PlanningBench-trained models on PlanBench (Valmeekam et al., 2023), which tests classical AI planning tasks (blocks-world, logistics, gripper) requiring action sequencing, precondition satisfaction, and goal-state achievement — structurally aligned with PlanningBench's Project and Production Operations and Scheduling families. If PlanningBench training improves PlanBench performance at levels comparable to the TravelPlanner gains (~18 All-pass points on average), then the training genuinely teaches domain-general planning skills. If transfer is negligible, then the travel-benchmark improvements reflect cross-task transfer within a narrow reasoning modality rather than general planning capability. Additional evaluations on scheduling benchmarks (derived from Operations Research problem sets for job-shop or nurse-scheduling) and allocation benchmarks (fair division or resource assignment problems) would triangulate transfer to the structural families that PlanningBench's taxonomy explicitly covers.

Data quality vs. quantity scaling for planning-oriented training. The training experiments use exactly 300 instances per condition, and Syn-PlanningBench substantially outperforms both Human-Authored and Syn-NotDetOptimal at this fixed size. But 300 instances is small for RL training — the paper does not test whether increasing data quantity (to 600, 1,200, or 2,400 instances) produces proportional gains, or whether the gains saturate. A scaling experiment would train models on {300, 600, 1200, 2400} Syn-PlanningBench instances (controlling for training steps) and measure transfer to TravelPlanner and the general benchmarks. The key question is whether the +18-point TravelPlanner All-pass gain from 300 instances represents the early slope of a curve that flattens (quality matters more than quantity beyond a small dataset) or a curve that continues to rise (more data continues to help). This would inform practitioners whether to invest in expanding the PlanningBench taxonomy to generate more diverse instances or in refining the existing instances for higher quality. If data quantity scaling saturates quickly — which the paper's early negative result with ~1,000 loose-optima instances hints might be the case — then the field should prioritize verification-signal quality over data volume in constrained-generation training.

Difficulty-tier diagnostic evaluation. The paper's three-tier difficulty taxonomy (basic/medium/hard) organizes constraints by the type of planning challenge they represent, but the paper does not report model performance disaggregated by difficulty tier. This is a missed diagnostic opportunity. A follow-up study would tag each of the 467 evaluation instances with its constraint composition (how many basic, medium, and hard constraints were sampled during synthesis) and report All-pass rates separately for each tier combination. For example: what is GPT-5.4-xhigh's All-pass on instances with only basic constraints? On instances with basic + medium? On instances with at least one hard constraint? This disaggregation would reveal whether the All-pass/Avg-pass gap is driven primarily by medium-tier optimization failures (the model can produce a feasible plan but can't optimize it) or by hard-tier meta-reasoning failures (the model can't recognize infeasibility or design robustness). Different failure profiles would imply different training priorities: an optimization-bottlenecked model needs more medium-tier training data; a meta-reasoning-bottlenecked model needs hard-tier exposure.

Fully automatic quality control to close the human-bottleneck gap. The current PlanningBench pipeline requires human quality-control auditing that reviews and revises every generated instance (13.85% requiring source correction in the audited batch). This creates a scalability bottleneck — the synthesis pipeline can generate arbitrarily many instances, but the usable data pool is rate-limited by annotator throughput. A follow-up study would develop and evaluate an automatic quality-control layer that replaces or augments human review. One approach: train a classifier on the human audit decisions (the 86.15% accepted, 13.85% revised, 0% discarded) to predict whether a new generated instance requires revision, using features like the Critic's verification score, the Responder's confidence, the number and type of constraints, and lexical overlap between the prompt and checklist. If the classifier can identify the 86% of instances that need no revision with high precision, human effort can be focused on the problematic 14%. An alternative approach: use the Critic model itself to detect the common failure modes that human revisers fix (missing checklist items, underspecified constraints, ambiguous references) by prompting it to self-audit generated instances before they reach human review. Success would be measured by whether an automatically filtered dataset of, say, 1,000 instances (no human review) produces training gains comparable to the 300-instance human-reviewed set — demonstrating that PlanningBench's scalability promise can be realized without the current human bottleneck. If automatic filtering fails to maintain data quality, this would reveal that the human audit is doing non-trivial cognitive work that current models cannot replicate, setting a research agenda for verifier robustness.

Cross-model-family replication of training gains. All training experiments use Qwen-A3B-30B as the base model. Replicating the Syn-PlanningBench vs. Syn-NotDetOptimal comparison on at least two other model families (e.g., a Llama-based model, a DeepSeek-based model) would test whether the determinate-optimality effect is model-specific or general. If Syn-PlanningBench produces similar relative gains across model families, the finding is robust and can inform training data design broadly. If the effect is specific to Qwen-A3B-30B (e.g., because its pretraining distribution already contains planning-like data that PlanningBench activates), then the determinate-optimality claim is contingent on base-model characteristics. A strong replication would also test whether the base model's starting capability level interacts with training effectiveness: does a stronger base model (with higher initial All-pass on PlanningBench) benefit more from training (because it can already produce approximately correct plans and just needs refinement), less (because it's near ceiling), or not at all? The Qwen3 family scaling in Table 2 (All-pass dropping from 12.15% to 0.00% as model size decreases) suggests a sharp capability threshold — does training effectiveness show a similar threshold?

Practical Applications and Downstream Use Cases

Cost-efficient diagnostic evaluation for LLM deployment decisions. Organizations deciding whether to deploy an LLM for planning-intensive applications (supply chain scheduling, workforce rostering, project timeline construction) currently lack standardized, domain-diverse benchmarks to compare candidate models. PlanningBench's evaluation suite — 467 instances across 30+ task types — provides a single benchmark that tests planning capability across structurally distinct families, revealing not just which model is best but what kind of planning each model excels or fails at. A team evaluating GPT-5.4-high vs. Seed-2.0-pro-high for a logistics deployment can see from Table 2 and Figure 3a that the models differ not just in aggregate capability (58.56% vs. 44.33% All-pass) but potentially in task-specific strengths (the paper's data shows task-dependent profiles for different models). The All-pass/Avg-pass gap further reveals whether a model's failures are catastrophic (low Avg-pass) or integration-related (high Avg-pass, low All-pass) — the latter suggesting that the model is "almost there" and might benefit from test-time verification or lightweight fine-tuning, while the former suggests fundamental unsuitability. For teams without the budget to develop custom planning benchmarks for each deployment domain, PlanningBench provides a off-the-shelf diagnostic that tests the constraint integration and global consistency skills that generalize across planning applications.

Training data generation for domain-specific planning fine-tuning. A company building a production scheduling system or a hospital building a nurse-rostering tool can use the PlanningBench framework to generate domain-adapted training data without manually writing hundreds of planning instances. The process would be: extend the PlanningBench taxonomy with domain-specific task types and constraint pools (e.g., for production scheduling, add machine-specific constraints, changeover times, maintenance windows; for nurse rostering, add qualification tiers, union rules, shift-swap constraints), run the constraint-driven synthesis pipeline to generate instances with adaptive difficulty control, apply the quality-control audit, and use the resulting verified data for GRPO-based fine-tuning of a base LLM. The paper's evidence that Syn-PlanningBench data generalizes better than human-authored data (+7.44 vs. +1.49 average All-pass gain on ChinaTravel) suggests that the taxonomy-guided synthesis approach produces training data with structural properties that manual authoring misses — specifically, systematic constraint composition across difficulty tiers that teaches constraint integration rather than pattern matching. The closed-loop difficulty escalation mechanism further ensures that the generated data targets the model's capability frontier rather than being uniformly easy or hard. The primary cost is the initial taxonomy extension (which requires domain expertise to identify the relevant constraints and their difficulty tiers) and the quality-control audit (which Section 3.5 shows requires revision of ~14% of instances). Once these are in place, the pipeline can generate new training instances as the model improves, maintaining a moving-target curriculum.

Verification-signal quality assessment for any RL-based instruction-following training. The paper's finding that Syn-NotDetOptimal training (diffuse rewards) produces near-null gains (+0.75 average points on general benchmarks) while Syn-PlanningBench training (determinate rewards) produces +7.06 point gains has immediate practical implications for any team using RL to train models on constrained-generation tasks. Before committing to an expensive RL training run, practitioners should audit their training data's reward determinacy: for a random sample of 100 training instances, how many admit multiple substantially different answers that would all receive maximum reward under the verification protocol? If the answer is "most of them," the training signal will be diffuse — the model will receive positive rewards for widely varying outputs, and the gradient will not point toward any specific capability improvement. If the answer is "few of them," the training signal will be directional — the model must learn to produce outputs within a narrow target distribution, and the RL process has a clear objective to optimize toward. PlanningBench provides a concrete methodology for this audit: define the verification protocol upfront, check how many training instances have a unique optimal solution under that protocol, and either redesign instances with loose optima or accept that training on them may not improve (and could degrade) downstream capabilities. The paper's negative result with Syn-NotDetOptimal — specifically the degradation on Inverse IFEval (−0.70 points) — suggests that ignoring reward determinacy is not just ineffective but potentially harmful, making this audit a prudent step before any RL training investment.