How2Everything: Mining the Web for How-To Procedures to Evaluate and Improve LLMs

ArXiv: 2602.08808

🎯 Pitch

LLMs generate flawed how-to guides about 60% of the time, but simply training them on the same web data doesn't fix it—even when they memorize the source documents verbatim.

---

1. Executive Summary

This paper introduces How2Everything, a scalable framework for evaluating and improving LLMs' ability to generate goal-conditioned step-by-step procedures—a capability that accounts for approximately 8.5% of real-world chatbot interactions. The framework includes How2Mine, which extracts 351K structured procedures from 980K web pages across 14 topics, and How2Score, an LLM-as-a-judge evaluation protocol that detects critical failures (omissions, extraneous actions, or deviations that would prevent achieving the goal, such as skipping a legally required 30-day waiting period before transferring property ownership). The authors distill a frontier judge into How2Judge, an open 8B model achieving 80.5% agreement with human annotators, and use How2Score as a reward signal for RL training, yielding >10-point improvements on How2Bench across three models (e.g., Qwen 3 8B Instruct rising from 38.52 to 48.62) without systematic regressions on standard out-of-domain benchmarks—establishing that procedural generation improvements from RL are not explained by surface format compliance or source-document memorization, with How2Score gains persisting even when source documents are aggressively repeated during midtraining (only a modest +3.3-point improvement for a 7B model despite perplexity dropping from 10.4 to 1.4).

2. Context and Motivation

The Core Problem: We Cannot Reliably Measure Whether an LLM Generates a Valid Procedure

The fundamental question this paper tackles is deceptively simple: if an LLM generates a step-by-step plan to accomplish a real-world goal, how do we know whether that plan would actually work? This matters because procedural generation is not a niche capability — it is one of the most common ways people use LLMs. Chatterji et al. (2025) found that approximately 8.5% of ChatGPT conversations fall under How-To Advice, ranking it as the fourth most frequent query category behind only Specific Info (18.3%), Edit or Critique Provided Text (10.6%), and Tutoring or Teaching (10.2%). At current ChatGPT scale, the paper notes this corresponds to tens to hundreds of millions of how-to interactions daily (Appendix A).

The real-world stakes are obvious but easy to underestimate. When someone asks an LLM how to file a legal document, replace a kitchen faucet, or prepare a recipe safely, the output is not just a text completion — it is a set of instructions that a human might follow. A single critical omission (e.g., forgetting to mention a legally required 30-day waiting period before transferring property ownership, as shown in Table 1) or a subtle ordering error (e.g., cutting wood into pieces and then describing them with contradictory size relationships) can render the entire procedure worthless or dangerous. Yet, unlike mathematical reasoning — where a final answer is either provably correct or not — real-world procedures have no executable ground truth. You cannot run a compiler on instructions for filing taxes to check whether they produce the correct outcome.

This creates a fundamental evaluation gap. The field has robust, scalable evaluation protocols for tasks with verifiable answers (math, code execution, multiple-choice QA) but has largely avoided the challenge of evaluating open-ended procedural validity at scale. The paper positions itself directly in this gap, arguing that both existing metrics (string overlap, perplexity) and existing dataset construction methods (narrow domains, single-source scraping) are insufficient for the core capability that drives ~8.5% of chatbot usage.

Why This Problem Is Hard (and Why Prior Approaches Fall Short)

The difficulty of evaluating procedural validity is not just a matter of finding the right metric — it stems from three intertwined challenges that the paper identifies and systematically addresses.

Challenge 1: Diversity of procedures. Real-world procedures span an enormous range of domains. They can involve legal processes (filing court documents), cooking (glazing nuts in a Crock Pot), home repair (replacing a faucet), education (writing a lesson plan), or crafting (making stamped images in a journaling Bible). Each domain has its own conventions, constraints, and failure modes. Prior datasets in the descriptive procedural setting, as the paper catalogs in Section 2, have typically been restricted along one of two axes:

• Topical domain restriction: Many datasets focus exclusively on cooking (Bień et al., 2020; Toyooka et al., 2025; Anika and Miah, 2025). While cooking procedures are well-structured and easy to collect, they represent a tiny slice of the procedural knowledge that LLMs need to master — knowing how to glaze nuts tells you nothing about how to file a legal notification to co-owners of a shared property.
• Source restriction: Other datasets draw from specific instructional platforms like WikiHow and Instructables (Zhou et al., 2022; Bolotova-Baranova et al., 2023; Brahman et al., 2024; Uzunoglu et al., 2024). These sources produce clean, well-formatted data, but they introduce selection bias: the procedures that get written up on WikiHow are not representative of the full distribution of procedural knowledge on the web, and models trained or evaluated exclusively on such data may develop brittle, format-specific behaviors.

The paper's key insight on this front is that the web itself — not any single curated source — is the natural distribution from which procedural knowledge should be mined, because that is the distribution from which LLMs actually learn during pretraining. How2Mine is designed to operate on arbitrary web corpora (Section 3), extracting procedures from 980K documents across 14 topics as a proof of concept, with the pipeline explicitly architected to scale to larger corpora without manual curation.

Challenge 2: The evaluation reliability–scalability tradeoff. This is the central methodological tension the paper confronts. On one end of the spectrum are automatic metrics like BLEU, ROUGE, or perplexity on a reference procedure. These are cheap to compute and infinitely scalable, but the paper cites prior work acknowledging that they are "insufficient proxies for procedural validity" (Lyu et al., 2021; Li et al., 2023; Brahman et al., 2024). The reason is intuitive: a generated procedure can use completely different wording, reorder non-dependent steps, or include equally valid alternative actions and still be perfectly correct — but score terribly on string overlap. Conversely, a generated procedure can paraphrase the reference almost verbatim while containing a critical factual error (e.g., wrong cooking temperature) and score well on BLEU. The metric measures surface similarity, not correctness.

On the other end of the spectrum is human evaluation. Human annotators can reliably identify whether a procedure would work (the paper's annotation protocol achieves reasonable inter-annotator agreement: Krippendorff's $\alpha = 0.593$ with binary score aggregation, Section 4.3). But human evaluation is expensive and slow — the paper's annotation effort cost $3,600 USD for 200 examples, or about$18 per example. Evaluating a 7,000-example benchmark at that rate would cost approximately $126,000, and using human evaluation as an RL reward signal (which requires scoring millions of generated procedures during training) would be completely infeasible.

LLM-as-a-judge protocols (Zheng et al., 2023; Dubois et al., 2025) have emerged as a middle ground, using a strong LLM to score model outputs. But the paper identifies a specific failure mode of generic preference-style judging when applied to procedures: it "can overemphasize surface qualities like coherence or helpfulness, and thus fail to capture end-to-end procedural validity" (Section 2). A generation might sound authoritative, be well-formatted, and appear helpful while still containing a critical omission — and a generic judge might reward those surface qualities rather than flagging the failure. The paper's contribution is to develop a validity-oriented evaluation protocol (How2Score) that targets task-level correctness rather than general quality, and then to distill that protocol into a small, open model (How2Judge) that makes it cheap enough to use at scale — including as an RL reward signal during training.

Challenge 3: Prior task formulations don't target end-to-end validity. The paper catalogs a range of prior work that studies procedural capabilities but through task formulations that avoid the full end-to-end generation problem:

• Edge prediction over step pairs (Sakaguchi et al., 2021): given two steps, determine whether one precedes the other. This tests local ordering knowledge but says nothing about whether a model can generate a complete, valid sequence from scratch.
• Step reordering (Anika and Miah, 2025): given a scrambled set of steps, reconstruct the correct order. This assumes the correct steps are already provided — the model doesn't need to determine what steps are needed, only their order.
• Question answering (Lal et al., 2024; Uzunoglu et al., 2024): answer questions about a procedure (e.g., "What comes after step 3?"). This tests comprehension of a given procedure, not generation.
• Constraint satisfaction (Yuan et al., 2023): ensure generated steps satisfy specified constraints. This is closer to full generation but still decomposes the problem rather than testing whether the whole sequence achieves the goal.

The paper argues that these subtask formulations, while useful for studying specific aspects of procedural reasoning, do not close the gap to the end-to-end capability that users actually need: given a goal and (optionally) a list of resources, generate a complete sequence of steps that would achieve that goal without any critical failures. This is the task that accounts for 8.5% of ChatGPT usage, and it is the task that How2Everything is designed to evaluate and improve.

Conflicting Demands: Why a Single Solution Has Been Elusive

The three challenges above create conflicting demands that explain why prior work has not produced a scalable solution for evaluating procedural validity:

• Diversity demands mining procedures from broad web sources, but broad web sources are noisy — full of non-procedural content, poorly formatted steps, and procedures that depend on specific named entities or UI interactions. Cleaning this data requires sophisticated filtering pipelines that prior work has not developed for the specific task of procedure extraction.
• Reliability demands evaluation protocols that correlate with human judgments of validity, but human judgment of open-world procedures is itself subjective and noisy (as the paper's annotation pilots revealed, where initial agreement was only $\alpha = 0.273$). Developing an evaluation rubric that achieves reasonable inter-annotator agreement — and that can be automated — requires careful definition of what constitutes a failure and iterative refinement with annotators, which is expensive and time-consuming.
• Scalability demands evaluation that is cheap enough to run on thousands of examples and fast enough to serve as a training reward, but LLM-as-a-judge protocols risk self-preference bias (a judge from one model family might favor its own outputs) and even frontier models cost money (GPT-5 evaluation of 7,000 examples costs approximately $15, which adds up quickly when evaluating many models or running RL training). Distillation into a small open model solves the cost problem but introduces the risk of reduced accuracy.

The paper's architecture — How2Mine for scalable data extraction, How2Score for validity-oriented evaluation, How2Judge for cheap distillation — is designed to resolve these conflicting demands simultaneously, with each component addressing a specific tension.

Where Prior Approaches Specifically Fall Short

The paper is careful to position itself against specific prior work rather than making vague claims about the field. Let me walk through the key comparisons:

Compared to WikiHow-based datasets (Zhang et al., 2020; Zhou et al., 2022; Brahman et al., 2024). These datasets extract procedures from WikiHow, a single-source instructional website. The limitation is not just coverage (WikiHow articles skew toward certain kinds of procedures that are "wiki-worthy") but also evaluation methodology. Brahman et al. (2024), for instance, uses perplexity on reference procedures as a metric and acknowledges that "string-overlap metrics like BLEU are fast to compute but inaccurate, and human annotation is accurate but expensive" — exactly the reliability–scalability tradeoff that How2Everything aims to resolve. The paper positions How2Mine as a generalization: instead of scraping one instructional website, mine any web corpus, enabling broader coverage and the ability to dynamically construct evaluation sets that change as web content changes.

Compared to domain-specific procedural datasets (Bień et al., 2020, RecipeNLG for cooking; Toyooka et al., 2025, also cooking; Anika and Miah, 2025, cooking). These datasets are valuable for in-domain analysis but cannot support claims about general procedural capability. A model that excels at generating cooking recipes may have learned cooking-specific conventions (ingredient lists, specific temperature ranges, step patterns) rather than general procedural reasoning. How2Bench explicitly balances across 14 topics (500 examples each, totaling 7,000) to ensure that performance reflects general procedural ability rather than domain-specific knowledge.

Compared to LLM-as-a-judge protocols (Zheng et al., 2023; Dubois et al., 2025). Standard LLM judging typically asks for pairwise preferences ("Is response A better than response B?") or Likert-scale ratings of overall quality. The paper argues that these are fundamentally misaligned with the evaluation goal: a generated procedure might receive a high overall quality rating for being well-written, detailed, and helpful, yet still contain a single critical omission that would cause a human following it to fail. How2Score is deliberately a targeted, failure-detection protocol rather than a quality-rating protocol. The judge's output is not "how good is this procedure on a scale of 1–5?" but rather "does this procedure contain any critical failure, and if so, what is it?" This framing gives the evaluation clear task-level validity: a procedure is successful if and only if it contains no critical failures, regardless of how polished or helpful it appears.

Compared to sub-task evaluations (Sakaguchi et al., 2021, edge prediction; Anika and Miah, 2025, step reordering). These evaluations are valuable for scientific analysis — they isolate specific procedural reasoning skills — but they do not test the end-to-end capability that users experience. The paper's position is not that these subtask evaluations are wrong, but that they are incomplete: a model might score perfectly on step reordering yet still fail to generate a valid procedure from scratch because it omits a necessary step entirely. End-to-end generation with critical-failure detection is a stricter, more ecologically valid test.

Compared to executable procedure benchmarks (Samiei et al., 2025, formal transition systems; Puig et al., 2018, VirtualHome; Shridhar et al., 2021, ALFWorld). The paper draws an explicit distinction in Section 2 between descriptive procedures (textual representations) and executable procedures (those where correctness is determined by execution in a grounded environment). Executable benchmarks are ideal for validation — you can literally run the procedure and see if it works — but they are inherently limited to simulated environments with pre-defined action spaces (e.g., a household simulator where the agent can only pick_up, put_down, open, close, etc.). They cannot represent the open-ended diversity of real-world how-to instructions ("file a notarial document," "wait 30 days," "receive a receipt"). The paper's focus on descriptive procedures with reference-based evaluation is explicitly a tradeoff: sacrifice the certainty of execution-based verification to gain coverage over the full diversity of real-world procedural knowledge.

How This Paper Positions Itself

The paper frames How2Everything not as a single new model or algorithm but as a framework — an integrated system of data pipeline, evaluation protocol, and training methodology that together enable a closed-loop development cycle. The key conceptual moves are:

1. Mining over curation: Rather than manually curating a dataset from specific sources, build a pipeline (How2Mine) that can extract structured procedures from arbitrary web corpora. This makes the approach scalable and dynamic — as web content changes, the evaluation and training data can be refreshed without manual effort.

2. Failure detection over quality rating: Rather than asking a judge to rate overall procedural quality, ask a targeted question: "Does this procedure contain any critical failure?" This gives the evaluation clear task-level validity and avoids rewarding surface qualities like coherence or formatting.

3. Distillation for reproducibility: Rather than relying on proprietary frontier models as judges (which are expensive, may change behavior over time, and are not reproducible), distill the judge into an open 8B model (How2Judge). This makes evaluation cheap enough for large-scale use (~$0 for local inference vs.$15 for GPT-5 on 7K examples) and ensures that results are reproducible — critical for a benchmark intended to support model comparison and RL training.

4. Closed-loop improvement: The same components used for evaluation (How2Score, How2Judge) also serve as a reward signal for RL training on procedure generation. This closes the loop: mine procedures from the web → evaluate model generation quality → use the evaluation signal to improve the model → re-evaluate. The paper demonstrates that this loop yields significant improvements (+10 points on How2Bench) without regressions on standard benchmarks, establishing that procedural generation is a capability that can be targeted and improved without sacrificing general performance.

The paper positions this framework as a "worked example of how pretraining web data can support a closed loop of capability evaluation and improvement at scale" (end of Section 1). This is a broader claim: the web provides a virtually unbounded supply of real-world reference documents, and by developing task-specific evaluation protocols that anchor on those references, we can turn otherwise hard-to-measure behaviors into practical development loops. The procedural domain is the test case, but the architecture — mine, evaluate, distill, improve — is presented as generalizable.

The Gap This Paper Fills (In Concrete Terms)

Before How2Everything, if you wanted to evaluate whether an LLM could generate valid step-by-step procedures, your options were:

• Use BLEU or perplexity against a reference (cheap but miscalibrated to validity)
• Hire human annotators (accurate but prohibitively expensive at scale)
• Use a narrow-domain dataset like cooking recipes (clean but not representative of real-world usage)
• Use a single-source dataset like WikiHow (limited coverage, potential format bias)
• Use an LLM-as-a-judge with a generic quality prompt (may reward surface features over correctness)

After How2Everything, you can use How2Judge — a small, open, reproducible model — to score generated procedures by detecting critical failures, achieving ~80% agreement with human annotators at effectively zero cost, on a benchmark spanning 14 real-world topics with procedures mined from diverse web sources. And you can use that same judge as a training signal to improve procedure generation via RL. This fills the reliability–scalability gap that the paper argues has held back work on procedural validity, and it provides the infrastructure to study procedural generation as a core LLM capability — not just a narrow application.

3. Technical Approach

3.1 Reader Orientation

How2Everything is a modular framework that mines procedural knowledge from web-scale corpora, transforms it into structured evaluation and training data, and then uses a distilled LLM judge to both evaluate and improve models' ability to generate step-by-step procedures. The system solves the fundamental problem of evaluating open-ended procedural validity at scale by replacing expensive human annotation or unreliable surface-form metrics with a targeted failure-detection protocol that asks a single, well-defined question: does this generated procedure contain any critical mistake that would prevent achieving the goal? — and then distills that protocol into a small, open-source model (How2Judge, an 8B parameter model fine-tuned from Qwen 3 8B) that can score thousands of examples at effectively zero cost while maintaining 80.5% agreement with human annotators.

3.2 Big-Picture Architecture (Diagram in Words)

The system has five major components connected in a pipeline that flows from raw web data to improved model capabilities:

1. How2Mine — A multi-stage extraction pipeline that takes a large corpus of web documents as input and outputs structured procedure instances: tuples of (topic, goal, resource list, reference steps). It uses LLM-based filtering, heuristic checks, and post-processing to convert noisy web text into standardized evaluation and training data.

2. How2Bench — A 7,000-example evaluation set created by sampling 500 procedures from each of 14 topics in the How2Mine output. This is the fixed test set used to benchmark model performance.

3. How2Train — The remaining ~344K procedures from How2Mine, used as training data for supervised fine-tuning (SFT) and as the prompt pool for reinforcement learning (RL).

4. How2Score — An evaluation protocol that defines critical failures and provides a structured prompt for an LLM judge to detect them. It takes a (goal, resources, reference procedure, generated procedure) tuple and outputs a binary label: has_failure (at least one critical error identified) or no_failure (no critical errors found). The score for a dataset is the fraction of examples labeled no_failure.

5. How2Judge — A distilled 8B model (fine-tuned from Qwen 3 8B) that executes the How2Score protocol. It is trained on 73K GPT-5 annotations of model-generated procedures, achieving 80.5% agreement with human annotators while costing approximately $0 for local inference (compared to ~$15 for GPT-5 on 7,000 examples). It serves dual purpose: as the evaluation engine for How2Bench and as the reward function for RL training.

Information flow during evaluation: A model receives a goal, resource list, and required step count → the model generates $N$ steps → How2Judge scores the generation by checking for critical failures against the reference procedure → the binary outcome (has_failure / no_failure) is recorded → success rates are aggregated across the 7,000 examples in How2Bench.

Information flow during RL training: A training example (goal + resources + reference from How2Train) is sampled → the model generates a procedure → How2Judge scores it against the reference → the binary score is converted to a reward → policy gradient updates the model → repeat for ~1,000 optimizer steps.

3.3 Roadmap for the Deep Dive

• First, How2Mine — the data pipeline that converts 980K web documents into 351K structured procedure instances. This is the foundation: everything downstream depends on having clean, diverse, realistic procedures as references.
• Second, How2Score — the evaluation protocol that defines what constitutes a critical failure and how judges should identify them. Understanding this definition is essential because it governs both evaluation and RL reward.
• Third, How2Judge — the distillation process that converts expensive, non-reproducible frontier-model judging into a cheap, open, reproducible 8B model. This is the scalability linchpin.
• Fourth, How2Bench evaluation setup — how models are prompted, how length is controlled, and how binary scores are aggregated.
• Fifth, RL training setup — how How2Score is converted into a reward signal, the three reward components (correctness, format, length), and the GRPO algorithm.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a systems and methodology paper whose core contribution is an integrated framework — data pipeline + evaluation protocol + distilled judge + RL training loop — that enables scalable, reproducible measurement and improvement of procedural generation.

---

How2Mine: From Web Documents to Structured Procedures

How2Mine is a five-stage pipeline that converts raw web documents into structured procedure instances. Each stage eliminates noise and standardizes format, with the overall pipeline processing 980K documents to yield 351,162 instances (approximately 36% yield) at a total API cost of $5,717 using the OpenAI batch API with GPT-4.1 (Section 3.2).

Source document selection (Stage 0). The pipeline begins by sampling documents from the DCLM web corpus (Li et al., 2025). To enrich for procedural content, only documents classified as Tutorial & How-to Guide by the WebOrganizer format classifier (Wettig et al., 2025) are retained. This is a pragmatic choice: tutorial-style documents have a high density of explicitly ordered, imperative steps, making them easier to extract procedures from reliably. However, the paper explicitly notes that How2Mine "can be easily extended to extract valid procedures from other formats such as academic writing and knowledge articles" (Section 3.1 footnote), and the restriction to tutorial format is for simplicity, not a fundamental limitation. The classification is performed using WebOrganizer's topic model, which assigns each document to one of 14 predefined topics. The pipeline then performs stratified sampling across these 14 topics to ensure equal topical representation in the final dataset, preventing any single domain (e.g., cooking recipes) from dominating.

The 14 topics covered are: Art & Design, Crime & Law, Education & Jobs, Electronics & Hardware, Fashion & Beauty, Food & Dining, Health, Home & Hobbies, Industrial, Religion, Science/Math & Technology, Sports & Fitness, Transportation, and Travel & Tourism. The stratified sampling ensures that the final pool of 351K procedures spans 189K unique domains (with the top 10 domains per topic reported in Appendix B.1, showing substantial diversity — for example, the top domain for Food & Dining is recipe-finder.com with 1,065 instances, and the top domain for Science/Math & Technology is education.com with 361 instances).

Stage 1: Procedure extraction. Given a candidate web document, GPT-4.1 first determines whether the document contains a well-formed sequential procedure and, if so, extracts the goal and an ordered list of steps. The LLM prompt (provided in the appendix) instructs the model to identify the overall objective and enumerate the actions in order. This stage is the primary source of yield loss: many tutorial-style documents contain procedural content that is not structured as clean sequential steps (e.g., comparative reviews, general advice, troubleshooting guides), and these are filtered out.

Stage 2: Heuristic filters. Two simple rule-based checks are applied to the extracted steps:

1. Step count bounds: Procedures with fewer than 5 or more than 15 steps are discarded. The lower bound removes trivial procedures (e.g., "How to boil water" in 2–3 steps) that would not test meaningful procedural reasoning. The upper bound removes overly complex procedures that would be difficult for both models and judges to handle reliably (long procedures increase the surface area for potential failures and make human annotation more challenging).

2. N-gram overlap: Procedures with high n-gram overlap within their extracted steps are removed. This catches degenerate cases where the extraction produces near-duplicate steps (e.g., "Add sugar to the mixture" followed by "Add sugar to the blended mixture"), which likely indicate extraction errors rather than genuine procedural content. The specific n-gram overlap threshold and implementation details are in the appendix.

Stage 3: LLM-based filtering. An LLM-based filter (again using GPT-4.1) excludes examples that exhibit any of six specific disqualifying properties. These criteria were developed through "multiple rounds of data inspection" (Section 3.2), iteratively examining extracted procedures to identify common failure modes:

1. Dependence on specific named entities: Procedures that only work for a particular person, organization, or location (e.g., "How to apply for a job at Google" — the steps would not generalize).
2. Purely mathematical calculations: Procedures that reduce to arithmetic or algebraic computation (e.g., "How to calculate compound interest"). These are better evaluated by math benchmarks.
3. Requires interacting with UI elements: Procedures that involve clicking buttons, navigating menus, or using specific software interfaces (e.g., "How to create a pivot table in Excel"). These are inherently executable but in ways that a text-only model cannot reliably represent.
4. Open-ended creative generation: Procedures that involve generating creative content where "correctness" is subjective (e.g., "How to write a poem about nature").
5. Non-sequential: Procedures where the steps have no meaningful ordering constraint (e.g., "How to pack for a trip" — most packing actions can happen in any order).
6. Unreasonable or nonsensical: Procedures that are contradictory, logically impossible, or otherwise invalid.

Each of these criteria is evaluated independently, and the presence of any one disqualifies the entire procedure. This filtering stage is critical because downstream evaluation depends on the reference procedure being a coherent, sequential plan against which generated procedures can be meaningfully compared. If the reference itself is nonsensical, detection of critical failures becomes impossible.

Stage 4: Post-processing and resource extraction. For each remaining procedure, the pipeline performs two refinement operations:

• Goal rewriting: The goal is rewritten to be "as specific and deterministic as possible, explicitly stating the required constraints and expected outcome" (Section 3.2). For example, a generic goal like "How to sell your share of an apartment" might be rewritten to "[Crime & Law] Sell your share of a common property apartment by following the required legal procedure for notifying co-owners and transferring ownership." The topic prefix (e.g., [Crime & Law]) is added for downstream organization. This rewriting reduces the space of valid solutions and makes it easier for both models and judges to determine whether the generated procedure is acceptable. The goal is to make the evaluation setup approximately deterministic: given this specific goal with these specific constraints, there should be a relatively narrow set of valid procedural approaches, with the reference procedure anchoring one valid path.

• Resource extraction: The pipeline identifies and lists the resources (if any) referenced by the steps in the reference procedure. For example, in the crime & law procedure from Table 1, the resources are "notary; notarial document; letter with a list of contents; receipt." These resources serve as soft constraints: a valid generated procedure should reference these resources (or reasonable alternatives), and a failure to mention a legally required resource (like a notarial document) would likely constitute a critical failure. The resource list is included in the prompt to the model at inference time to condition generation and constrain the solution space.

Together, the rewritten goal and resource list serve to "narrow the space of valid solutions" (Section 3.2). This is a deliberate design choice: rather than evaluating open-ended procedural generation in the wild (where "How do I sell my apartment share?" could have many valid answers depending on jurisdiction, specific circumstances, and level of detail), the framework creates a controlled evaluation setup where there is a clear, well-specified task with deterministic success criteria. This makes the evaluation tractable while preserving the essential challenge: generating a complete, correct sequence of steps that achieves the stated goal under the stated constraints.

Stage 5: Final validation. A final LLM-based sanity check (using GPT-4.1) removes any remaining nonsensical or otherwise invalid procedures that may have passed through earlier stages. This is a lightweight filter to catch edge cases that the heuristic and LLM filters might have missed — for example, procedures that are syntactically valid but contain internal contradictions that only become apparent when the full procedure is considered holistically.

Pipeline output format. Each procedure instance in the final pool is a structured record with four fields:

• topic: One of 14 categories (e.g., "Crime & Law")
• goal: The rewritten, constraint-specified objective (e.g., "[Crime & Law] Sell your share of a common property apartment by following the required legal procedure for notifying co-owners and transferring ownership.")
• resources: A list of items referenced by the procedure (e.g., "notary; notarial document; letter with a list of contents; receipt")
• steps: An ordered list of reference steps (e.g., ["1. Prepare a notification to all co-owners stating the conditions of sale of your share.", "2. Visit a notary to draw up a notarial document including all sale conditions.", ..., "5. Sell your share to a third party."])

Data split. From the 351K total instances, How2Bench is constructed by sampling 500 instances per topic (7,000 total). The remaining ~344K instances form How2Train, the training pool. The sampling for How2Bench is balanced across topics (500 each) to ensure that aggregate benchmark scores are not dominated by topics that happen to have more procedures. The training set used for RL (100K examples, described in Section 6.1) is further subsampled from How2Train, balanced across topics and deduplicated against How2Bench using semantic similarity to prevent train-test leakage (specifics in Appendix — the paper mentions embedding-based similarity thresholds).

Design choices and their justifications:

• Why 5–15 steps? The bounds were determined empirically through data inspection. Fewer than 5 steps rarely requires meaningful multi-step reasoning; more than 15 steps becomes difficult for both models to generate coherently and for judges to evaluate reliably (since the probability of containing at least one critical failure increases with procedure length).
• Why LLM-based filtering instead of purely heuristic? Many disqualifying properties (e.g., dependence on named entities, non-sequentiality) require semantic understanding that simple pattern matching cannot provide. The paper acknowledges the cost of LLM-based filtering but argues it is necessary for data quality, and the batch API makes it economically feasible ($5,717 total for the full pipeline).
• Why goal rewriting? A generic goal admits too many valid solutions, making it impossible for a judge to determine whether a deviation from the reference is an acceptable alternative or a critical failure. By making goals specific and deterministic, the framework creates a proxy for ground-truth correctness: while the reference is not the only valid procedure, the space of valid alternatives is narrow enough that the judge can reliably identify failures.
• Why topic stratification? Without stratification, certain high-density topics (e.g., cooking recipes from food domains, DIY projects from instructables.com) would dominate the dataset, and benchmark scores would primarily reflect domain-specific rather than general procedural capability. The 14 topics provide broad coverage of everyday procedural domains while keeping the benchmark size manageable (7,000 examples).

---

How2Score: Defining and Detecting Critical Failures

How2Score is the evaluation protocol that determines whether a generated procedure is valid. It does not assign a quality score or compare procedures — it makes a single binary determination: does this procedure contain any critical failure that would prevent achieving the stated goal?

The definition of a critical failure. The paper defines this concept through a combination of formal specification and worked examples. Formally (Section 4.1):

"We define a critical failure as an omission, extraneous action, contradiction, severe vagueness, or other deviation from the reference that is severe enough to prevent achieving the goal, or to make the procedure unusable as instructions."

Several elements of this definition merit unpacking:

• Omission: A required step from the reference is entirely missing from the generation, and that step is necessary for achieving the goal. For example, in Table 1's crime & law example, the model omits the 30-day waiting period that is legally required before transferring ownership — without it, the procedure is legally invalid.
• Extraneous action: The generation includes a step that is not in the reference and that would interfere with achieving the goal. This is rarer than omission and requires careful judgment (most extra steps that are harmless are not critical failures).
• Contradiction: Two steps in the generated procedure contradict each other, making it impossible to execute both as specified. For example, "cut the wood board into 5 pieces of equal size" followed by "place the largest piece on top and the smallest piece on the bottom" — if the pieces are equal, there is no largest or smallest.
• Severe vagueness: A step is too underspecified to be executable. For example, in Table 1's cooking example, the model says "Cook on low heat, stirring occasionally, until the glaze thickens" without specifying times or lid/heat settings — information that is critical because the recipe requires a two-phase cooking process (15 minutes on high with lid, then 2–3 hours on low with lid off) that cannot be approximated by "until the glaze thickens."
• Other deviations: The definition is deliberately open-ended to accommodate edge cases that the codebook might not anticipate. The protocol provides examples to guide judges.

The role of the reference procedure. The paper is explicit that the reference procedure is an anchor, not a gold standard. Section 4.1 states:

"We use the reference procedure as an anchor, but aim not to penalize alternative valid procedures or superficial differences."

This is crucial: a generated procedure might use different wording, combine or split steps, include additional (harmless) details, or even follow a different valid approach while still achieving the goal. The judge's task is to distinguish between acceptable variations and critical failures. For example, if the reference says "Add powdered sugar and stir until completely blended" and the generation says "Mix in the powdered sugar until the mixture is smooth," this is not a critical failure — it is a superficial rewording of the same action. Conversely, if the generation says "Add granulated sugar and stir" (changing the ingredient type), this is a critical failure because granulated sugar would not produce the required glaze.

The paper provides a concrete example in Section 4.1: if the goal is to make a terracotta pot as a gift, a different gift message than the one in the reference is not a critical failure (it is an acceptable creative variation), while omitting the step that fires the pot in a kiln would be a critical failure (the pot would not harden).

Assumption of reference correctness. The paper acknowledges that despite rigorous filtering, "some references can still contain errors" (Section 4.1). To quantify this, the authors prompt GPT-4.1 to judge whether each How2Bench reference procedure reasonably achieves the stated goal, and 96.6% are accepted as valid. The remaining 3.4% are retained in the benchmark, meaning that How2Score may inherit some noise from reference errors — if the reference itself contains a mistake, the judge might flag a correct generation as having a "failure" because it deviates from the erroneous reference. The paper treats this as an acceptable level of noise, noting that "in our formulation, we use the reference procedure to make the task more deterministic and suitable for evaluation, not as a perfect ground-truth solution" (Section 4.1).

The evaluation protocol in detail. Given an evaluation example $x = (g, R, S, \hat{S})$ where $g$ is the goal, $R$ is the resource list, $S$ is the reference procedure with $n$ steps, and $\hat{S}$ is the model-generated procedure (also with $n$ steps due to length control), the judge performs the following:

1. Receives a detailed prompt (provided in the appendix) that includes the annotation codebook, examples of critical vs. non-critical failures, and the specific instance to evaluate.
2. Produces a structured output listing all critical failures identified, each accompanied by a description and references to relevant steps in $S$ and $\hat{S}$.
3. From this list, a binary label is derived: has_failure if at least one critical failure is identified, and no_failure otherwise.

The binary aggregation is formalized as:

$\text{Score}(D) = \frac{1}{|D|} \sum_{x \in D} \mathbb{I}\bigl[J(g, R, S, \hat{S}) = \texttt{no\_failure}\bigr]$

where $D$ is the evaluation dataset, $J(\cdot)$ is the binary judgment function (the judge's derived output), and $\mathbb{I}[\cdot]$ is the indicator function that equals 1 when the condition is true and 0 otherwise.

What it computes: the fraction of examples in the dataset for which the judge finds no critical failures. This is a success rate: if Score(D) = 0.45, it means 45% of generated procedures were judged to contain no critical failures. For each individual example, the model either succeeds (no critical failure identified) or fails (at least one critical failure identified).

Why this binary aggregation: The paper considered but rejected the alternative of localizing the first critical failure (as is common in process reward models for mathematical reasoning, where the first incorrect step is identified). In preliminary experiments, requiring annotators to agree on the location of the first failure produced much lower inter-annotator agreement (Krippendorff's $\alpha = 0.307$) compared to binary aggregation ($\alpha = 0.593$). The reason is that in open-world procedures, multiple valid orderings may exist, and two annotators might identify different failures in the same generation while agreeing that it has at least one failure. Binary aggregation captures this agreement while being robust to disagreement about which specific element constitutes the "first" failure.

Why not a quality score? A continuous quality score (e.g., "rate the procedure from 1–5") would be harder to calibrate across annotators (what does a "3" mean, exactly?), more susceptible to surface-form biases (well-written procedures might receive higher scores even with errors), and harder to use as a clean RL reward. Binary failure detection gives clear signal: the model either succeeded or failed on this example. This simplicity is specifically designed to make How2Score usable as both an evaluation metric and a training signal.

Human validation of the protocol. To validate How2Score's definition of critical failures, the paper conducts a human annotation study on 200 examples (Section 4.3). Three annotators are recruited via Prolific, pre-screened to avoid procedures requiring specialized domain knowledge, and paid an average of $28/hour (total cost:$3,600). The annotation process involved iterative refinement:

• Early pilots (300 annotations): Initial inter-annotator agreement was low (Krippendorff's $\alpha = 0.273$). The paper identifies two common failure modes in these early annotations: (1) annotators flagging any difference from the reference as critical, regardless of whether it would actually prevent goal achievement, and (2) annotators overlooking indisputable failures that were masked by coherent surface form (the paper gives the example of the contradictory wood-cutting procedure, where the contradiction is embedded in fluent text and easy to miss on casual reading).

• Iterative refinement: The training materials were expanded with more examples clarifying the boundary between non-critical variations (acceptable rewordings, harmless additions, valid alternative approaches) and critical failures (omissions of required steps, incorrect parameters, contradictions, severe vagueness).

• Final qualification: A short qualification test was used to screen annotators, and the three who best demonstrated understanding of the criteria were selected for the final 200-example labeling task.

With binary score aggregation in the final round, Krippendorff's $\alpha = 0.593$. The paper contextualizes this value: "Given the non-executable, open-world setting and the existence of multiple valid procedures per goal, we do not expect near-perfect agreement; instead, we target a metric that is stable for relative comparisons and usable as an RL reward" (Section 4.3). An $\alpha$ of 0.593 is in the range typically considered "moderate agreement" and is sufficient for the framework's purposes: the metric need not be perfectly calibrated at the individual-example level to produce meaningful aggregate scores, reliable model rankings, and a useful RL reward signal.

Evaluating LLM judges against human labels. Five frontier LLM judges (GPT-5, GPT-4.1, Gemini 2.5 Pro, Claude 4.5 Opus, and Claude 4.5 Sonnet) are evaluated on the same 200 examples and compared against the human majority label. The results (Figure 3):

• GPT-5 achieves the highest agreement at 83.0%, with good calibration across classes: 83.7% agreement on human-majority has_failure cases and 82.4% on no_failure cases.
• Leave-one-out human agreement (comparing each individual annotator against the majority of the other two) ranges from 84.7% to 88.5%, meaning GPT-5's agreement falls within approximately 1–5 percentage points of human-level performance.
• All five LLM judges achieve agreement between 76.5% and 83.0%, indicating that the critical failure detection task is within the capability range of frontier models but with meaningful variation in accuracy.

The calibration across classes is particularly important because an unbalanced judge (e.g., one that almost always says no_failure or one that almost always says has_failure) would produce misleading aggregate scores even with high overall agreement. GPT-5's near-equal performance on both classes (83.7% vs. 82.4%) suggests it is not systematically biased toward leniency or strictness.

---

How2Judge: Distilling a Cost-Effective, Reproducible Judge

While GPT-5 achieves strong agreement with human labels, using it to evaluate How2Bench's 7,000 examples costs approximately $15 per full benchmark evaluation. For large-scale use — evaluating dozens of model checkpoints, or (critically) generating reward signals during RL training where the judge must score thousands of generated procedures per training run — these costs become prohibitive. Moreover, proprietary API-based judges are not reproducible: model versions change, API behavior may shift, and results cannot be independently verified without access to the same API endpoint. These concerns make API-based judges unsuitable for a benchmark intended to support long-term model comparison and open research.

Distillation process. The solution is to distill the frontier judge (GPT-5) into a smaller, open model that can be run locally at negligible cost and shared for reproducibility. The distillation proceeds as follows:

1. Collect teacher annotations: GPT-5 is used to annotate 73K generated procedures from a diverse set of generator models. The generators include three 1B checkpoints, four 7B checkpoints, three 32B checkpoints, and four closed-source models, ensuring that the training data covers a wide range of generation quality and failure types. This diversity is important: if the distillation data only contained outputs from weak models (which tend to make obvious errors), the student might not learn to detect subtle failures that stronger models produce.

2. Deduplication: Any examples overlapping with the human-annotated set from Section 4.3 are removed to prevent contamination of the human-agreement evaluation. This ensures that How2Judge's reported 80.5% human agreement is measured on examples it was never trained on.

3. Fine-tuning: Qwen 3 8B (the base model) is fine-tuned on the 73K GPT-5 annotations for three epochs. The task is sequence-to-sequence: given the evaluation prompt (goal, resources, reference, and generated procedure), output the structured critical failure analysis. The paper uses standard supervised fine-tuning with the hyperparameters described in the appendix.

4. Evaluation on human-annotated set: The resulting How2Judge model is evaluated on the same 200 human-annotated examples used to validate GPT-5. How2Judge achieves 90.5% agreement with GPT-5 (teacher-student agreement) and 80.5% agreement with the human majority label. It is similarly well-calibrated across classes: 79.6% human agreement on has_failure and 81.4% on no_failure (Figure 3).

Why three epochs? The paper does not provide extensive ablation of training duration, but the three-epoch setting is standard in instruction fine-tuning and likely balances overfitting to the teacher's specific annotation style against underfitting to the task. The 90.5% teacher-student agreement suggests the 8B model successfully captures most of GPT-5's behavior, while the small drop from 83.0% (GPT-5 human agreement) to 80.5% (How2Judge human agreement) suggests that some of GPT-5's nuanced judgment does not fully transfer to the smaller model. The paper treats this as an acceptable tradeoff: a 2.5-point drop in agreement in exchange for effectively free, reproducible evaluation.

Why Qwen 3 8B? The paper chose this base model because it is open-weight, moderately sized (enabling fast inference), and a strong general-purpose model that can be fine-tuned for structured output tasks. The 8B size is specifically chosen as a balance point: larger models would be slower and more expensive to run (undermining the goal of cheap evaluation), while smaller models might lack sufficient capacity to perform the nuanced judgment required. The distillation experiment demonstrates that 8B parameters are sufficient to capture most of GPT-5's critical-failure detection capability.

Operational characteristics. Once distilled, How2Judge can evaluate How2Bench's 7,000 examples using local GPU inference at approximately zero marginal cost (beyond the electricity and hardware amortization of running inference). For comparison:

• Human annotation: ~$126,000 (7,000 ×$18/example at the paper's annotation rate)
• GPT-5 API: ~$15 (7,000 ×$0.00214/example, based on the paper's cost estimate)
• How2Judge: ~$0 (local inference)

More importantly for RL training, How2Judge can score generated procedures fast enough to serve as an online reward signal — each training step requires scoring multiple generations (the paper uses a group size of 8 in GRPO), and a full training run of 1,000 steps with a group size of 8 requires approximately 8,000 judge calls. At GPT-5 prices, this would cost roughly $17 per training run in judge API costs alone, but the bigger issue is latency: API calls would bottleneck the training loop. Local How2Judge inference removes both the cost and latency constraints.

Reproducibility implications. The paper emphasizes that How2Judge addresses a key limitation of LLM-as-a-judge protocols: non-reproducibility. Proprietary models can change behavior silently (model updates, prompt format changes, temperature adjustments), making it impossible to exactly replicate benchmark results across time. How2Judge, as an open-weight model with fixed weights and a published prompt format, enables fully reproducible evaluation — any researcher with the model weights can exactly replicate the paper's benchmark scores.

Self-preference bias check (across judges, not just How2Judge). A common concern with LLM-as-a-judge evaluation is that judges might systematically favor outputs from their own model family (Zheng et al., 2023). While How2Judge is a Qwen 3 8B model and could theoretically exhibit bias toward Qwen-generated procedures, the paper's cross-judge robustness analysis (Figure 5) provides reassurance: when rescoring the same set of generated procedures with four different judges (How2Judge, GPT-5, Gemini 2.5 Pro, Claude 4.5 Opus), the relative model rankings remain unchanged even though absolute score values vary. This invariance of ranking across judge families suggests that self-preference bias, if present, is not strong enough to distort the benchmark's primary use case — comparing model performance. The paper does not report a specific self-preference analysis for How2Judge in isolation (e.g., comparing How2Judge scores of Qwen generators vs. non-Qwen generators against a human baseline), which is a minor limitation of the analysis.

---

How2Bench: Evaluation Setup and Inference Controls

How2Bench is the fixed evaluation set of 7,000 procedure instances (500 per topic). The evaluation protocol for models is standardized to reduce confounding variables and ensure comparability across models.

Inference prompt structure. At inference time, the model receives a prompt containing:

• The goal $g$ (e.g., "[Crime & Law] Sell your share of a common property apartment...")
• The resource list $R$ (e.g., "Resources: notary; notarial document; letter with a list of contents; receipt")
• The required step count $n = |S|$, the number of steps in the reference procedure
• Instructions to output exactly $n$ steps, each as a single concise sentence containing one main action
• Few-shot examples demonstrating the expected output format and concision level

The paper acknowledges that conditioning on $R$ and $n$ "may not reflect real-world usage" but argues that these are "evaluation controls to reduce degrees of freedom and improve comparability across model outputs" (Section 5.1). In real usage, a user would not specify "give me exactly 7 steps" — but without this constraint, models might produce procedures of wildly varying lengths (some might give 3 steps, others 15), making it difficult to compare quality. By fixing the step count to match the reference, the evaluation isolates the model's ability to identify the right steps and order them correctly, rather than its ability to judge how many steps are needed.

Length control. Each step is required to be "a single, concise sentence containing one main action," with the model instructed to "closely follow the concision level in the provided examples." This prevents models from gaming the evaluation by generating excessively detailed steps (which might appear more thorough to a judge but could artificially inflate scores through verbosity bias — a known failure mode in LLM-as-a-judge settings).

Format verification. The paper also imposes step-number format compliance: each generated step must begin with a number followed by a period (e.g., "1.", "2."), and the total number of labeled steps must match $n$. This is a lightweight structural check — it does not assess content validity but ensures that the model follows the basic formatting conventions of the task. During RL training, this format compliance is used as a separate reward component (format verifier reward).

How2Score computation for How2Bench. For each of the 7,000 examples, the model generates $\hat{S}$ and How2Judge scores it against the reference $S$. The binary score is computed per example, and the aggregate How2Bench score is the fraction of the 7,000 examples judged no_failure:

$\text{How2Bench} = \frac{\text{count}(\texttt{no\_failure})}{7000}$

This produces a single number between 0 and 1 (typically reported as a percentage in the paper) that represents the model's success rate at generating valid procedures.

Generator models evaluated. The paper evaluates a range of models on How2Bench (Figure 4):

• Closed-source: GPT-5, GPT-4.1, Gemini 2.5 Pro, Claude 4.5 Opus, Claude 4.5 Sonnet
• Open-source (instruct): Qwen 3 8B Instruct, Qwen 3 4B Instruct, Olmo 3 7B Think, Olmo 3 7B Instruct, Qwen 2.5 7B Instruct
• Open-source (base): Corresponding base (non-post-trained) checkpoints for the above
• Intermediate checkpoints: Multiple pretraining, midtraining, and post-training snapshots from five Olmo training runs (Olmo 2 1B/7B/32B; Olmo 3 7B/32B) to study scaling behavior (Figure 1b)

Selected results from Figure 4:

• GPT-5 leads at approximately 65% (estimated from Figure 4 — exact numbers require the figure, which is referenced but not numerically quoted in the text).
• Qwen 3 8B Instruct (the base model for How2Judge) achieves 38.52%, with the base checkpoint substantially lower.
• The gap between base and instruct variants is consistent across model families, indicating that post-training (instruction tuning, RLHF) significantly improves procedural generation.
• Smaller models (1B parameters) can achieve non-trivial performance: the Olmo 2 1B checkpoint reaches approximately 3–5% by the end of pretraining.

Scaling behavior detail (Figure 1b, referenced in Section 5.2). The paper reports that How2Bench "exhibits smooth scaling across both model size and training stage" with a consistent ordering of model performance across five Olmo training runs. Non-trivial performance emerges by about 5% into pretraining for a 1B model (around $10^{21}$ training FLOPs), after which performance continues to improve throughout pretraining, midtraining, and post-training. This is significant because it demonstrates that How2Bench provides signal across the entire training pipeline, from early pretraining through final post-training, making it suitable for performance forecasting and studying how procedural capabilities develop during training.

Why this matters for benchmark design. Many existing benchmarks either saturate early (models quickly reach near-ceiling performance) or target only frontier models (near-zero performance at smaller scales, providing no differentiation). How2Bench's ability to discriminate across a wide range of model sizes and training stages — from 1B models early in pretraining to GPT-5 — is a desirable property identified in recent work on benchmark quality (Heineman et al., 2025; Xu et al., 2025). It means the benchmark can be used not just for leaderboard-style comparisons but for studying scaling laws and training dynamics of procedural capabilities.

---

RL Training: Using How2Score as a Reward Signal

Beyond evaluation, How2Score and How2Judge serve as the reward function for RL training on procedure generation. The training setup uses Group Relative Policy Optimization (GRPO; Shao et al., 2024) with three reward components.

Training data. A training set of 100K examples is sampled from How2Train (the ~344K procedures not in How2Bench), balanced across the 14 topics and deduplicated against How2Bench using semantic similarity to prevent train-test leakage. The deduplication uses embedding-based similarity with Qwen3-Embedding-0.6B (Qwen Team, 2025b), removing training examples that are too similar to any How2Bench instance (specific threshold in appendix). This is important because procedures describing similar goals could appear on multiple web pages, and without deduplication, the RL training might effectively train on near-copies of test examples.

Models trained. RL training is applied to three models:

• Qwen 3 4B Instruct
• Qwen 3 8B Instruct
• OLMo 3 7B Think (the "thinking mode" variant that includes explicit intermediate reasoning)

These are trained for 1,000 optimizer steps using GRPO with a group size of 8 (8 generations per prompt, scored and compared against the group mean).

The three reward components. Each generated procedure receives a reward that is the sum of three terms:

1. How2Score reward (binary, correctness): The procedure is scored by How2Judge. If the judgment is no_failure, the reward is +1; if has_failure, the reward is 0. This is the primary signal driving procedural improvement.

2. Step-format verifier reward (binary, structural): A simple rule-based check verifies that the generation follows the expected format: each step begins with a number followed by a period, and the total number of steps matches the requested count $n$. If the format is correct, +0.1; otherwise 0. This small structural reward helps the model learn the expected output format without letting format compliance dominate the total reward.

3. Reference-calibrated length reward (continuous, anti-gaming): This reward penalizes generations that deviate substantially from the reference procedure's length. It is designed to prevent length gaming — a known failure mode in LLM-as-a-judge settings where models learn to generate longer, more detailed procedures that appear more thorough to the judge and thus receive higher scores, without actually improving procedural correctness.

The length reward is calibrated so that when the generated procedure's token length equals the reference's token length, the reward is 0 (no penalty). As the generation length deviates from the reference length, a penalty is applied. The precise formulation is in the appendix, but the operational effect is that models are incentivized to match the reference's verbosity level, preventing them from exploiting verbosity bias in the judge.

Why three reward components? The decomposition addresses different aspects of the generation task:

• The How2Score reward provides the core signal: generate procedures without critical failures. This is the capability the training aims to improve.
• The format verifier reward provides a weak structural signal that helps the model learn basic output conventions. Without it, the model might generate procedurally valid content but in an unparseable format (e.g., missing step numbers, wrong step count), which would make evaluation inconsistent.
• The length reward prevents reward hacking. Without it, models learn to generate verbose procedures (sometimes 1.3×–1.5× the reference length) that exploit the judge's verbosity bias, producing artificially inflated How2Bench scores that do not reflect genuine procedural improvement. The paper reports (Section 6.2) that models trained without the length reward inflate their output length to 1.34×–1.53× the reference and achieve large apparent How2Bench gains, but these gains largely vanish when controlled for length — they are artifacts of the judge's bias, not real improvements.

Why GRPO? GRPO is a variant of policy gradient methods that compares a group of outputs for the same prompt to compute relative advantages, rather than requiring a learned value function (as in PPO). For this task, GRPO is appropriate because:

• The reward signal is sparse (binary 0/1 from How2Judge) and noisy (the judge makes errors), making advantage estimation from a learned value function difficult.
• Group-relative comparison stabilizes training: even if the judge's absolute scores are noisy, the relative ordering of a group of generations tends to be more reliable (the best generation in the group is likely genuinely better than the worst).
• GRPO has been shown effective for reasoning tasks (DeepSeekMath, Shao et al., 2024), and procedure generation shares structural similarities with multi-step reasoning.

Training hyperparameters. The paper specifies:

• 1,000 optimizer steps (approximately 8,000 total judge calls: 1,000 steps × group size 8)
• Group size: 8 (8 parallel generations per prompt for advantage estimation)
• Learning rate and other optimizer settings in appendix
• The RL training is applied directly to the instruct/think checkpoints without an intervening SFT stage (the SFT stage is explored separately and found to yield limited gains on instruct checkpoints, as discussed in Section 6.2)

Why no SFT before RL on instruct checkpoints? The paper reports that SFT yields small gains on base model checkpoints (likely because it teaches the specific output format and task structure) but does not improve instruction-tuned checkpoints. The hypothesized reason is "objective mismatch: SFT maximizes likelihood of a single reference text per goal, which need not align with minimizing critical failures under How2Score" (Section 6.2). In other words, an instruction-tuned model may already generate well-formatted, superficially plausible procedures, and SFT on reference procedures just reinforces the surface patterns without addressing the specific failure modes that How2Score penalizes. RL, by contrast, directly optimizes for the How2Score metric, providing signal about which generations are actually correct rather than which are similar to the reference.

Results summary. As shown in Table 2, RL training with How2Score as the reward improves How2Bench performance by more than 10 points across all three models:

• Qwen 3 4B Instruct: 30.29 → 43.52 (+13.23)
• Qwen 3 8B Instruct: 38.52 → 48.62 (+10.10)
• OLMo 3 7B Think: 27.30 → 37.89 (+10.59)

---

Design Choices Summary: Why This Architecture?

The How2Everything architecture makes several non-obvious design choices that merit explicit justification:

Why mine from web corpora rather than curate from WikiHow? WikiHow (used by prior work like Brahman et al., 2024) provides clean, well-structured procedures, but it represents a biased sample of procedural knowledge. Tutorial-style web documents span a much wider range of domains and are drawn from the same distribution LLMs see during pretraining, making them more ecologically valid as reference procedures. The cost is noise — web documents require heavier filtering — but the paper demonstrates that the five-stage How2Mine pipeline can extract clean procedures at scale (351K from 980K documents) while maintaining quality (96.6% reference validity as judged by GPT-4.1).

Why binary failure detection rather than a quality score? A continuous quality score would provide finer-grained signal but introduces calibration challenges that are especially problematic when the metric must serve as both an evaluation benchmark and an RL reward. Binary failure detection is (1) easier to define and achieve reasonable inter-annotator agreement on, (2) less susceptible to surface-form biases (a well-written but wrong procedure receives the same 0 as a poorly-written wrong one), and (3) provides a clean reward signal (0/1) for RL. The paper's finding that binary aggregation yields substantially higher annotator agreement than first-failure localization ($\alpha = 0.593$ vs. $0.307$) empirically validates this choice.

Why distill into 8B rather than use a larger open model? The 8B parameter size is a deliberate sweet spot: large enough to capture most of GPT-5's judgment capability (90.5% teacher-student agreement), small enough to run quickly on commodity hardware for both evaluation and RL training. A 70B open model might achieve higher agreement with GPT-5 but would be slower and more expensive to run, defeating the purpose of distillation. The 2.5-point drop in human agreement from GPT-5 (83.0%) to How2Judge (80.5%) is acceptable given the dramatic cost reduction (effectively $0 vs.$15 per benchmark evaluation).

Why GRPO with three reward components rather than simpler RL? The sparse, noisy binary reward from How2Judge makes standard RL challenging — the model might go many steps with no positive reward signal, and false positives/negatives from the judge could misdirect the policy. GRPO's group-relative advantage estimation provides stable training signal even with sparse rewards, and the format and length rewards provide dense auxiliary signals that guide the model toward well-formed outputs while the correctness reward improves content. The three-component design explicitly reflects the paper's understanding of failure modes in LLM-as-a-judge settings (verbosity bias, self-preference) and actively defends against them.

Why condition on reference step count $n$ during evaluation? This is an evaluation control, not a feature meant for deployment. In real-world usage, a user would not specify "give me exactly 7 steps," but without this constraint, comparing models becomes difficult: a model that generates 3 overly-vague steps might score similarly to one that generates 12 detailed steps, but for different reasons. By fixing $n$ to the reference step count, the evaluation isolates the model's ability to (1) identify what the $n$ necessary actions are and (2) order them correctly — the core procedural reasoning skills. This is analogous to multiple-choice evaluation, where providing answer options controls for the open-endedness of generation while still testing the underlying capability.

4. Key Insights and Innovations

Innovation 1: Reframing Procedure Evaluation as Targeted Failure Detection Rather Than Quality Rating

The paper's most significant conceptual move is not a new model architecture or training algorithm — it is a reframing of the evaluation problem that makes scalable, reliable measurement possible where prior approaches failed.

Before How2Everything, the natural approach to evaluating procedural generation was to ask a judge to rate overall quality: "How good is this procedure on a scale of 1–5?" or "Is response A better than response B?" This is how LLM-as-a-judge protocols typically work (Zheng et al., 2023; Dubois et al., 2025), and it is intuitively reasonable — after all, human evaluations of generated text usually ask for holistic judgments. But the paper identifies a specific failure mode of this framing when applied to procedures: a quality-rating judge tends to reward surface qualities like coherence, fluency, and thoroughness, which are orthogonal to procedural validity. A well-written procedure that omits a legally required 30-day waiting period (as in Table 1's crime & law example) might receive a high quality score for being detailed and authoritative while being completely invalid.

The paper's alternative framing — "does this procedure contain any critical failure?" — is a diagnostic shift from assessment to detection. Rather than asking how good something is (which requires weighing multiple dimensions and is inherently subjective), it asks a specific, falsifiable question: is there a mistake that would prevent achieving the goal? This reframing has several properties that make it more suitable for scalable evaluation:

• It is easier to calibrate. Human annotators can more reliably agree on whether a specific error exists than on what overall quality score to assign. The paper's empirical evidence supports this: binary failure detection achieves Krippendorff's α = 0.593, while localization of the first failure (a closer analog to holistic judgment) achieves only α = 0.307 (Section 4.3). Moderate agreement is achievable for error detection; near-random agreement is what you get when you ask annotators to pinpoint which error matters most.

• It is less susceptible to surface-form bias. A verbose, well-formatted procedure with a critical omission receives the same binary label (has_failure) as a terse, poorly-written one with the same omission. The judge is directed to look for specific types of errors, not to form an overall impression. The paper's length-control experiments (Section 6.2) demonstrate the practical importance of this: without explicit anti-verbosity measures, models learn to inflate their output length by 1.34×–1.53× and achieve artificially high scores, revealing that even a failure-detection judge has residual verbosity bias that must be actively mitigated.

• It provides a clean training signal. Binary success/failure is a natural reward for RL — no need to calibrate a continuous score, no need to normalize across prompts of varying difficulty. The paper exploits this directly: How2Score's binary output maps cleanly onto a 0/1 RL reward (Section 6.1), making the same protocol serve as both evaluation metric and training objective without modification.

This reframing is not merely cosmetic. It is what enables the entire closed-loop architecture: the same definition of "correctness" used to benchmark models is also the signal used to improve them. In a quality-rating paradigm, the gap between "what the evaluation measures" and "what the training optimizes" would require bridging (e.g., converting a 1–5 quality score into a reward, which introduces calibration assumptions). In the failure-detection paradigm, the evaluation is the training objective — the RL agent literally optimizes for the probability that How2Score finds no critical failures. This tight alignment between measurement and optimization is a distinctive architectural contribution, not just a metric choice.

Comparison to prior work: Process reward models (PRMs) for mathematical reasoning (Lightman et al., 2023) also use error-detection framing — identifying the first incorrect step in a solution — but mathematical reasoning has ground-truth correctness (you can verify the final answer). The paper's adaptation of this framing to non-executable real-world procedures — where there is no oracle to verify correctness — is a generalization that required developing an operational definition of "critical failure" that works without ground-truth verification. The definition itself (omission, extraneous action, contradiction, severe vagueness, or other deviation severe enough to prevent goal achievement) is a conceptual contribution: it provides a rubric for evaluating procedural validity in open-world settings where execution-based verification is impossible, and it was developed through iterative refinement with human annotators (Section 4.3, where initial agreement of α = 0.273 improved to α = 0.593 after training materials clarified the boundary between non-critical variations and critical failures).

Significance beyond performance: This reframing has implications beyond How2Everything. Any domain where "correctness" cannot be verified programmatically but can be assessed by identifying specific failure modes — legal document generation, medical advice summarization, technical documentation — could adopt a similar failure-detection evaluation protocol. The key insight is that detecting errors is often easier and more reliable than rating quality, and that difficulty can be exploited to build scalable evaluation systems. The paper provides a worked example of this principle, complete with the annotation codebook, inter-annotator agreement analysis, and distillation methodology needed to operationalize it.

What makes this fundamental rather than incremental: The paper is not proposing a slightly better prompting strategy for LLM judges. It is proposing a different category of evaluation protocol — one optimized for reliability and scalability in domains without ground-truth verification — and demonstrating that this different category enables a development loop (evaluate → distill → train → re-evaluate) that would be impractical under quality-rating approaches. This is a conceptual contribution to evaluation methodology, not just to procedural generation.

---

Innovation 2: The Distilled Judge as a Reproducibility and Scalability Linchpin

A common pattern in LLM evaluation research is to validate an LLM-as-a-judge protocol against human annotations, report strong agreement, and then use the proprietary API-based judge for all subsequent evaluations. The paper's second distinctive contribution is recognizing that this pattern is insufficient for a benchmark intended to support long-term research — and that distillation into an open, fixed-weight model is not a nice-to-have but a requirement for the framework to function as a closed-loop development tool.

The argument has two parts. First, reproducibility: proprietary API-based judges are not reproducible. Model versions change (GPT-4.1 replaced GPT-4; GPT-5 replaced GPT-4.1), API behavior may shift silently, and results cannot be independently verified without access to the same API endpoint. For a benchmark that claims to measure a core LLM capability and support model comparison over time, this is fatal — a score of 48.62 on How2Bench in 2026 means something different if the judge model changed. How2Judge, as an open 8B model with fixed weights, makes the benchmark fully reproducible: any researcher can exactly replicate the paper's evaluation.

Second, scalability for training: the paper's ambition is not just to evaluate models but to improve them using the same signal. RL training requires thousands of judge calls per run (the paper uses approximately 8,000: 1,000 GRPO steps × group size 8). At GPT-5 API prices (~$15 for 7,000 evaluations, or roughly$0.002 per call), the judge cost alone is approximately $17 per training run — manageable, but the bigger constraint is latency. API calls would bottleneck the training loop, making online RL impractical. Local How2Judge inference removes both barriers, enabling the judge to serve as a fast, free reward signal.

What makes this distinctive is not the distillation itself — knowledge distillation from large to small models is a standard technique — but rather the system-level argument that distillation is what closes the loop. Without How2Judge, How2Everything would be an evaluation framework (mine data, validate a judge protocol, benchmark models) but not a training framework (use the judge as a reward). The distillation step is what transforms the system from measurement-only to measurement-plus-improvement. The paper's results in Table 2 — where three models gain >10 points on How2Bench after RL with How2Judge as reward — demonstrate the practical significance of this transformation.

Comparison to prior work: Prior benchmarks that use LLM-as-a-judge (e.g., AlpacaEval, Dubois et al., 2025; MT-Bench, Zheng et al., 2023) typically rely on proprietary API-based judges for the reference evaluation and do not distill them into open models for reproducibility. This is acceptable for leaderboard-style comparisons where the cost of evaluation is relatively small (a few thousand examples, evaluated once per model). But it breaks down when evaluation must be repeated many times (e.g., during RL training) or when exact reproducibility is required. The paper's decision to invest in distillation — collecting 73K GPT-5 annotations at nontrivial cost, fine-tuning an 8B model, and validating against the original human-annotated set — reflects a deliberate prioritization of long-term utility over short-term convenience. The 2.5-point drop in human agreement from GPT-5 (83.0%) to How2Judge (80.5%) is accepted as an explicit tradeoff: slightly reduced accuracy in exchange for free, reproducible, training-compatible evaluation.

The "distilled judge" as a design pattern: The paper implicitly proposes a design pattern for evaluation in domains without ground-truth verification: (1) define a targeted evaluation protocol (failure detection, not quality rating), (2) validate it against human annotations using a frontier LLM, (3) distill the frontier judge into a small open model for reproducibility and cost, and (4) use the distilled judge for both evaluation and training. The paper does not abstract this into a named pattern, but the architecture is generalizable — any domain with reference-based evaluation and no executable ground truth could follow the same template.

Significance beyond performance: The distilled judge makes How2Everything self-contained. The benchmark does not depend on access to a particular proprietary API, does not become obsolete when API pricing or model versions change, and does not require researchers to pay for evaluation. This aligns with broader trends toward open, reproducible benchmarking in NLP (e.g., the OLMo and DCLM projects that How2Everything builds on) and lowers the barrier to entry for researchers who want to study procedural generation without API access or budget.

---

Innovation 3: Demonstrating That RL Gains on Procedure Generation Are Not Format Compliance or Memorization

A persistent concern when training language models on benchmark-derived data is that improvements may be superficial: the model learns the format, style, or surface patterns of the benchmark rather than the underlying capability, or it memorizes training examples that closely resemble test examples. The paper's third distinctive contribution is a systematic, multi-pronged diagnostic that rules out both confounds for How2Bench RL gains — and in doing so, provides a template for how to stress-test training improvements on any benchmark.

Diagnostic 1: Pretraining maturity axis (Figure 6). The paper holds the post-training recipe (SFT + GRPO) constant and varies the pretraining checkpoint from which training begins, using Olmo 3 7B checkpoints at 10%, 25%, 50%, 75%, and 100% of pretraining. If RL gains were primarily driven by learning an implicit task format (e.g., "output numbered steps, one sentence each, matching the reference step count"), they should be similarly recoverable from any checkpoint — format is a surface behavior that even a partially-trained model can learn. Instead, the paper finds that SFT gains are approximately constant across checkpoints (3.4–5.4 points) while RL gains grow substantially with pretraining FLOPs (3.6 points at 10% pretraining to 20.3 points at 100% pretraining). This pattern matches the signature of genuine capability amplification: RL extracts and refines procedural knowledge that was only partially present in early checkpoints but crystallized during pretraining. The SFT gains, by contrast, look like format-level improvements — teaching the model how to output procedures in the expected structure, which a partially-trained model can learn roughly as well as a fully-trained one.

This finding also connects to broader narratives about RL vs. SFT in post-training (Ouyang et al., 2022; Zhao et al., 2025). SFT is often characterized as shaping surface-level behavior, while RL amplifies pretrained capabilities. The paper's results — SFT gains constant across checkpoints, RL gains growing with pretraining — provide clean empirical support for this characterization in the procedural domain, and the diagnostic methodology (varying pretraining maturity while holding the post-training recipe fixed) is a generalizable tool for distinguishing surface learning from capability amplification.

Diagnostic 2: Topic-restricted training (Section 7.1). The paper runs RL on Qwen 3 8B using training data from only two topics — Science/Math & Technology (a broadly dispersed topic in embedding space) and Food & Dining (a specialized cluster) — and measures transfer to all other topics. If RL gains were primarily format-level, training on a narrow topic should transfer broadly (format is topic-independent). The paper finds that science-only RL generalizes strongly (+9.4 overall vs. +10.1 from training on all topics), but dining-only RL transfers more weakly (+5.6 overall). The strong generalization from the science topic is consistent with format learning, but the differential transfer — science transfers better than dining — suggests that content coverage matters. Science/Math & Technology procedures span diverse subdomains, and exposure to this diversity during RL produces skills that transfer to other topics; Food & Dining procedures are more homogeneous, and skills learned from them transfer less broadly. If gains were purely format-level, this asymmetry should not exist.

Diagnostic 3: Memorization sensitivity (Table 3). The paper constructs a controlled experiment where source documents appear 0, 1, 3, 6, or 10 times during midtraining, and then measures both (a) how well the model fits the source documents (perplexity) and (b) how well it performs on procedure instances extracted from those documents (How2Score). The results are striking: as document exposure increases from 0 to 10, perplexity on the source documents drops dramatically (Olmo 3 7B: 10.4 → 1.4; Olmo 3 32B: 8.0 → 1.2), indicating that the model is memorizing the source content. But How2Score improves only modestly and non-monotonically (peaking at +3.3 for 7B, +6.1 for 32B, then declining). This dissociation — near-perfect memorization yields only marginal procedural improvement — suggests that knowing the source document verbatim is not sufficient for generating valid procedures from it, and that How2Score is measuring a capability (procedural generation) that is not simply reducible to source document recall.

What makes this diagnostic approach distinctive: Prior work has acknowledged concerns about benchmark contamination and superficial learning (e.g., Wang et al., 2025), but the paper provides a concrete, multi-pronged methodology for testing specific confounds: format compliance (Diagnostics 1 and 2) and memorization (Diagnostic 3). The diagnostics are not merely correlational (e.g., "we checked and there was no overlap") but experimental — they manipulate independent variables (pretraining maturity, topic coverage, document exposure frequency) and measure the effect on the dependent variable (How2Score gains). This experimental approach provides stronger causal evidence than overlap-based contamination analyses.

Significance beyond performance: The paper's RL gains (+10 points on How2Bench, Table 2) would be interesting but potentially hollow if they could be explained away as format learning or memorization. The diagnostics transform an empirical result into a finding about the nature of the capability being improved: procedural generation is a learnable skill that builds on pretrained knowledge and transfers across topics, not a surface pattern that can be quickly acquired or a recall task that benefits mainly from memorization. This distinction matters for interpreting the RL results and for guiding future work — it suggests that further improvements will require better pretrained procedural knowledge (not just better RL), and that training data diversity (topic coverage) matters for generalization.

---

Innovation 4: Procedural Generation as a Core Capability — Not a Narrow Application — Evidenced by Scaling and Transfer

A subtle but important conceptual move in the paper is the implicit claim that goal-conditioned procedure generation is not a narrow domain-specific task but a core, general LLM capability — analogous to mathematical reasoning or code generation — that can be studied, measured, and improved using the same methodological tools applied to those capabilities. The paper does not state this claim explicitly, but the architecture of the work — a dedicated benchmark, scaling law analysis across training stages, RL improvement experiments, out-of-domain transfer evaluation, and confound diagnostics — is the architecture used to study core capabilities, not narrow applications.

Evidence for the "core capability" claim:

• Scaling behavior across model sizes and training stages (Figure 1b): How2Bench shows smooth, monotonic improvement across five Olmo training runs spanning 1B to 32B parameters, and across pretraining, midtraining, and post-training stages. Non-trivial performance emerges by about 5% into pretraining for a 1B model and continues improving through the final post-trained checkpoint. This pattern — early emergence, steady improvement, no saturation — is characteristic of benchmarks that measure general capabilities (Heineman et al., 2025; Xu et al., 2025), not narrow tasks that saturate quickly or require specific fine-tuning.

• Out-of-domain transfer from RL (Table 2): RL on procedure generation does not cause systematic regressions on 12 standard benchmarks spanning knowledge (MMLU-Pro), reasoning (GPQA, ZebraLogic, Omega), math (GSM8K, Minerva, AIME), code (HumanEval+, LiveCodeBench, MBPP+), and chat (AlpacaEval). In some cases, out-of-domain performance even improves (e.g., AIME25 gains of +3.4, +2.2, and +5.4 across the three models). This pattern — in-domain improvement without out-of-domain degradation, with occasional positive transfer — is what you would expect if the trained capability (procedural generation) shares underlying competencies with other tasks, rather than being a narrow, isolated skill that competes with general capabilities for model capacity.

• Connection to reasoning: The paper notes (Section 1) that "exposure to procedural content at all stages of model training has been shown to improve downstream tasks that rely on reasoning and planning" (citing Ruis et al., 2025; Zhang et al., 2020; Brahman et al., 2024). This positions procedural generation not as an end in itself but as a capability that underlies or transfers to broader reasoning. The RL transfer results provide suggestive evidence for this connection: improving procedure generation does not harm (and sometimes helps) math reasoning, logical reasoning, and code generation — tasks that share structural similarities with procedural planning (sequential step execution, constraint satisfaction, goal-directed action selection).

Implications of the "core capability" claim: If procedural generation is a core capability rather than a narrow application, then (a) benchmarks like How2Bench should be included in standard evaluation suites alongside math and code benchmarks, (b) training on procedural data may benefit downstream reasoning tasks (as the transfer results hint), and (c) studying how procedural capabilities develop during pretraining may provide insights into how models acquire planning and sequential reasoning more generally. The paper provides the infrastructure (How2Bench, How2Judge, the scaling analysis) to support this broader research agenda, even though the paper itself focuses on the domain of "how-to" procedures.

Comparison to prior work: Prior procedural datasets (WikiHow-based, cooking-specific) treated procedure generation as a relatively narrow NLP task — script learning, step ordering, goal-oriented generation — with domain-specific evaluation. The paper's topic-balanced, web-mined approach, combined with the emphasis on scaling behavior, transfer, and confound analysis, implicitly elevates the task to a different status: a window into a fundamental capability that matters across applications, not just for how-to chatbots. This reframing is itself a contribution: it argues, through the architecture of the work, that procedure generation deserves the same level of systematic study as mathematical reasoning or code generation.

What makes this distinctive: The paper does not claim "procedural generation is a core capability" as a theoretical contribution — it demonstrates it through the way the framework is designed and evaluated. The inclusion of scaling analysis (Figure 1b), out-of-domain transfer (Table 2), and confound diagnostics (Section 7) is not incidental; it is the methodological apparatus needed to establish that an evaluation measures something fundamental rather than something narrow. By building this apparatus for procedural generation, the paper creates the conditions for the field to treat it as a first-class capability — not just an application area.

---

Innovation 5: How2Mine as a Generalizable Pattern for Extracting Task-Specific Data from Pretraining Corpora

The paper's fifth contribution is methodological and architectural: How2Mine demonstrates a generalizable pipeline for converting raw pretraining web data into structured task-specific evaluation and training data, using LLMs to perform the extraction and filtering that would traditionally require manual curation. The specific pipeline (five stages: extraction, heuristic filtering, LLM filtering, post-processing, final validation) is tailored to procedures, but the pattern — mine, filter, structure, validate — applies to any domain where reference examples exist naturally in web corpora but are not in a structured format.

What makes this pattern generalizable: The key insight is that LLMs can serve as data refineries — transforming raw, unstructured text into structured, task-appropriate formats — at a cost that is low enough to process millions of documents. The How2Mine pipeline processes 980K documents for $5,717 using the OpenAI batch API (Section 3.2), yielding 351K structured procedure instances. At approximately$0.006 per input document and a ~36% yield, the cost per usable instance is about $0.016 — dramatically cheaper than human curation (which, at the paper's annotation rate of ~$28/hour, would cost hundreds of thousands of dollars for the same volume).

The pattern can be abstracted as follows:

1. Source selection: Use metadata (topic classifiers, format classifiers) to identify documents likely to contain the target content type. For procedures, this was WebOrganizer's Tutorial & How-to Guide classification.
2. LLM-based extraction: Use a frontier model to extract structured records from raw documents, with explicit filtering for documents that do not contain the target content type.
3. Heuristic filtering: Apply cheap, rule-based checks to remove obviously malformed extractions.
4. LLM-based quality filtering: Use a frontier model to identify and remove examples with specific disqualifying properties (domain-inappropriate content, non-sequential structures, entity dependence).
5. Post-processing: Standardize format, add metadata, and validate quality.

This pattern is applicable to any domain where web documents contain exemplars of the target task in a recognizable but unstructured form: extracting step-by-step debugging procedures from programming forums, extracting experimental protocols from scientific papers, extracting workflow descriptions from business documentation. The specific filters and extraction prompts would differ, but the pipeline architecture transfers.

Comparison to prior work: Prior dataset construction in NLP has largely followed two paradigms: (1) manual curation (hire annotators to create examples from scratch or carefully select from a source) — expensive, slow, and hard to scale; or (2) scraping from structured sources (WikiHow, Wikipedia, Reddit) — cheap but limited to sources that are already structured. How2Mine occupies a middle ground: it mines from unstructured web sources (which are virtually unlimited in quantity and diversity) but uses LLMs to impose structure during extraction, achieving coverage that manual curation cannot match and diversity that single-source scraping cannot match. The topic-stratified sampling across 189K unique domains (Appendix B.1) demonstrates the breadth that this approach enables — it would be impossible to manually curate a dataset with this level of domain diversity.

Evidence of quality: The 96.6% reference validity rate (GPT-4.1 acceptance of How2Bench references as reasonable procedures, Section 4.1) provides evidence that the extraction pipeline produces usable reference procedures despite operating on noisy web data. This is not perfect — 3.4% of references may contain errors — but it is high enough to support large-scale evaluation and training, and the paper acknowledges the residual noise as an acceptable tradeoff for coverage.

Significance beyond procedural generation: How2Mine is a worked example of a trend that is likely to accelerate: using LLMs to create structured training and evaluation data from the same web corpora that LLMs are pretrained on. This creates a direct feedback loop between pretraining data and task-specific improvement — the same web documents that teach models about the world during pretraining can be mined to create targeted benchmarks and training sets for specific capabilities. The paper's framing of How2Everything as a "closed loop of capability evaluation and improvement at scale" (end of Section 1) captures this broader vision, with How2Mine as the data engine that powers the loop.

What makes this fundamental rather than incremental: Single-source scraping was the default approach for procedural datasets (WikiHow, cooking sites). How2Mine demonstrates that LLM-powered extraction from arbitrary web corpora can produce data that is both broader in coverage and sufficient in quality for benchmarking and training. This is a shift in how task-specific datasets are constructed — not by finding a clean source, but by building a pipeline that creates clean data from unclean sources — and it enables the scale and diversity that make the rest of the How2Everything framework viable.

5. Experimental Analysis

Evaluation Methodology

• Dataset. How2Bench, a 7,000-example evaluation set created by sampling 500 procedures from each of 14 topics in the How2Mine output. The underlying source is the DCLM web corpus (Li et al., 2025), from which 980K topic-stratified tutorial documents were processed through the How2Mine pipeline to yield 351K total procedure instances; the 7,000 reserved for How2Bench are held out from all training and RL data (Section 3.2, Section 5).

• Base model(s). Evaluations span open-weight models (Qwen 3 4B and 8B Instruct, Qwen 2.5 7B Instruct, OLMo 3 7B Instruct and Think, plus corresponding base checkpoints), intermediate pretraining/midtraining/post-training checkpoints from five OLMo runs (OLMo 2: 1B/7B/32B; OLMo 3: 7B/32B) (OLMo et al., 2025; Olmo et al., 2025), and closed-source frontier models (GPT-5, GPT-4.1, Gemini 2.5 Pro, Claude 4.5 Opus, Claude 4.5 Sonnet). The OLMo checkpoints are specifically chosen because they span a wide range of training FLOPs (~10²¹ to full training) and stages, enabling scaling analysis; the closed-source models establish the performance ceiling (Figure 4, Figure 1b).

• Metrics. The primary metric is How2Score, computed as the fraction of the 7,000 How2Bench examples for which How2Judge detects no critical failure in the generated procedure. Formally, Score(D) = (1/|D|) × Σ_{x∈D} 𝕀[J(g, R, S, Ŝ) = no_failure], where J(·) is How2Judge's binary judgment (Section 4.2). Secondary metrics include average generated token length (to detect verbosity bias; Figure 4) and step-count matching rates (to measure format compliance; Appendix). For the cross-judge robustness analysis (Figure 5), the same generations are rescored by GPT-5, Gemini 2.5 Pro, and Claude 4.5 Opus to check ranking consistency. For the perplexity analysis (Section 7.2, Table 3), conditional perplexity on the reference steps is used as a comparison metric.

• Baselines. The paper does not define formal "baselines" in the traditional sense (since How2Bench is a new benchmark), but several comparison points serve as baselines: (a) base model checkpoints (no post-training) vs. instruct/think checkpoints (post-trained) — this comparison isolates the effect of instruction tuning and RLHF on procedural generation (Figure 4); (b) pretraining maturity axis — applying identical SFT+RL recipes to checkpoints at 10%, 25%, 50%, 75%, and 100% of pretraining to measure how much improvement comes from format learning vs. capability amplification (Figure 6); (c) conditional perplexity on reference steps — testing whether How2Score is reducible to likelihood of the reference text (Section 5.2, Appendix); (d) external judges — GPT-5 and Gemini 2.5 Pro are used to verify that RL gains persist under judges not used during training (Appendix).

• Generation budget / compute accounting. Generation cost is measured implicitly through model parameter count and training stage rather than through a fixed per-example compute budget. Each model generates exactly n steps (where n is the reference step count), with step length controlled by the prompt (each step must be "a single, concise sentence containing one main action"; Section 5.1). This means all models expend approximately equal inference compute per example — there is no best-of-N sampling, beam search, or iterative revision. The RL training budget is quantified as 1,000 GRPO optimizer steps with a group size of 8 (approximately 8,000 generations scored by How2Judge per training run; Section 6.1). The FLOPs analysis uses training FLOPs as the x-axis for scaling curves (Figure 1b), with the 1B model at ~5% pretraining corresponding to approximately 10²¹ FLOPs.

• Cross-validation / statistical protocol. No cross-validation is used for the main evaluation — How2Bench is a fixed 7,000-example test set, and How2Judge is a fixed distilled model, so scores are deterministic given the same model and generation parameters. For the human annotation validation (Section 4.3), three annotators label 200 examples, and Krippendorff's α is reported for inter-rater reliability (α = 0.593 for binary aggregation). For the RL training, train-test separation is enforced via embedding-based deduplication between the 100K training examples (sampled from How2Train) and the 7K How2Bench examples to prevent leakage (Section 6.1, Appendix). For the memorization analysis (Section 7.2), an evaluation set of 13,500 examples is created specifically from the midtraining documents, balanced across the five occurrence groups (0, 1, 3, 6, 10 exposures).

Main Quantitative Results

How2Bench Performance Across Model Scales and Families

The headline result from How2Bench evaluation is that performance scales smoothly with both model size and training stage, and there is a substantial gap between open and closed models (Figure 4, Figure 1b).

Frontier vs. open models (Figure 4). GPT-5 leads at approximately 65% (exact value requires reading the figure; the paper does not quote it numerically in text). The best open instruct model, Qwen 3 8B Instruct, achieves 38.52% — a gap of roughly 26 points to the frontier. Other frontier models (GPT-4.1, Gemini 2.5 Pro, Claude 4.5 Opus, Claude 4.5 Sonnet) cluster in the ~50–65% range. Among open instruct models, Qwen 3 8B Instruct (38.52%) leads Qwen 3 4B Instruct (30.29%) and OLMo 3 7B Think (27.30%), with Qwen 2.5 7B Instruct and OLMo 3 7B Instruct falling in between. The base checkpoint for Qwen 3 8B scores substantially lower (estimated ~10–15% from Figure 4), demonstrating the contribution of post-training to procedural generation.

Scaling across training stages (Figure 1b). Across five OLMo training runs (OLMo 2 1B, 7B, 32B; OLMo 3 7B, 32B) with checkpoints sampled throughout pretraining, midtraining, and post-training, How2Score exhibits smooth monotonic improvement. Non-trivial performance emerges by approximately 5% into pretraining for the 1B model (around 10²¹ FLOPs), after which scores continue to rise through the end of post-training. The 32B runs outperform the 7B runs, which outperform the 1B run at all stages, establishing clear model-size scaling. This pattern — early emergence, steady improvement, no saturation — is significant because many benchmarks either saturate early (models quickly reach ceiling) or target only frontier models (near-zero performance at smaller scales). How2Bench's ability to discriminate across the full training pipeline makes it suitable for performance forecasting and scaling law analysis (Heineman et al., 2025; Xu et al., 2025).

Base vs. instruct gap (Figure 4). Across all model families where both base and instruct checkpoints are available (Qwen 3 4B and 8B, Qwen 2.5 7B, OLMo 3 7B), the instruct variant substantially outperforms the base variant. For example, Qwen 3 8B Instruct achieves 38.52% while its base counterpart is substantially lower (exact base scores require reading Figure 4; the gap appears to be >15 points). This indicates that post-training (instruction tuning, RLHF) significantly improves procedural generation — consistent with the paper's claim in Section 1 that "exposure to procedural content at all stages of model training has been shown to improve downstream tasks."

Average generation length (Figure 4, right bars). The paper reports average generated tokens alongside How2Score. The average reference length is 97.44 tokens. Most models generate procedures within a comparable range, though some variation exists. This metric serves as a control: if a model achieved high How2Score primarily through verbose generation (exploiting verbosity bias in the judge), its average token count would be anomalously high. The paper does not report specific numbers for each model's length, but the Figure 4 visualization suggests that length variation is modest and does not explain the performance differences.

Cross-Judge Robustness (Figure 5)

A critical validation of the How2Score protocol is whether model rankings are invariant to the choice of judge — particularly important given concerns about self-preference bias in LLM-as-a-judge settings (Zheng et al., 2023). The paper rescored the same set of generated procedures from closed-source models (GPT, Gemini, Claude families) using four different judges: How2Judge, GPT-5, Gemini 2.5 Pro, and Claude 4.5 Opus.

The key finding: absolute score values vary across judges, but the relative model ranking is unchanged (Figure 5). For example, GPT-5's outputs might receive a slightly higher score from GPT-5 than from Claude (which would be consistent with self-preference), but the ordering of which models perform best to worst is preserved. Specifically, all four judges rank GPT-5 as the best generator, followed by the other frontier models in a consistent order. This invariance across judge families suggests that How2Bench produces reliable model comparisons that are not artifacts of the specific judge used. The paper does not report a dedicated self-preference analysis for How2Judge alone (e.g., comparing How2Judge scores of Qwen-generated procedures vs. non-Qwen procedures against a human baseline), which is a minor gap in the analysis.

Scaling Behavior Detail: Formatting Emerges Early, Validity Continues Improving

The paper tracks simple formatting proxies across OLMo checkpoints: step-count mismatch (does the generated number of steps match the reference?), duplicate-step frequency, and n-gram repetition. Across all five OLMo runs, these formatting errors drop sharply during early pretraining and then plateau — surface-level formatting stabilizes early, while How2Score continues to improve throughout training (Section 5.2, detailed in Appendix). This decoupling is important because it suggests that the continued How2Score gains observed later in training are not driven by models simply learning to output well-formatted numbered lists. The format is acquired quickly; the procedural validity — actually knowing which steps to include and in what order — takes much longer to develop. This pattern resembles an emergence-like phenomenon: surface behaviors stabilize early, while the underlying capability keeps improving.

How2Score vs. Perplexity: Not Simply Reducible to Reference Likelihood

A natural question is whether How2Score is simply measuring how well the model can reproduce the reference text — i.e., is procedural generation reducible to conditional generation of the reference steps? To test this, the paper compares checkpoint ordering under How2Score to checkpoint ordering under conditional perplexity on the reference steps across the five OLMo runs (Section 5.2, Appendix).

The Spearman rank correlation between these two orderings varies dramatically across runs:

• OLMo 2 1B: ρ = 0.967 (nearly identical ordering)
• OLMo 2 7B: ρ = 0.667
• OLMo 2 32B: ρ = 0.233 (weak correlation)
• OLMo 3 7B: ρ = 0.867
• OLMo 3 32B: ρ = 0.483

The fact that these correlations range from near-perfect (1B) to near-chance (0.23 for 32B) indicates that How2Score is not simply measuring conditional likelihood of the reference procedure. If it were, the correlation would be consistently high across all runs. The variation suggests that for smaller models, improvements in How2Score and perplexity track together (both improve with training), but for larger models, the two metrics diverge — the model can become much better at generating valid procedures without becoming proportionally better at predicting the exact reference text. This is consistent with procedural validity being a distinct capability from text reproduction.

Topic as a Difficulty Axis: Step Count Is the Dominant Predictor

The paper examines simple instance properties that correlate with How2Score across models (Section 5.2, Appendix). The finding: reference step count |S| is the dominant predictor of difficulty — procedures requiring more steps are consistently harder across all models. This makes |S| a simple, monotonic difficulty knob for slicing How2Bench results. Resource count shows a weaker correlation. Topic itself matters (some topics are inherently harder than others), but step count is the most consistent predictor within and across topics. This is a practically useful finding: it means that when interpreting aggregate How2Bench scores, one can control for difficulty by stratifying by step count, and the benchmark's balanced topic sampling (500 per topic) ensures that aggregate scores are not dominated by any single topic's difficulty level.

Qualitative Failure Analysis

The paper performs a small-scale qualitative analysis over model generations to categorize the types of non-formatting failures that occur (Section 5.2, Appendix). The main failure types identified:

• Critical omissions of required actions: the generated procedure is missing a step that is necessary for achieving the goal (the most common failure type, as exemplified by the missing 30-day waiting period in Table 1).
• Missing parameters: times, quantities, temperatures, or other specific values are omitted, making steps non-executable — for example, specifying "cook on low heat" without giving a duration when the reference specifies 2-3 hours.
• Wrong values for critical parameters: the parameter is present but incorrect — e.g., "cook on high for 30 minutes" when the reference specifies 15 minutes, which would burn the dish.
• Unsafe or invalid actions: steps that would be physically impossible, dangerous, or logically contradictory.
• Internal contradictions: one step contradicts another, making the procedure impossible to execute as written (the paper gives the example of "cut into 5 equal pieces" followed by "place the largest piece on top" — if pieces are equal, there is no largest).

Additionally, the paper notes occasional refusals, primarily in frontier models (e.g., refusing to generate a procedure involving legal or health advice). These are counted as failures under How2Score (since no valid procedure was generated), which may slightly penalize models with conservative safety policies.

RL Training Results (Table 2)

The core training result: RL using How2Score as a reward (computed by How2Judge) improves How2Bench performance by >10 points across all three trained models, without systematic regressions on 12 out-of-domain benchmarks (Table 2).

The specific gains at optimizer step 1,000:

• Qwen 3 4B Instruct: 30.29 → 43.52 (+13.23)
• Qwen 3 8B Instruct: 38.52 → 48.62 (+10.10)
• OLMo 3 7B Think: 27.30 → 37.89 (+10.59)

The out-of-domain benchmark suite spans five categories: knowledge (MMLU-Pro, GPQA), reasoning (ZebraLogic, Omega), math (GSM8K, Minerva, AIME24, AIME25), code (HumanEval+, LiveCodeBench, MBPP+), and chat (AlpacaEval). The changes are mixed but generally modest, with the mean out-of-domain change (Δ̄_OOD) being +0.52 for Qwen 3 4B, +0.06 for Qwen 3 8B, and +1.05 for OLMo 3 7B Think — essentially flat on average. Individual benchmarks show some variation:

• AIME25 improves across all three models (+3.44, +2.19, +5.42), suggesting positive transfer from procedural generation to difficult math reasoning.
• Some math and code benchmarks show small declines (e.g., LiveCodeBench: -0.22, -0.21, -2.45; ZebraLogic: -1.2, +0.5, -2.3), but these are within typical variance for these benchmarks and are not systematic.
• AlpacaEval (chat quality) improves modestly (+2.95, +0.32, +1.44).

The key interpretation: no benchmark shows a large, consistent negative effect across all three models. The pattern is one of targeted improvement on procedure generation with minimal collateral impact — exactly what you would want from a post-training signal added to an existing pipeline.

RL Gains Under External Judges

The paper verifies that RL gains are not specific to How2Judge (which was used to compute the reward during training) by re-evaluating the same RL-trained model generations with GPT-5 and Gemini 2.5 Pro. The gains persist under both external judges (Appendix, Section 6.2). This rules out the concern that the model learned to exploit idiosyncrasies of How2Judge rather than genuinely improving procedural validity — if the improvement were judge-specific, it would not transfer to different judges.

SFT vs. RL Contribution (Figure 6)

When applying the same post-training recipe (SFT followed by GRPO) starting from different OLMo 3 7B pretraining checkpoints (10% through 100% of pretraining), the paper finds:

• SFT gains are approximately constant across checkpoints, ranging from +3.39 to +5.36 points (Figure 6). This is consistent with SFT primarily teaching the output format and task structure — surface-level behaviors that even a partially-trained model can acquire.
• RL gains increase substantially with pretraining FLOPs, from +3.56 at the 10% checkpoint to +20.33 at the 100% checkpoint. This is consistent with RL amplifying pretrained procedural knowledge that is only partially present in early checkpoints.
• At the final (100%) checkpoint, SFT contributes +5.36 and RL contributes +20.33, for a total post-training gain of +25.69 — but RL accounts for ~79% of the improvement. At the 10% checkpoint, the total gain is ~+7, with SFT and RL contributing roughly equally.

This finding aligns with broader characterizations of SFT vs. RL in post-training (Ouyang et al., 2022; Zhao et al., 2025): SFT shapes surface behavior while RL amplifies pretrained capabilities. For How2Bench specifically, this means that the +10-point RL gains reported in Table 2 are not primarily about learning a task format — they represent genuine improvement in procedural validity.

Length Control During RL

Without the reference-calibrated length reward (the anti-gaming component described in Section 6.1), models inflate their output length to 1.34×–1.53× the reference and achieve large apparent How2Bench gains. With the length reward active, generation length stays close to 1.0× the reference (|gen|/|ref| ≈ 1.0), and the reported gains reflect genuine procedural improvement rather than verbosity exploitation (Section 6.2, Appendix). This control is critical because verbosity bias is a well-documented confound in LLM-as-a-judge evaluation (Dubois et al., 2025) — judges tend to rate longer, more detailed outputs as better, even if the extra length does not add correctness.

Topic-Restricted RL Transfer (Section 7.1)

To test whether RL gains depend on broad topic coverage or can be achieved by learning a generic format from a narrow topic, the paper runs RL on Qwen 3 8B using training data restricted to two topics with contrasting embedding-space characteristics: Science/Math & Technology (broadly dispersed in embedding space) and Food & Dining (a specialized cluster). Results:

• All-topics RL: +10.10 overall (the baseline from Table 2)
• Science-only RL: +9.41 overall — transfers strongly to nearly all other topics, nearly matching all-topics performance
• Dining-only RL: +5.55 overall — transfers more weakly, with notably lower gains on dissimilar topics

The strong transfer from the science topic is consistent with two (non-mutually-exclusive) explanations: (a) format learning transfers universally (science-only RL teaches the output format, which helps on all topics), and (b) the science topic's diversity provides broad procedural knowledge that transfers. The weaker transfer from the dining topic suggests that (b) is at least partially operative — if gains were purely format-level, dining-only RL should transfer just as well as science-only RL. The asymmetry (science transfers better than dining) is evidence that content coverage matters, not just format.

Ablation Studies and Robustness Checks

• LLM judge agreement with human majority (Figure 3): Five frontier LLM judges are compared against human majority labels on 200 examples. GPT-5 achieves the highest agreement at 83.0% (83.7% on has_failure, 82.4% on no_failure), falling within 1–5 points of leave-one-out human agreement (84.7–88.5%). GPT-4.1, Gemini 2.5 Pro, Claude 4.5 Opus, and Claude 4.5 Sonnet achieve 76.5–81.5%. The balanced calibration across classes (no systematic leniency or strictness bias) is notable for GPT-5.

• How2Judge distillation quality (Figure 3, Section 4.4): How2Judge achieves 90.5% agreement with its teacher (GPT-5) and 80.5% agreement with human majority on the same 200-example human-annotated set. The 2.5-point drop from GPT-5's 83.0% to How2Judge's 80.5% human agreement represents the cost of distillation, but How2Judge maintains balanced calibration (79.6% on has_failure, 81.4% on no_failure). Training uses 73K GPT-5 annotations from a diverse set of generator models (three 1B, four 7B, three 32B, four closed-source), deduplicated against the human-annotated set.

• Reference validity check (Section 4.1): GPT-4.1 judges whether each How2Bench reference procedure reasonably achieves the stated goal. Acceptance rate: 96.6%. The remaining 3.4% of references may contain errors, representing a noise floor for How2Score. This is not ablated further — the paper accepts this noise as a tradeoff for coverage.

• Binary vs. first-failure aggregation (Section 4.2): Requiring annotators to agree on the location of the first failure produces Krippendorff's α = 0.307, compared to α = 0.593 for binary aggregation (any failure detected). This is the key empirical justification for the binary scoring protocol.

• Self-preference bias check via cross-judge ranking (Figure 5): Rescoring the same generations with four different judges (How2Judge, GPT-5, Gemini 2.5 Pro, Claude 4.5 Opus) preserves the relative model ranking even though absolute score values differ. This suggests that self-preference bias, if present, does not distort benchmark comparisons.

• How2Score vs. perplexity correlation across OLMo runs (Section 5.2, Appendix): Spearman ρ between checkpoint rankings by How2Score and by conditional perplexity on reference steps ranges from 0.233 (OLMo 2 32B) to 0.967 (OLMo 2 1B). The low correlation for larger models indicates How2Score captures something beyond reference likelihood.

• SFT contribution on instruct vs. base checkpoints (Section 6.2): SFT yields small gains on base model checkpoints (+3.4 to +5.4; Figure 6) but does not improve already instruction-tuned checkpoints. The paper attributes this to objective mismatch: SFT maximizes reference likelihood, while How2Score targets failure minimization. This is a practical negative result — adding an SFT stage before RL on instruct models provides no benefit.

• RL without length reward (Section 6.2, Appendix): Without the reference-calibrated length penalty, models inflate output length to 1.34×–1.53× the reference and achieve large apparent How2Bench gains. With the penalty, generation length stays near 1.0× the reference, and the reported gains reflect genuine improvement. Length control is validated as essential for preventing verbosity exploitation.

• RL gains under external judges (Section 6.2, Appendix): RL-trained model generations re-evaluated with GPT-5 and Gemini 2.5 Pro show that How2Bench gains persist. This rules out the concern that models learn to exploit How2Judge-specific biases rather than genuinely improving.

• Pretraining maturity axis for RL vs. SFT (Figure 6): Holding the post-training recipe fixed and varying the pretraining checkpoint (10%, 25%, 50%, 75%, 100% of OLMo 3 7B pretraining), SFT gains are approximately constant (+3.4 to +5.4) while RL gains grow with pretraining FLOPs (+3.6 at 10% to +20.3 at 100%). This is the primary evidence that RL gains represent capability amplification, not format learning.

• Topic-restricted RL transfer (Section 7.1): RL on a single broadly-dispersed topic (Science/Math & Technology) transfers strongly (+9.41 overall vs. +10.10 from all-topics RL), while RL on a specialized cluster (Food & Dining) transfers more weakly (+5.55). This asymmetry suggests that content coverage and diversity matter for generalization — gains are not purely format-level.

• Memorization sensitivity via controlled midtraining exposure (Table 3): As source documents are repeated 0, 1, 3, 6, or 10 times during midtraining for OLMo 3 7B and 32B, document perplexity drops sharply (7B: 10.4 → 1.4; 32B: 8.0 → 1.2), indicating strong memorization. How2Score improves only modestly and non-monotonically (7B: 14.0 → peak 17.3 at 1×, declining to 16.5 at 10×; 32B: 33.3 → peak 39.4 at 3×, declining to 37.9 at 10×). The dissociation between near-perfect memorization and marginal procedural improvement is the core evidence that How2Score does not primarily measure source document recall.

• Early emergence of procedural formatting (Section 5.2, Appendix): Surface formatting proxies (step-count mismatch, duplicate-step frequency, n-gram repetition) drop sharply during early pretraining and plateau, while How2Score continues to improve. This decoupling suggests that later-stage How2Score gains reflect procedural validity improvement, not format correction.

• ReST^EM revision model (not in main paper, referenced in prior sections): The paper does not contain a ReST^EM experiment — this was mentioned in the prior sections example and does not apply here. The paper's RL approach uses GRPO directly, without an intermediate EM-style optimization phase.

Critical Assessment

The paper makes five central claims in its contributions (Section 1):

Claim 1: "How2Mine mines 351K procedures from 980K web pages across 14 topics and readily scales to larger corpora." The paper demonstrates the pipeline's output (351K instances) and provides topic-stratified domain distributions (Appendix B.1, showing 189K unique domains). However, the claim of "readily scales" is asserted, not demonstrated — the paper does not run the pipeline on a larger corpus (e.g., 10M documents) to show that yield remains stable, quality does not degrade, or costs scale linearly. The $5,717 cost for 980K documents is provided, which allows readers to extrapolate costs, but the scalability claim rests on the architectural design (stages are parallelizable, batch API is used) rather than empirical demonstration. The 96.6% reference validity rate (Section 4.1) provides evidence of output quality, but this is measured by GPT-4.1 — the same model family used in the pipeline — which introduces a circularity concern: if GPT-4.1 systematically misses certain error types during filtering, it might also fail to flag them during validation. An external validity check (e.g., human annotation of reference quality on a sample) would strengthen this significantly.

Claim 2: "How2Score achieves high agreement with human annotators (80.5% for How2Judge), enabling low-cost, reproducible evaluation." This is well-supported by the human annotation study (Section 4.3), but with important caveats. The 200-example human-annotated set is small and pre-screened to avoid procedures requiring specialized domain knowledge — this means the measured agreement may overestimate performance on the most technically complex procedures in How2Bench. Additionally, Krippendorff's α = 0.593 for binary aggregation is in the "moderate" range — sufficient for aggregate comparisons but not for high-stakes per-example decisions. The paper is transparent about this (Section 4.3: "we do not expect near-perfect agreement... instead, we target a metric that is stable for relative comparisons"), but readers should understand that How2Score is a proxy for procedural validity, not a ground-truth oracle. The cross-judge ranking robustness (Figure 5) is strong evidence that the metric produces stable model comparisons, which is likely its primary use case.

Claim 3: "How2Bench can meaningfully rank models across a large range of compute budgets, from 10²¹ FLOPs to frontier models." This is well-supported by Figure 1b, which shows smooth scaling across five OLMo runs spanning 1B–32B parameters and multiple training stages. The benchmark does not saturate even at GPT-5 (~65% accuracy), leaving headroom for future models. However, all scaling data comes from a single model family (OLMo). It is possible that other model families with different pretraining data mixtures or architectures would show different scaling trajectories — a Qwen scaling curve or a Llama scaling curve might look different. The paper acknowledges that all evaluations use models pretrained on web data (which is where the How2Mine procedures originate), and models trained on substantially different data distributions might perform differently. A broader scaling analysis across model families would increase confidence that How2Bench measures a general capability.

Claim 4: "RL using How2Score as a reward improves How2Bench performance by >10 points across three models without systematic regressions on standard benchmarks." This is the most strongly supported claim in the paper, with clear evidence from Table 2 and the out-of-domain evaluation suite. However, several caveats merit attention:

• Training scale is modest: 1,000 GRPO steps with a group size of 8 represents a relatively light RL phase. The paper does not explore whether continued training would yield further gains or eventually cause regressions. The out-of-domain benchmarks show no systematic degradation, but this might change with more aggressive RL.
• Out-of-domain gains are small and inconsistent: While the mean Δ̄_OOD values are near zero, individual benchmarks show variation (e.g., AIME25 improves, LiveCodeBench slightly declines). The paper interprets this as "no systematic degradation," which is fair, but the experiments do not establish that procedural RL improves out-of-domain capabilities — they establish that it does not harm them. The positive transfer to AIME25 is intriguing but not explained or systematically investigated.
• The SFT stage provides limited benefit: SFT helps base models learn the output format but does not improve instruct checkpoints. This is a practical finding (SFT can be skipped for instruct models) but also a missed opportunity — better SFT data construction (e.g., using multiple valid references per goal rather than a single reference) might mitigate the objective mismatch and yield SFT gains complementary to RL.
• RL reward is computed by the same model family as the judge distillation base: How2Judge is distilled from Qwen 3 8B. While the paper shows that RL gains persist under GPT-5 and Gemini 2.5 Pro (reducing self-preference concerns), there remains a theoretical risk that Qwen-based models are advantaged because How2Judge's training distribution (from GPT-5 annotations on diverse generator outputs) may be more calibrated for some model families than others.

Claim 5: "Improvements in procedure generation are not driven by format compliance or memorization." This is partially supported with strong evidence against memorization (Table 3 — near-perfect document memorization yields only marginal How2Score improvement) and against pure format compliance (Figure 6 — RL gains grow with pretraining maturity, while format-level SFT gains are constant). However, the evidence against format compliance is less definitive than the paper suggests:

• The "pretraining maturity axis" argument (Figure 6) shows that RL gains are larger from stronger base checkpoints, which is consistent with capability amplification. But an alternative interpretation is that stronger base models are better at exploiting RL to learn complex format patterns (e.g., domain-specific formatting conventions that vary by topic) that weaker models cannot learn. The paper's binary framing — "format learning" vs. "capability amplification" — may oversimplify; in reality, RL likely improves both format adherence and content validity simultaneously, with the balance shifting toward content as the base model improves.
• The topic-restricted RL transfer experiment (Section 7.1) provides evidence that content matters (science transfers better than dining), but the science-only RL still achieves +9.41 — nearly matching all-topics RL at +10.10. This suggests that a substantial fraction of the gain (~93%) can be achieved with training on a single topic, which is more consistent with format or general procedural skill learning than with topic-specific content acquisition. The paper acknowledges this ("RL trained on a single topic can transfer") but does not quantify what fraction of the gain is topic-general vs. topic-specific.
• The paper does not include a "format-only" baseline — e.g., training on procedures with shuffled or randomized steps (which would preserve format but destroy content validity). Such a baseline would directly measure the contribution of format learning to RL gains and is a notable absence from the experimental design.

Additional concerns and missing experiments:

• Human evaluation of RL improvements: All RL gains are measured by How2Judge, which is itself an imperfect proxy (80.5% human agreement). The paper does not conduct human evaluation on a sample of RL-trained model outputs to verify that the +10-point How2Bench improvement corresponds to human-perceived improvement. Without this, the possibility remains that RL improves How2Judge scores without improving actual procedural validity (i.e., the model learns to exploit residual judge biases that the length penalty and format checks do not fully prevent).
• No breakdown of RL gains by topic or difficulty: The paper reports aggregate How2Bench gains but does not analyze where the gains concentrate — easy topics? hard topics? short procedures? long procedures? This analysis would reveal whether RL primarily helps with certain types of procedural errors or generalizes broadly.
• Single RL algorithm: Only GRPO is tested. Comparing with simpler RL approaches (e.g., best-of-N rejection sampling, DPO on judge preferences) would help contextualize whether the specific algorithm matters or whether any reasonable optimization of How2Score yields similar gains.
• No analysis of what specific failure types RL reduces: Section 5.2 identifies failure categories (omissions, missing parameters, wrong values, contradictions), but the paper does not report which failure types RL most effectively addresses. This would provide insight into how RL improves procedural generation — does it make models more complete (fewer omissions)? more precise (better parameter values)? more consistent (fewer contradictions)?
• Cost of RL training not reported: While the paper reports the cost of How2Mine ($5,717), it does not report the compute cost of the RL training runs (GPU hours, approximate dollar cost). For practitioners considering adoption, this is an important missing detail.

Overall assessment: The paper's experimental design is thorough for establishing the core claims about benchmark construction and evaluation methodology. The scaling analysis, distillation validation, and memorization diagnostics are particularly well-executed. The RL results convincingly demonstrate that How2Score can serve as a training signal, but the nature of what is being improved (content knowledge vs. format sophistication vs. judge-specific optimization) is less precisely characterized than the paper claims. The absence of human evaluation of RL-trained outputs and the lack of ablation into what types of failures RL addresses are the most significant gaps. The framework's value as an evaluation tool is well-established; its value as a training tool is demonstrated but would benefit from deeper analysis of what is being learned.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Is Prohibitively Expensive and Unaddressed

The assumption or constraint. The paper's central difficulty-control mechanism is conditioning generation on the reference step count n = |S| and the resource list R (Section 5.1). This works for evaluation — the benchmark has pre-computed reference procedures from which n and R are extracted — but it side-steps a fundamental deployment challenge: in real-world usage, no one tells the model how many steps to produce or what resources are needed. The paper acknowledges this tension explicitly: "While conditioning generations on R and n may not reflect real-world usage, it is an evaluation control to reduce degrees of freedom and improve comparability across model outputs" (Section 5.1). This means the entire evaluation framework — How2Bench, How2Score, and the RL training — operates in an artificial setting where models receive privileged information (reference step count, resource list) that would not be available to a deployed system.

The consequence. The evaluation measures a constrained version of procedural generation — "generate exactly n steps using (possibly) these resources" — rather than the end-to-end capability implied by the paper's motivation: a user asks "how do I file a legal document?" and the model must determine both what steps are needed and how many steps to include. The paper's 8.5% ChatGPT usage statistic (Section 1) describes the unconstrained setting; the evaluation tests the constrained one. A model that excels on How2Bench might fail in deployment because it generates too few steps (oversimplifying) or too many (losing coherence), or because it cannot identify which resources are necessary without being told. The paper does not quantify how much the difficulty-control information inflates performance relative to unconstrained generation.

What evidence exists in the paper. None. The paper does not include an ablation where models generate procedures without being told n or R, does not measure how performance degrades when this information is withheld, and does not analyze whether RL-trained models learn to infer appropriate step counts or resource lists from goals alone. The cross-judge robustness (Figure 5) and scaling analyses (Figure 1b) all use the controlled inference setup. The paper provides no evidence about how How2Score would behave if generation length were unconstrained — a setting where verbosity bias becomes a much larger confound than the length-reward mitigation can address.

Mitigation status. Not addressed. The paper acknowledges the limitation in passing (Section 5.1) but treats it as an acceptable evaluation design choice rather than a problem to be solved. There is no proposal for removing the n and R conditioning in future work, no analysis of how much performance is attributable to these controls, and no path sketched for transitioning from the controlled evaluation setting to realistic deployment.

---

6.2 Hard Problems Remain Effectively Unsolved — and the Framework Provides No Path Forward

The assumption or constraint. How2Everything measures and improves procedural generation within a specific capability envelope: the model must already possess sufficient knowledge to represent valid procedures, and How2Score detects failures against reference procedures that exist in the training data distribution. The paper's topic-restricted RL transfer results (Section 7.1) show that RL generalizes within the procedural domain, but the overall framework is bounded by what the base model knows. As with all reference-based evaluation, procedures that require knowledge the model never acquired during pretraining cannot be generated correctly, and no amount of RL on existing How2Train examples will teach that knowledge.

The consequence. The hardest examples in How2Bench — those where the base model's pass rate is near zero — are unlikely to improve meaningfully through RL, because RL amplifies existing capabilities rather than teaching new domain knowledge. The paper demonstrates this indirectly through the pretraining maturity analysis (Figure 6): RL gains are minimal when starting from early pretraining checkpoints (only +3.6 points at 10% pretraining) because the base model lacks the procedural knowledge that RL would amplify. Similarly, the memorization experiment (Table 3) shows that even aggressive document memorization (perplexity dropping from 10.4 to 1.4) yields only a +3.3-point How2Score improvement for a 7B model — evidence that procedural generation requires capabilities beyond text recall. The implication is that for genuinely difficult procedures — those requiring specialized legal, medical, or technical knowledge — How2Everything's training loop hits a ceiling determined by pretraining data coverage, with no mechanism to break through it.

What evidence exists in the paper. The pretraining maturity axis (Figure 6) provides the closest evidence: RL gains grow from +3.6 at 10% pretraining to +20.3 at 100% pretraining for OLMo 3 7B. If RL were teaching new procedural knowledge (rather than amplifying existing knowledge), we would expect more uniform gains across checkpoints — a 10%-trained model should be able to learn from How2Train examples regardless of its pretraining maturity. The fact that gains grow with pretraining strongly suggests that RL depends on knowledge already present. The topic-restricted RL results (Section 7.1) provide additional evidence: training on a narrow topic (Food & Dining) transfers less effectively to other topics (+5.55 overall vs. +10.10 from all-topics RL), suggesting that RL gains are partially topic-content-dependent, not purely procedural-skill-dependent. Neither analysis directly quantifies the ceiling, but both indicate its existence.

Mitigation status. Not addressed. The paper does not analyze performance stratified by difficulty (beyond the step-count correlation noted in Section 5.2), does not identify which procedure types or topics remain stubbornly difficult after RL, and does not propose mechanisms for teaching new procedural knowledge (e.g., retrieval augmentation, iterative data mining from new web sources, training on a continuously expanding How2Mine corpus). The closed-loop framework is compelling within the capability envelope but silent on how to expand the envelope itself.

---

6.3 The Distilled Judge Inherits and Potentially Amplifies GPT-5's Systematic Biases Without Independent Calibration

The assumption or constraint. How2Judge is trained on 73K GPT-5 annotations (Section 4.4), and its quality is measured by two metrics: 90.5% agreement with GPT-5 (teacher-student fidelity) and 80.5% agreement with human majority on 200 examples (Section 4.3). The distillation validation implicitly assumes that (a) GPT-5's annotations on the 73K training examples are correct, (b) GPT-5's errors are random rather than systematic, and (c) the 200-example human-annotated set is representative of the error types and difficulty distribution in the full 7K-example How2Bench. None of these assumptions are tested.

The consequence. If GPT-5 exhibits systematic biases — for example, consistently over-penalizing certain types of deviations (treating alternative valid resource choices as critical failures), under-penalizing others (missing contradictions embedded in fluent text), or being miscalibrated for specific topics or procedure lengths — these biases will be inherited and potentially amplified by How2Judge. The 90.5% teacher-student agreement means How2Judge replicates ~9.5% GPT-5 "errors" (relative to human judgment); the 2.5-point drop from GPT-5's 83.0% to How2Judge's 80.5% human agreement means How2Judge introduces additional error. Both sources of error could be systematic — How2Judge might be consistently stricter or more lenient for certain topics, model families, or failure types. The cross-judge ranking robustness check (Figure 5) shows that model rankings are stable across judges, but this does not rule out correlated biases: if GPT-5, Gemini, and Claude all under-detect a particular failure type (e.g., unsafe actions that appear superficially plausible), How2Judge will too, and evaluations using any judge will overestimate model performance on that failure type.

A particularly concerning scenario is RL reward hacking against shared biases: if both GPT-5 (the teacher) and How2Judge (the student) share a blind spot — for example, both fail to detect missing safety constraints in procedures involving chemicals or tools — then RL training will not penalize models for omitting those constraints, and How2Bench scores will not reflect the degradation. The model learns to satisfy the judge, not the real-world task, and the evaluation cannot detect the gap because it uses the same (biased) judge.

What evidence exists in the paper. The human annotation study provides the only independent calibration point, and it has significant limitations for detecting systematic biases. The 200 examples are pre-screened to avoid specialized domain knowledge (Section 4.3: "pre-screened to avoid procedures requiring specialized domain knowledge") and represent only 2.9% of How2Bench (200/7,000). If GPT-5's biases are concentrated in technically complex procedures (legal, medical, industrial) that were excluded from human annotation, the 80.5% agreement figure overstates How2Judge's reliability on the full benchmark. There is no per-topic breakdown of judge-human agreement, no analysis of how agreement varies with procedure length or complexity, and no adversarial stress-testing of the judge (e.g., constructing edge cases where reasonable people disagree and measuring whether judges produce calibrated uncertainty rather than confident errors). The 96.6% reference validity check (Section 4.1) is performed by GPT-4.1 (same model family as the pipeline), introducing a circularity concern noted in prior sections.

Mitigation status. Partially addressed by the cross-judge robustness analysis (Figure 5), which demonstrates ranking stability but not calibration. The paper acknowledges the limitation implicitly by keeping the judge-prompt codebook in the appendix for scrutiny and by releasing How2Judge weights for reproducibility, but these are transparency measures, not mitigations — they let users detect biases (by running their own human annotation studies) but do not prevent them. The RL external-judge validation (Appendix, Section 6.2) checks whether RL gains transfer to GPT-5 and Gemini but does not check whether both the training judge and validation judges share biases. A more robust validation would require human evaluation of RL-trained outputs, which the paper does not provide.

---

6.4 Single Model Family for Scaling Analysis, Single Domain (MATH Analogue: "How-To") for All Experiments

The assumption or constraint. All scaling analyses (Figure 1b, Figure 4), RL training (Table 2), and confound diagnostics (Section 7) are conducted within a single domain — text-based how-to procedures from web documents — and primarily within a single model family (OLMo) for the scaling and memorization experiments. The RL training uses three model families (Qwen 3 4B, Qwen 3 8B, OLMo 3 7B) but all are open-weight models with similar pretraining paradigms (web-scale data, standard autoregressive LM objectives). No experiments involve models with substantially different architectures (non-transformer), pretraining objectives (masked LM, encoder-decoder), or data distributions (non-English-heavy, code-only, scientific-domain).

The consequence. The paper cannot distinguish between findings that are procedural-generation-specific (replicable across domains and model families) and findings that are domain-specific or model-family-specific. Several key claims depend on generalizability that is not tested:

• Scaling behavior (Figure 1b): The smooth scaling from 10²¹ FLOPs to GPT-5 across OLMo runs might not hold for models pretrained on substantially different data mixtures. If procedural generation capability depends on exposure to tutorial-style documents during pretraining, models pretrained primarily on code or academic text might show different scaling trajectories — or might not develop the capability at all at comparable FLOPs budgets.
• RL transfer to out-of-domain benchmarks (Table 2): The finding that procedural RL does not cause systematic regressions on math, code, and knowledge benchmarks is demonstrated for three models, but all are open-weight instruct/think models with relatively similar post-training recipes. A frontier model with extensive RLHF might respond differently — its reward model might conflict with the How2Score reward, or the model's existing capabilities might be more brittle to additional fine-tuning.
• Memorization analysis (Table 3): The midtraining experiment uses OLMo 3 7B and 32B. The finding that document memorization yields limited How2Score improvement might not hold for models with different architectural inductive biases (e.g., models with different context utilization patterns might rely more or less on verbatim recall during generation).
• Topic-restricted RL transfer (Section 7.1): The finding that science-only RL transfers broadly while dining-only RL transfers narrowly might depend on the specific Science/Math & Technology and Food & Dining clusters in the DCLM corpus. Different corpora or topic taxonomies might yield different transfer patterns.

What evidence exists in the paper. The paper evaluates a diverse set of models on How2Bench (Figure 4 includes GPT-5, GPT-4.1, Gemini 2.5 Pro, Claude 4.5 Opus/Sonnet, Qwen models, OLMo models), establishing that the benchmark discriminates across model families. However, this establishes cross-sectional validity (the benchmark ranks models meaningfully) without establishing that the dynamics (scaling behavior, RL improvements, memorization resistance) generalize. The OLMo scaling analysis establishes within-family dynamics; the cross-family How2Bench scores establish between-family discriminability. These are different questions, and answering the second does not answer the first.

Mitigation status. Not addressed. The paper does not claim generalizability beyond the studied models and domain, but the framing — procedural generation as a "core capability" (Innovation 4, Section 4 of this analysis), How2Everything as a "worked example of how pretraining web data can support a closed loop" (Section 1) — implies broader applicability that the experiments do not substantiate. The paper does not suggest specific future work on cross-domain or cross-architecture validation, treating the demonstrated domain and model range as sufficient for the contributions claimed.

---

6.5 RL Training Runs Are Short, and the Relationship Between Training Budget and Performance Is Unexplored

The assumption or constraint. All RL experiments use 1,000 GRPO optimizer steps with a group size of 8 (Section 6.1), corresponding to approximately 8,000 judge-scored generations per training run. This is a relatively light RL phase — the paper does not explore whether continued training (5,000 steps? 10,000?) would yield further How2Bench gains, plateau, or eventually cause regressions. The training budget was chosen as a fixed hyperparameter rather than through systematic exploration.

The consequence. Several important practical questions are unanswered:

• What is the performance ceiling for RL on this task? The +10-point gains in Table 2 might be far from saturation — a longer training run might yield substantially larger improvements. Or the gains might saturate quickly, meaning the 1,000-step budget is already near-optimal. Without a training-budget scaling curve, practitioners cannot estimate the ROI of additional RL compute.
• Do out-of-domain regressions emerge with more aggressive RL? Table 2's Δ̄_OOD values are near zero, but this might hold only for moderate RL budgets. More aggressive optimization of How2Score might cause the model to overfit to procedural generation at the expense of general capabilities. This is a known pattern in RL fine-tuning (over-optimization of the reward model), and the paper's 1,000-step budget might be below the threshold where it becomes visible.
• Is GRPO specifically important, or would simpler methods work? The paper tests only GRPO with three reward components. Simpler approaches — best-of-N rejection sampling from How2Score, Direct Preference Optimization (DPO) using How2Judge comparisons between good and bad generations, or even continued SFT on high-scoring examples — might achieve comparable gains with lower complexity. Without baselines, the contribution of the specific RL algorithm is unclear.
• How does the RL training curve interact with model size? The three trained models (4B, 7B, 8B) show similar relative gains (+10.1 to +13.2), but this might not hold for smaller models (where RL might saturate earlier) or larger models (where RL might continue improving for longer). The paper's own scaling analysis (Figure 1b) suggests that procedural capability depends on model scale; whether RL efficiency also depends on scale is unexplored.

What evidence exists in the paper. Only the single 1,000-step data point. Figure 6 shows the pretraining maturity axis (varying the starting checkpoint) but holds the RL budget constant — it does not show RL training curves (How2Bench score vs. optimizer step) for any model. The paper does not report intermediate checkpoints during RL training, so the reader cannot assess whether performance was still improving at step 1,000 or had plateaued. There is no ablation of the group size (what if group size = 4 or 16?) or the number of GRPO steps. The RL algorithm choice is justified by reference to prior work (DeepSeekMath, Shao et al., 2024) but not compared against alternatives in this specific setting.

Mitigation status. Not addressed. The paper treats the +10-point gain as a proof of concept that How2Score can serve as an RL reward, not as a comprehensive optimization of the training recipe. This is a reasonable scope decision for a framework paper — demonstrating the closed loop matters more than maximizing performance — but it leaves practitioners without guidance on scaling RL training for procedural generation. Future work on training budget scaling, algorithm comparisons, and over-optimization thresholds is implied by the paper's framing (Section 8 in the full paper likely discusses future directions) but not addressed experimentally.

---

6.6 The Evaluation Protocol Penalizes Models for Refusals and Assumes Reference Correctness Without Verification

The assumption or constraint. How2Score's binary judgment (has_failure / no_failure) depends on two structural assumptions that create systematic measurement biases:

1. Refusals are treated as failures. When a model refuses to generate a procedure — for example, declining to provide legal or health advice — this receives a has_failure label under How2Score (Section 5.2, qualitative analysis: "The paper notes occasional refusals, primarily in frontier models... These are counted as failures under How2Score"). A refusal is fundamentally different from an incorrect procedure — it reflects a safety policy decision, not a capability failure — but How2Score conflates them.
2. Reference correctness is assumed. How2Score judges deviations from the reference as potential critical failures. The paper acknowledges that 3.4% of How2Bench references may contain errors (96.6% validity rate; Section 4.1), but these references remain in the benchmark. For those 3.4% (~238 examples), a model that generates a correct procedure that deviates from an incorrect reference will be penalized, while a model that reproduces the reference error will be rewarded.

The consequence.

• Refusal bias against frontier models: GPT-5, which leads How2Bench at ~65% (Figure 4), may be penalized relative to models with less conservative safety policies. If GPT-5 refuses on 5% of examples where it could have generated a valid procedure, its true procedural capability is underestimated by up to 5 points. This makes How2Bench scores less comparable across models with different safety stances — a Qwen model that never refuses might outscore GPT-5 on certain topics not because it is better at procedures but because it is less cautious. The paper's qualitative analysis acknowledges refusal behavior but does not quantify its frequency or impact on scores.
• Reference error propagation: For the 3.4% of references with errors, How2Score actively penalizes correctness. A model that correctly identifies that a reference step is unsafe or unnecessary and omits it receives has_failure; a model that faithfully reproduces the error receives no_failure. This introduces noise into the aggregate score and creates perverse incentives during RL — the model is rewarded for reproducing reference mistakes. The effect on aggregate scores is small (3.4% of examples, assuming models are equally likely to reproduce or correct reference errors), but it represents a systematic bias toward reference conformity over actual procedural validity. The bias is particularly concerning for RL because the model might learn to prefer reference-conforming errors to correct-but-different approaches, subtly degrading real-world reliability even as How2Bench scores improve.

What evidence exists in the paper. The 96.6% reference validity rate is reported (Section 4.1) but not stratified by topic or difficulty — some topics might have substantially lower validity rates, creating topic-specific biases. The paper does not report refusal rates per model or analyze whether refusal-affected scores change model rankings. The qualitative analysis acknowledges both phenomena but treats them as minor observations rather than systematic measurement concerns. There is no experiment where refusals are excluded from scoring, no analysis of whether How2Bench rankings change when refusals are handled differently, and no per-topic validity breakdown that would reveal whether certain topic scores are less reliable.

Mitigation status. The paper treats both issues as acceptable noise. The 96.6% validity rate is presented as evidence of data quality (implicitly: the noise is small enough to ignore), and refusals are mentioned in qualitative analysis without corrective action. Neither issue is flagged as a limitation requiring future work. For refusal bias, a straightforward mitigation would be to report scores both with and without refusal examples, or to develop a separate "refusal rate" metric alongside How2Score. For reference errors, the paper could apply a stricter validity threshold (e.g., only include examples where GPT-4.1 confidence is high) or conduct human verification on the 3.4% of suspect references. Neither mitigation is implemented or proposed.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper makes a methodological contribution whose significance extends beyond procedural generation: it provides a worked example of how to construct scalable, reproducible evaluation and training loops for capabilities where ground-truth verification is impossible. The architecture — mine reference examples from pretraining web data, define a targeted failure-detection protocol, validate it against human annotators, distill it into an open model for reproducibility and cost, and use it as both a benchmark and an RL reward — is a generalizable pattern. It is not a paradigm shift in the sense of introducing a new model architecture or training objective, but it is a substantive reframing of how to approach evaluation for open-ended generation tasks where execution-based verification (as in math or code) is unavailable.

The specific reframing that matters most is the shift from quality-rating to failure-detection. Prior to this work, the natural approach to evaluating open-ended generation was holistic: "How good is this output on a scale of 1–5?" or "Is response A better than response B?" This is how LLM-as-a-judge protocols operate (Zheng et al., 2023; Dubois et al., 2025), and it is intuitively appealing — after all, human evaluations of text quality are typically holistic. But the paper demonstrates that this framing has a specific, consequential failure mode when applied to procedural content: it rewards surface qualities (coherence, fluency, thoroughness) that are orthogonal to task success. A well-written procedure with a critical omission scores well on quality ratings but fails in practice. The paper's alternative — ask a specific, falsifiable diagnostic question ("Does this procedure contain any critical failure?") rather than a holistic one ("Is this procedure good?") — produces evaluation that is simultaneously more reliable (higher inter-annotator agreement: α = 0.593 for binary detection vs. 0.307 for first-failure localization; Section 4.3) and more scalable (binary labels map directly to RL rewards; Section 6).

This reframing has implications for domains beyond how-to procedures. Any task where "correctness" cannot be automatically verified but can be assessed by identifying specific failure modes — legal document generation, medical advice summarization, technical documentation, safety-critical instruction following — could adopt the same pattern. The key insight is that detecting errors is often easier and more reliable than rating quality, and that this asymmetry can be exploited to build evaluation systems that are both more accurate and more practical as training signals. The paper does not claim this insight as a theoretical contribution, but it demonstrates it through the architecture of How2Everything — making it accessible for other researchers to adopt and adapt.

The paper also reconciles a tension in the LLM-as-a-judge literature. Prior work (Zheng et al., 2023) raised concerns about self-preference bias (judges favoring their own model family's outputs) and verbosity bias (judges rewarding longer outputs regardless of quality). These concerns have led some researchers to view LLM-as-a-judge evaluation as inherently unreliable for anything beyond rough comparisons. The paper's cross-judge robustness analysis (Figure 5) demonstrates that while absolute score values vary across judges, relative model rankings are stable — even when the judge comes from a different model family than the generators being evaluated. This suggests that LLM-as-a-judge protocols, when designed as targeted failure-detection tasks rather than holistic quality ratings, can produce reliable comparative results. The stability of rankings across GPT, Gemini, Claude, and How2Judge is evidence that the critical-failure detection task taps into a shared capability of frontier models rather than into model-specific biases. This finding should increase confidence in well-designed LLM-as-a-judge protocols and redirect concern from "should we use LLM judges at all?" to "how should we design LLM judge prompts to target specific failure modes?"

The distillation of How2Judge establishes a design pattern for reproducible benchmarking: if you need a judge that can be used thousands of times (for evaluation) or millions of times (for RL training), invest in collecting frontier-model annotations on a diverse set of generator outputs, distill into an open model, and validate against human labels. The 2.5-point drop from GPT-5's 83.0% human agreement to How2Judge's 80.5% is the explicit cost of reproducibility — and the paper argues it is worth paying for a benchmark intended to support long-term research. This is a practical contribution to the infrastructure of LLM evaluation, and it provides a template that other benchmark creators can follow.

Finally, the paper demonstrates that procedural generation is a learnable, generalizable capability — not a narrow format-specific skill or a memorization artifact. The confound diagnostics (Section 7) are methodologically significant because they establish a template for stress-testing training improvements on any benchmark: test whether gains depend on pretraining maturity (Figure 6), test whether gains require diverse topic coverage or transfer from narrow training (Section 7.1), and test whether gains can be explained by data memorization (Table 3). This multi-pronged experimental approach provides stronger causal evidence than the overlap-based contamination analyses common in prior work. It establishes that procedural generation improvements from RL represent genuine capability amplification — the model is getting better at the underlying skill, not just learning the benchmark's surface patterns — and it provides a diagnostic toolkit that other researchers can apply to their own benchmarks and training improvements.

Follow-Up Research This Work Enables

1. Human evaluation of RL-trained outputs to validate that How2Score gains correspond to real-world improvement. The paper demonstrates that RL with How2Score as the reward improves How2Bench scores by >10 points (Table 2), but these gains are measured by How2Judge — which has 80.5% agreement with humans and may share biases with the teacher model (GPT-5) used for distillation. A critical validation experiment would sample ~200 RL-trained generations from the Qwen 3 8B Instruct RL model (48.62 How2Bench) and the base instruct model (38.52), present them to human annotators (using the same protocol from Section 4.3), and measure whether the +10.1-point How2Bench gain corresponds to a human-perceived improvement of similar magnitude. If human evaluators confirm the gain, the closed-loop framework is validated end-to-end. If humans perceive a smaller gain (e.g., +5 points rather than +10), it would indicate that RL partially optimizes for judge-specific biases that humans do not share — a finding that would motivate work on judge debiasing or ensemble judging. If humans perceive no gain, the framework's training loop is fundamentally broken, and the paper's central practical contribution would need reevaluation. This experiment costs approximately $3,600–$5,000 (extrapolating from the paper's annotation costs in Section 4.3) and is the single highest-priority follow-up.

2. Difficulty-stratified analysis of what RL actually improves. The paper reports aggregate How2Bench gains from RL (+10.10 for Qwen 3 8B Instruct; Table 2) but does not analyze where the gains concentrate. A natural follow-up would stratify How2Bench results by reference step count (the dominant difficulty predictor identified in Section 5.2) and by topic, measuring RL gains separately for procedures with 5–7 steps, 8–10 steps, 11–13 steps, and 14–15 steps, and for each of the 14 topics. This analysis would reveal whether RL helps primarily on short, simple procedures (where the base model already has partial knowledge and RL refines it) or on long, complex procedures (where RL teaches the model to maintain coherence over many steps). It would also reveal whether certain topics (e.g., cooking vs. legal) benefit disproportionately from RL, which would inform data mixture strategies for future training runs. Additionally, categorizing pre-RL and post-RL errors by failure type (omissions, missing parameters, wrong values, contradictions, unsafe actions — using the taxonomy from the qualitative analysis in Section 5.2) would reveal which failure modes RL most effectively addresses. If RL primarily reduces omissions but not parameter errors, that suggests a specific direction for improving the reward function (e.g., adding a parameter-accuracy reward component). This analysis requires no new data collection — only re-running How2Judge on already-generated pre-RL and post-RL outputs and parsing the structured failure descriptions that How2Judge already produces.

3. Training-budget scaling curves for RL on procedural generation. The paper uses a fixed 1,000 GRPO steps for all RL experiments without exploring how performance changes with training duration. A systematic study would train Qwen 3 8B Instruct with How2Score reward for 5,000–10,000 steps, evaluating How2Bench and the 12 out-of-domain benchmarks every 500 steps. Three questions this would answer: (a) Does How2Bench performance plateau at some step count, or does it continue improving? If it plateaus at ~2,000 steps, the paper's 1,000-step budget was near-optimal; if it continues improving through 10,000 steps (e.g., reaching 60–65% instead of 48.62%), the paper's results significantly understate the potential of this approach. (b) At what training budget do out-of-domain regressions begin to appear? The paper's Δ̄_OOD values are near zero at 1,000 steps (Table 2), but more aggressive optimization of How2Score might eventually cause the model to overfit at the expense of general capabilities — a pattern well-documented in RL fine-tuning. Identifying the "safe" training budget would have practical value for practitioners adopting this method. (c) How does the RL training curve interact with model size? Running the same experiment on 1B, 4B, 8B, and (if resources permit) 32B models would reveal whether larger models benefit from longer RL training or reach diminishing returns more quickly than smaller models. The paper's scaling analysis (Figure 1b) shows that pretraining FLOPs matter; how RL training FLOPs interact with pretraining scale is an open and practically important question.

4. Comparison of GRPO against simpler optimization methods for How2Score improvement. The paper uses GRPO with three reward components (How2Score, format verifier, length penalty) based on prior work in mathematical reasoning (Shao et al., 2024), but does not test whether a simpler approach would achieve comparable gains. Three baselines would contextualize the GRPO results: (a) Best-of-N rejection sampling: generate N=8 or N=16 candidate procedures for each How2Train prompt, score them with How2Judge, and fine-tune on the highest-scoring generation (or on all generations with no_failure labels). This is simpler to implement, requires no reward shaping, and has been effective for other tasks. If rejection sampling with N=16 achieves +8–9 points (compared to GRPO's +10.10), the additional complexity of GRPO might not be justified for most practitioners. (b) Direct Preference Optimization (DPO): use How2Judge to compare pairs of model-generated procedures (one scored no_failure, one scored has_failure for the same prompt) and apply DPO to push the model toward the successful generation. This avoids the need for a length penalty (since both generations in a pair are from the same prompt, length differences are implicitly controlled) and the need for a format verifier reward (format quality would naturally emerge from preference learning). (c) GRPO without the format verifier and length rewards: ablate the two auxiliary rewards to isolate the contribution of the How2Score signal alone. If GRPO with only the binary How2Score reward achieves >8 points, the auxiliary rewards are adding marginal benefit; if it achieves only 2–3 points, the auxiliary rewards are load-bearing and the framework's dependence on careful reward engineering is higher than the paper suggests. These comparisons would cost approximately 3–6 additional training runs per model and would substantially clarify which components of the training setup are essential versus incidental.

5. Cross-domain generalization: How2Mine applied to non-tutorial formats and non-procedural tasks. The paper restricts How2Mine to documents classified as "Tutorial & How-to Guide" by WebOrganizer, acknowledging that the pipeline "can be easily extended to extract valid procedures from other formats such as academic writing and knowledge articles" (Section 3.1 footnote). A direct follow-up would apply the five-stage How2Mine pipeline (extraction, heuristic filtering, LLM filtering, post-processing, validation) to ~100K documents from a different format category — e.g., "Academic Writing" or "Knowledge Article" — and measure (a) the yield rate (fraction of documents producing valid procedures), (b) the reference validity rate (GPT-4.1 acceptance, analogous to the 96.6% reported for tutorial documents), and (c) the qualitative differences in extracted procedures (are academic procedures systematically longer? more technical? more domain-specific?). If yield and quality remain high, How2Mine is validated as a format-agnostic extraction pipeline; if yield drops substantially (e.g., <10% vs. ~36% for tutorial documents), the pipeline's dependence on format-specific cues is higher than the paper implies, and format-adaptive extraction prompts or filtering criteria would be needed. More ambitiously, the How2Mine pattern could be adapted to extract structured task data for non-procedural tasks — for example, extracting "argument-counterargument" pairs from debate forums to create a dataset for evaluating reasoning, or extracting "problem-diagnosis-solution" triples from technical support threads to create a dataset for troubleshooting evaluation. Each adaptation would require domain-specific extraction prompts and filtering criteria, but the pipeline architecture (mine from web corpora, use LLMs to structure, validate against human labels, distill into evaluation protocol) would transfer. This follow-up would test the paper's implicit claim that How2Mine is a "generalizable pipeline for converting raw pretraining web data into structured task-specific evaluation and training data" (from Section 4 of this analysis) rather than a procedure-specific one-off.

6. Stress-testing How2Judge with adversarial examples and domain-expert annotation. The paper's human validation of How2Judge uses 200 examples pre-screened to avoid specialized domain knowledge (Section 4.3), and the 80.5% agreement figure may not hold for technically complex procedures. A direct stress-test would construct a set of ~100 How2Bench examples from the most technically demanding topics (Crime & Law, Health, Industrial, Science/Math & Technology) — including edge cases where the reference procedure is correct but a generated alternative is also correct (requiring the judge to distinguish acceptable variations from critical failures), where the reference procedure contains a subtle error (testing whether How2Judge correctly identifies deviations that improve validity), and where safety constraints are implied but not explicit in the goal (testing whether How2Judge penalizes unsafe generated procedures). These examples would be annotated by domain experts (e.g., paralegals for legal procedures, medical students for health procedures) rather than general crowdworkers, providing a higher-quality human baseline than the paper's Prolific annotations. If How2Judge maintains >75% agreement with domain experts on these difficult examples, the judge's reliability extends to the most challenging evaluation cases. If agreement drops substantially (e.g., to 60–65%), the framework has a reliability cliff at high difficulty that users must be aware of, and the paper's aggregate How2Bench scores for frontier models (which score highest on difficult examples) would be less trustworthy. This experiment would cost more than the paper's annotation effort (domain experts are more expensive than Prolific workers) but would provide essential calibration for the benchmark's reliability on the examples where evaluation quality matters most — not the easy cases where all judges agree, but the difficult cases that distinguish strong models from very strong ones.

Practical Applications and Downstream Use Cases

1. Cost-efficient evaluation for how-to chatbot quality assurance. The paper's motivational analysis (Section 1, Appendix A) establishes that 8.5% of ChatGPT conversations are how-to requests — tens to hundreds of millions of interactions daily. For organizations deploying LLM-powered assistants, monitoring the quality of how-to responses is critical (incorrect procedures can cause real harm) but expensive with human review. How2Judge provides a practical solution: for any domain where a corpus of reference procedures exists (or can be mined using How2Mine), the distilled 8B model can score generated procedures at approximately zero marginal cost per evaluation. A deployment pipeline could sample 1,000 how-to interactions per day, extract the model's generated steps, compare them against reference procedures from a domain-specific How2Mine corpus, and flag any procedure with a has_failure judgment for human review. At GPT-5 API prices ($0.002 per call), scoring 1,000 procedures daily would cost ~$2/day; with How2Judge, the cost is effectively zero. The 80.5% human agreement means approximately 4 out of 5 flagged procedures would be confirmed as problematic by human reviewers — a high enough precision to make the pipeline useful as a triage mechanism, dramatically reducing the volume of procedures requiring expensive human review while catching most critical failures. The paper's finding that relative model rankings are stable across judges (Figure 5) means that this pipeline could also track whether model updates improve or degrade procedural quality over time, even as the underlying LLM changes.

2. Training data generation for procedural fine-tuning in specialized domains. The How2Mine pipeline can be pointed at any web corpus to extract domain-specific procedures. For organizations building LLMs for specialized applications — e.g., a legal tech company developing an assistant that helps pro se litigants file court documents, a healthcare company building a patient-facing procedure-explanation system, or a manufacturing company creating a technician support chatbot — How2Mine could extract thousands of reference procedures from domain-specific web corpora (legal aid websites, medical guideline documents, equipment maintenance manuals). These procedures could then be used for SFT (teaching the model the domain's procedural format and conventions) and RL (using How2Score as the reward to optimize for procedural validity). The RL results in Table 2 (+10.10 points for Qwen 3 8B, with no systematic out-of-domain regression) suggest that domain-specific RL on how-to procedures would improve procedural quality in the target domain without degrading the model's general capabilities. The paper's topic-restricted RL results (Section 7.1) further suggest that training on a single broadly-dispersed topic (analogous to a domain like "legal procedures" which spans many subdomains) transfers well to other topics — meaning a legal-specific RL phase might even improve the model's procedural generation in adjacent domains (e.g., general how-to advice) as a positive side effect.

3. Diagnostics for procedural capability development during pretraining. The paper's scaling analysis (Figure 1b) demonstrates that How2Bench provides signal across the entire training pipeline, from early pretraining (~10²¹ FLOPs for a 1B model) through post-training, with clear differentiation across model sizes and training stages. For organizations training large language models from scratch, this makes How2Bench a useful diagnostic tool: evaluate intermediate pretraining checkpoints on How2Bench to track when and how procedural generation capabilities emerge, and use this signal to inform data mixture decisions (e.g., "procedural capability is lagging behind math and code at the 50% pretraining mark — should we increase the proportion of tutorial-style documents in the next training phase?"). The paper's finding that surface formatting stabilizes early while procedural validity continues improving throughout training (Section 5.2, Appendix) provides a specific diagnostic pattern: if a training run shows continued improvement in formatting proxies but plateauing How2Score, that indicates a data or capacity bottleneck specifically for procedural content (as opposed to a general training issue). The availability of How2Judge as an open, reproducible model means these diagnostics can be run at scale across multiple training runs without API costs or reproducibility concerns, enabling systematic study of how procedural capabilities develop during pretraining — a research question that the paper's scaling analysis opens but does not exhaustively answer.

4. A calibration target for improving LLM-as-a-judge reliability more broadly. The paper's finding that targeted failure detection achieves higher inter-annotator agreement than holistic quality rating (α = 0.593 vs. 0.307; Section 4.2) provides a concrete design principle for LLM-as-a-judge protocols: decompose evaluation into specific, falsifiable checks rather than asking for holistic assessments. The How2Score protocol — a detailed prompt with a codebook, examples of critical vs. non-critical failures, and structured output specifying each failure with references to relevant steps — is a template that can be adapted to other domains where holistic quality ratings are unreliable. For example, evaluating generated legal documents could be decomposed into checks for specific failure types (missing clauses, incorrect citations, logical contradictions between sections) rather than asking "is this legal document good?" Evaluating generated medical advice could be decomposed into checks for safety violations, factual errors, and omissions of critical warnings. In each case, the paper's methodology — iterative codebook refinement with human annotators, validation against human majority labels, and distillation into a specialized small model — provides a replicable template. The key practical insight is that the upfront investment in developing a detailed failure-detection codebook (which the paper did through multiple rounds of pilot annotations and training material refinement; Section 4.3) pays off in both improved human agreement and improved LLM judge calibration, enabling the subsequent distillation and RL training that would be unreliable with a cruder evaluation protocol.