A Case Study of Web App Coding with OpenAI Reasoning Models

ArXiv: 2409.13773

🎯 Pitch

OpenAI's o1 reasoning models break records on single-task web coding, but collapse entirely—falling behind Claude 3.5—when tasks are combined, showing a 0% pass rate in some multi-task formats. The reasoning process that powers their top performance becomes a liability when key instructions are missed, causing the model to double down on flawed plans. This reveals that better reasoning only helps if the base model first absorbs every detail correctly.

1. Executive Summary

This paper presents a case study evaluating OpenAI's o1 reasoning models on practical web app coding tasks using the WebApp1K single-task benchmark and the newly introduced WebApp1K-Duo duo-task benchmark. The o1 models achieve state-of-the-art pass@1 results on WebApp1K — 0.952 for o1-preview and 0.939 for o1-mini, lifting the prior SOTA by 7% — but their performance declines sharply on WebApp1K-Duo, falling behind Claude 3.5 under the normalized format (0.652–0.667 vs. 0.679) and catastrophically failing all problems under the raw format (0.0 pass@1). The paper attributes this variability to instruction comprehension: the reasoning mechanism amplifies success when all expectations are captured from the prompt but exacerbates failure when key expectations are missed or misprioritized during the reasoning chain — such as committing to multi-module splitting strategies from the first planning step, or overlooking error-handling expectations possibly due to input length — establishing that reasoning models' coding success hinges on meticulous instruction adherence only when base model quality and SFT ensure expectations are fully internalized before reasoning proceeds.

2. Context and Motivation

The Core Problem: Reasoning Models Exhibit Wildly Variable Performance on Practical Coding Tasks

This paper addresses a specific, puzzling phenomenon: OpenAI's o1 reasoning models achieve stunning results on single-task web app coding problems but then collapse dramatically when those same tasks are combined into slightly more complex scenarios. The core question is not simply "how good are o1 models at coding?" but rather: why does the reasoning mechanism that enables breakthrough performance on one set of tasks become a liability on another, superficially similar set of tasks?

This matters because it reveals a fundamental tension in how reasoning models operate. The reasoning process — the model's internal chain-of-thought that systematically works through a problem before producing an answer — is supposed to make models more robust, more careful, and less prone to the kinds of superficial pattern-matching errors that plague standard LLMs. Yet the paper documents cases where reasoning models make errors that non-reasoning models occasionally avoid, and cases where reasoning models fail every single problem in a category that non-reasoning models solve roughly a third of the time. This is not a marginal performance difference — it is a qualitative failure mode that challenges assumptions about what reasoning models can and cannot do.

Why This Is Important: Real-World Software Development Is Not Single-Task

The practical stakes are high because the paper's benchmarks are explicitly designed to reflect real-world software development: implementing web apps that satisfy specific test cases, handle user interactions, and interact with mocked APIs. This is not competitive programming (where o1 models also excel) or mathematical reasoning (where they set SOTA). It is the kind of task that working developers face daily — "build a component that does X, Y, and Z, and here are the tests it must pass."

The key distinction the paper draws is that this domain is results-oriented rather than explorational (Section 1). In math or science benchmarks, models often benefit from exploring multiple solution paths, and the reasoning process helps them navigate complex deductive chains. But in practical coding with explicit test cases, the "right answer" is defined precisely by the test expectations — there is no ambiguity or creative interpretation. The challenge is not discovering novel insights but meticulously attending to all specified requirements. As the paper puts it:

"The problem is less explorational and more results-oriented than other benchmarks. The specific instructions are laid out in the form of test setup and expectations."

This framing matters because it suggests that the strengths reasoning models demonstrate in other domains — exploration, hypothesis search, multi-step deduction — may not translate directly to instruction-following tasks where the intellectual challenge is primarily about comprehensive attention to detail rather than deep reasoning. If reasoning models underperform on such tasks in deployment, it limits their practical utility for software engineering workflows where developers want models to faithfully implement specifications, not creatively reinterpret them.

Additionally, the paper notes that these tasks require no external knowledge — React is a well-established framework with abundant training data on the internet (Section 1). This eliminates confounds about knowledge cutoff or factual recall. Any failures stem from how the model processes the given instructions, not from knowledge gaps.

The Gap in Prior Work: Reasoning Evaluations Focus on Different Domains

Prior evaluations of reasoning models — and the reasoning techniques they build on — have concentrated on domains that are fundamentally different from practical specification-following:

Mathematical reasoning and competitive programming. The o1 models were evaluated primarily on math benchmarks (e.g., those in Zhang et al., 2023), scientific reasoning (GPQA; Rein et al., 2023), and competitive programming (Codeforces; Mirzayanov, 2009), where they achieved SOTA results. These domains share a common structure: the problem statement describes a scenario, and the model must discover the correct solution through logical deduction, mathematical manipulation, or algorithmic insight. The reasoning chain is genuinely additive — it produces new conclusions from premises.

Existing software engineering benchmarks have different focuses. The paper situates WebApp1K in the context of other code evaluation benchmarks (Section 4):

• SWE-bench (Jimenez et al., 2024) targets core software engineering activities — code generation, completion, error detection, debugging — using real GitHub issues. This is more realistic but also more open-ended, without the precise test-case-gated evaluation that WebApp1K provides.
• BFCL (Yan et al., 2024) evaluates function calling, including prompt interpretation and argument handling — relevant to instruction following but focused on API call generation rather than full component implementation.
• BIRD (Gao et al., 2023) evaluates text-to-SQL translation, a narrower and more structured domain.
• Aider Leaderboard (Aider, 2024) ranks models on real-world programming tasks like bug fixing and refactoring, but the tasks are not systematically controlled for complexity or difficulty.

None of these benchmarks specifically isolate the instruction comprehension vs. reasoning capability tension that WebApp1K is designed to probe. WebApp1K's structure — single-task problems with exactly two test cases each, defining atomic actions with clear success and failure expectations — creates a controlled setting where the model either attends to all specified requirements or it doesn't. There is no ambiguity about what "correct" means.

Where Existing Approaches Fall Short

The paper identifies specific limitations in how prior work conceptualizes reasoning model capabilities:

No prior work examines reasoning models on specification-following coding tasks. The releases and evaluations of reasoning models (OpenAI, 2024; the self-play, self-taught reasoning, and chain-of-thought literature cited in Section 4) have focused on domains where the reasoning process adds value by discovering non-obvious solution paths. In these domains, occasional reasoning errors are tolerable because the reasoning chain still produces a better answer than no reasoning at all. But in specification-following, a single missed requirement in the reasoning chain can derail the entire implementation — there is no partial credit for "almost passing all tests."

The relationship between reasoning and instruction adherence is unexplored. The literature on reasoning models (Section 4) covers self-play fine-tuning (Chen et al., 2024; Zhang et al., 2024), self-taught reasoning (Zelikman et al., 2022, 2024), reinforcement learning for decision-making optimization (Silver et al., 2017), chain-of-thought reasoning (Wang and Zhou, 2024; Lightman et al., 2023), inductive reasoning and hypothesis search (Wang et al., 2024), and advanced sampling methods like tree search (Anthony et al., 2017; Uesato et al., 2022). All of these techniques are designed to help models explore reasoning space more effectively. None of them address the orthogonal challenge of faithfully encoding all input constraints into the reasoning process.

This is a critical blind spot. If reasoning models systematically fail to incorporate certain types of instructions into their reasoning chains, then improving the reasoning mechanism (better search, better self-play, better chain-of-thought prompting) will not help — the problem is upstream of reasoning, at the point where the model parses and prioritizes the instructions themselves.

WebApp1K itself has been studied, but only with non-reasoning models. The author's prior work (Cui, 2024a, 2024b) benchmarked frontier non-reasoning models on WebApp1K, establishing the ~0.88 pass@1 ceiling that o1 models subsequently break (0.952). That work identified common error patterns — semantic misunderstandings, overlooked edge cases, frontend-vs-backend validation confusion — but did not have access to reasoning models to study how the reasoning mechanism interacts with these error patterns.

How This Paper Positions Itself

The paper positions itself as a diagnostic case study, not as a proposal for new methods or benchmarks. The WebApp1K-Duo benchmark is introduced specifically to probe the failure modes observed with o1 models — it is not presented as a general-purpose benchmark but as a targeted instrument for understanding when and why reasoning models' performance varies.

The paper's thesis is stated directly in the introduction and reinforced throughout: the variability in o1 model performance is due to instruction comprehension. Specifically:

"the reasoning mechanism boosts performance when all expectations are captured, meanwhile exacerbates errors when key expectations are missed, potentially impacted by input lengths."

This is a more nuanced claim than "reasoning models are good" or "reasoning models are bad." It asserts that the reasoning mechanism functions as an amplifier: it makes the model's output more extreme in the direction established early in the reasoning chain. If the chain correctly captures all test expectations, the reasoning process systematically works through them and produces a correct implementation. But if the chain commits to a misunderstanding or misses a key requirement — particularly in the early planning steps — the reasoning process systematically elaborates and solidifies that error, making recovery nearly impossible.

The paper supports this amplification hypothesis with concrete evidence from reasoning chain analysis. In Section 2.3.1, a ChatGPT reenactment shows o1-preview's reasoning chain committing to frontend validation (the wrong approach) at an early step, then having a subsequent "verification" step that reinforces rather than corrects the error:

"More interestingly, the step Verifying form submission does not correct the wrong direction, but solidify it."

The paper speculates that this derailing is "due to preemption of original expectations by model's inherent knowledge" — the model's pretraining prior (frontend validation is best practice) overrides the explicit test requirement (the test expects a backend API call with a 400 response). And critically, once the reasoning chain takes this wrong turn, "subsequent verification is derived from neighboring steps already derailed, instead of original expectations only accessible from the input tokens."

This is a specific, mechanistic hypothesis about how reasoning models fail, not just a report that they do. It implies that the remedy is not better reasoning but better base model quality and SFT that ensure "meticulous adherence to instructions" (Section 5) — because once the instruction is properly internalized, the reasoning mechanism amplifies its correct implementation, but if the instruction is missed or overridden, no amount of reasoning can recover the lost information.

In the broader landscape of reasoning model research, this paper provides a counterpoint to the narrative of unqualified improvement. It demonstrates that reasoning is not a universal robustness enhancer — it creates new failure modes that are qualitatively different from those observed in non-reasoning models, and in some regimes (the raw format of WebApp1K-Duo) these failure modes are catastrophic (0.0 pass@1). This establishes an important boundary condition for reasoning model deployment: on tasks where comprehensive instruction adherence is the primary challenge, reasoning models may be less reliable than non-reasoning models if the base model's instruction-following capability is insufficient to prevent early-stage reasoning derailment.

3. Technical Approach

3.1 Reader Orientation

This paper does not build a new system, model, or training method — it is a diagnostic case study that probes the behavior of existing reasoning models (OpenAI's o1-preview and o1-mini) on practical coding tasks by systematically varying task complexity and observing where and how performance breaks down. The core analytical strategy is to compare reasoning models against non-reasoning frontier models on identically structured coding problems, then use qualitative analysis of reasoning chains (obtained via ChatGPT reenactment) to formulate mechanistic hypotheses about why reasoning models succeed on single tasks but catastrophically fail on combined tasks — specifically, that the reasoning mechanism acts as an amplifier that magnifies both correct instruction-following and early-stage instruction miscomprehension.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental apparatus has four components:

1. WebApp1K Single-Task Benchmark — a set of 1,000 coding problems where each problem requires implementing a single atomic action (e.g., submitting a form, retrieving posts) to satisfy exactly two test cases (one success, one failure), establishing the baseline for what models can achieve when complexity is minimal.

2. WebApp1K-Duo Benchmark — a newly constructed benchmark where pairs of atomic tasks from WebApp1K are randomly combined into duo tasks, doubling both the input test cases (four instead of two) and the implementation scope (one module implementing both tasks), in two format variants: raw format (preserving original export names, which creates an unintuitive double-default-import structure) and normalized format (unifying exports under a single App name), designed to systematically probe failure modes.

3. Model Evaluation Pipeline — a pass@1 measurement protocol where each model receives the test files as a prompt and must generate code that passes all tests in a single attempt, with results aggregated across all 1,000 tasks per benchmark.

4. Qualitative Reasoning Chain Analysis — a forensic method where, for specific problems where reasoning models succeed or fail, the identical prompt is fed to ChatGPT to produce a reasoning chain reenactment, and the chain's steps are examined for evidence of when and how the model captures, misses, or reprioritizes test expectations.

Information flows as follows: a prompt (test cases + instruction to generate code) enters the model → the reasoning model produces an internal chain-of-thought (invisible in the OpenAI API, reconstructed via ChatGPT reenactment) → the model outputs implementation code → the code is evaluated against the test cases using the standard Jest testing framework → pass/fail is recorded → for selected problems, the reasoning chain is analyzed to identify where expectations were captured or lost.

3.3 Roadmap for the Deep Dive

• First, the WebApp1K single-task benchmark structure — because the duo-task benchmark is built from it, and understanding what makes single tasks "atomic" is essential to understanding why combining them creates qualitatively different challenges.
• Second, the WebApp1K-Duo construction method and its two format variants — because the format distinction (raw vs. normalized) is the paper's primary experimental manipulation and drives the most dramatic performance differences.
• Third, the evaluation protocol and prompt design — because all results depend on exactly what information models receive and how success is measured.
• Fourth, the reasoning chain recovery method — because the paper's central mechanistic claims about amplification and early-stage commitment depend on analyzing reasoning steps that are not directly observable in the o1 API.
• Fifth, the model selection and comparison strategy — because understanding which models are compared and why establishes the baseline against which o1 behavior is interpreted.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a diagnostic case study paper whose core idea is that the reasoning mechanism in o1 models functions as an amplification process: it systematically elaborates correct solutions when test expectations are fully captured in early reasoning steps, but it systematically elaborates and solidifies errors when key expectations are missed or overridden by pretraining priors in those same early steps.

WebApp1K Single-Task Benchmark Structure

The WebApp1K benchmark (Cui, 2024b) is the foundation on which all experiments are built. Understanding its design is essential because every task in WebApp1K-Duo is constructed by pairing two WebApp1K tasks, and the paper's central question — why does performance degrade when tasks are combined? — can only be answered relative to the characteristics of the single-task baseline.

Task definition. Each of the 1,000 problems in WebApp1K specifies a single atomic action — a well-defined user-facing operation that involves interaction with the UI and access to a mocked backend API. Examples from the paper include "submitting a form" (the ticketSubmission problem), "retrieving all blog posts" (the retrieveAllBlogPosts problem), "adding a comment to a post" (the addComment problem), and "editing an existing post" (the postEditing problem). The term "atomic" here means that the task represents one complete, self-contained user workflow — not a subtask or a partial implementation.

Test case structure. Each single-task problem is defined by exactly two test cases, both of which must pass for the problem to be considered solved:

• Success case: sets up mocked API responses indicating successful operation (HTTP 200 for POST endpoints, valid response bodies for GET endpoints), simulates user interactions that trigger the operation, and asserts that the UI displays the expected success message and that the mocked API was called with the correct parameters exactly the specified number of times.
• Failure case: sets up mocked API responses indicating server errors (HTTP 400 or 500 status codes, error message bodies), simulates the same user interactions, and asserts that the UI displays the expected error message and that the mocked API was still called exactly once (i.e., the model should not skip the API call when fields are missing — it should call the API and then display the error from the server's response).

Each test file follows a fixed Jest testing pattern with React Testing Library, as illustrated in Table 1 of the paper. A concrete fragment of the pattern shown:

import TaskA from './TaskA';
test("Success at task A", async () => {
    ...
    render(<MemoryRouter><TaskA /></MemoryRouter>);
    ...
}, 10000);

The MemoryRouter wrapper provides routing context for React components. The timeout is set to 10,000 milliseconds (10 seconds). The actual test logic — mock setup, user interaction simulation via fireEvent, and expectations via expect — is specific to each task but follows this structural template.

What makes the tasks "atomic" and "results-oriented." The paper emphasizes that these tasks differ from exploratory reasoning benchmarks in a crucial way. In a math competition problem or a scientific reasoning question, the model benefits from exploring multiple solution paths and the reasoning process adds value by discovering non-obvious intermediate conclusions. In WebApp1K, the correct implementation is completely determined by the test expectations — the model does not need to discover anything novel; it needs to read the test file, identify every requirement (what mocks are set up, what user interactions are simulated, what assertions are made about the UI and API calls), and produce code that exactly satisfies those requirements. The intellectual challenge is comprehensive attention to detail, not creative problem-solving.

Implicit vs. explicit expectations. The paper identifies a critical property of the test cases: some expectations are "less explicit or less typical than others" (Section 1). For example, in the postEditing problem, the test simulates a fireEvent.change on an element carrying the text 'Edit'. For the test to work, the model's implementation must include a form element that visibly displays the string Edit — but this requirement is implicit in the test's selector logic, not stated as a separate instruction. A model that implements a form for editing but labels it with placeholder text like 'Enter new content' would functionally work for a human but fail the automated test because fireEvent.change targets elements by their displayed text.

Similarly, in the ticketSubmission problem, the test expects an API call to be made even when the title field is empty, with the error message 'Title is required' displayed only after receiving a 400 response from the server. This is an atypical pattern — standard frontend best practice, prevalent in pretraining data, performs client-side validation and skips the API call when required fields are empty. The model must override its pretraining prior (frontend validation is correct) and follow the test's explicit but unusual requirement (always call the API, display the server's error).

These implicit and atypical expectations are where reasoning models and non-reasoning models diverge — the paper's analysis centers on understanding how the reasoning mechanism processes (or fails to process) these subtle requirement signals.

Implementation scope. The paper notes that "The resulting lines of code is typically between 40 and 50." Each task requires a single React component file (conventionally named TaskA.js or similar) that imports necessary dependencies (React hooks, routing utilities), implements the component logic, and exports the component as a default export. The implementation involves standard React patterns: useState for form state management, fetch or equivalent for API calls, conditional rendering for success/error messages, and JSX for the UI structure. No external libraries beyond React and React Router are needed.

Benchmark organization. The 1,000 tasks are organized into categories (e.g., "Social Media" for post and comment operations, "Customer Support" for ticket operations) with 50 tasks per category. This categorization is used when constructing WebApp1K-Duo, where pairs are formed within categories to maintain thematic coherence.

WebApp1K-Duo Construction and Format Variants

WebApp1K-Duo is the paper's primary experimental instrument for probing why o1 model performance degrades. It is constructed systematically from WebApp1K to introduce two specific complexity factors — longer input and longer output — while preserving the exact same task semantics.

Construction procedure. The paper states: "Under each category of WebApp1K, we randomly pair up two atomic tasks into a duo task." Since each category in WebApp1K has 50 tasks, random pairing within each category produces 25 duo tasks per category, for a total of 500 duo tasks. However, the paper reports that "The benchmark still consists of 1000 tasks, with 50 for each category." This implies that the pairing is done twice per category (50 pairs, each pair constituting one duo task, yields 50 duo tasks × 20 categories = 1,000 duo tasks) or that the random pairing is repeated to reach 50 duo tasks per category, with different random pairings producing distinct problems. The exact pairing count mechanism is not elaborated, but the output is a benchmark of 1,000 duo tasks mirroring the structure of WebApp1K.

What changes when two atomic tasks are combined. Each duo task inherits all four test cases from its two constituent single-task problems — two test cases from Task A (success and failure) and two test cases from Task B (success and failure). The model receives four test cases in its prompt instead of two. The implementation must satisfy all four tests in a single component file — the model cannot split the implementation across separate modules (unless it produces an incorrect multi-module solution, as discussed below). This means the model must handle:

• Longer input: the prompt contains twice as many test cases, each with its own mock setups, user interaction sequences, and assertions. The total token count for the prompt roughly doubles.
• Longer output: the implementation must incorporate logic for both tasks, handling two sets of API endpoints, two sets of UI states (loading, success, error for each task), and potentially two sets of form inputs and interaction handlers. The resulting component is more complex than a single-task component.
• Non-interference requirement: the two tasks must coexist in one component without interfering with each other — the state management, API calls, and UI rendering for Task A must not break the expectations for Task B, and vice versa.

The two format variants. The paper introduces two versions of the duo-task prompt, distinguished by how the component imports are specified in the test files. This format distinction is the paper's key experimental manipulation — it is not a minor syntactic detail but a deliberate probe of how reasoning models handle implicit structural constraints.

Raw format (Table 5(a)). In the raw format, the test file retains the original export names from the constituent single-task problems:

import TaskA from './TaskA_B';
import TaskB from './TaskA_B';

test("Success at task A", async () => {
    render(<MemoryRouter><TaskA /></MemoryRouter>);
    ...
});

test("Failure at task A", async () => {
    render(<MemoryRouter><TaskA /></MemoryRouter>);
    ...
});

test("Success at task B", async () => {
    render(<MemoryRouter><TaskB /></MemoryRouter>);
    ...
});

test("Failure at task B", async () => {
    render(<MemoryRouter><TaskB /></MemoryRouter>);
    ...
});

The critical detail here is the JavaScript import semantics. Both TaskA and TaskB are default imports — they are imported without curly braces (import TaskA from ..., not import { TaskA } from ...). In JavaScript ES6 module syntax, a module can have only one default export. Default imports are name-agnostic — the importing file can assign any local name to the imported value, regardless of what name was used in the export statement. The notation import TaskA from './TaskA_B' means "import the default export from ./TaskA_B and bind it to the local name TaskA." The notation import TaskB from './TaskA_B' means "import the default export from ./TaskA_B and bind it to the local name TaskB."

Since both imports reference the same file ('./TaskA_B') and both are default imports, they must refer to the same underlying value — the single default export from TaskA_B.js. The test file then renders this same component twice, once wrapped in a <TaskA> JSX tag (which resolves to the imported value bound to TaskA) and once wrapped in a <TaskB> JSX tag (which resolves to the same imported value bound to TaskB). The two JSX tags are functionally identical — they render the same component.

This means the test file is semantically demanding a single component that handles all four test cases. The two different import names are a syntactic artifact, not a requirement for two separate components. But this semantic equivalence is non-obvious — it requires understanding that both imports resolve to the same module's default export. A model that misses this subtlety might incorrectly infer that two separate components (or two named exports) are needed.

Normalized format (Table 5(b)). In the normalized format, the imports are unified under a single name:

import App from './TaskA_B';

test("Success at task A", async () => {
    render(<MemoryRouter><App /></MemoryRouter>);
    ...
});

// ... all four test cases use <App />

This format makes the single-component requirement explicit and intuitive. There is one import, one component name, and four test cases that all render <App />. The semantic equivalence between the raw and normalized formats — both demand a single component with a single default export — is preserved, but the normalized format removes the syntactic misdirection that the double-default-import pattern creates.

Why the format distinction matters. The paper hypothesizes that the raw format's double-default-import pattern triggers a specific failure mode in reasoning models: the model's first reasoning step, which typically serves as a planning step, commits to a multi-component architecture based on a surface-level reading of the imports, and subsequent reasoning steps elaborate this incorrect plan without revisiting the import semantics. The normalized format removes this trigger, allowing the reasoning process to start from the correct premise (one component, one export) from the beginning.

The export rule context (Table 8). To help readers unfamiliar with JavaScript module semantics, the paper includes a reference table contrasting named exports and default exports. Key differences:

• Named exports (export const x = ...) can appear multiple times per module and must be imported with curly braces using the exact export name: import { x } from './module'.
• Default exports (export default ...) can appear at most once per module and are imported without curly braces using any local name: import anyName from './module'.

The raw format exploits the fact that default imports are name-agnostic — the two import statements are importing the same thing under different names, not two different things.

Evaluation Protocol and Prompt Design

The evaluation protocol is deliberately minimal and standardized, designed to measure pure instruction-following capability without confounds from few-shot examples, system prompts, or iterative refinement.

Pass@1 measurement. The paper reports only pass@1 results: each model receives the prompt exactly once and produces one implementation. If the implementation passes all test cases for that problem, it counts as a success; otherwise, it counts as a failure. The pass@1 rate for a benchmark is the fraction of its 1,000 problems that the model solves on the first attempt. There is no best-of-N sampling, no temperature variation, no iterative debugging — the model gets one shot.

The paper states: "Due to budget constraints, we only obtained pass@1 results for the o1 models." This is a significant methodological choice: pass@1 is the most stringent evaluation metric because it measures what the model can do without any retries or selection among multiple candidates. For non-reasoning models, pass@k (where k > 1) typically shows higher values because sampling multiple solutions and selecting the best one (or using majority voting) compensates for variability in individual generations. The pass@1 focus means the paper is measuring the model's raw reliability on a single attempt, which is the relevant metric for deployment scenarios where retrying is expensive or impossible.

Prompt structure. The prompt for each problem follows a consistent template:

Generate TaskA.js to pass the tests below:
(1)
{Test Case 1}
{Test Case 2}
. RETURN CODE ONLY.

For duo tasks, the prompt is adapted to reference the duo file name (e.g., TaskA_B.js) and includes four test cases instead of two. The instruction "RETURN CODE ONLY" is capitalized and sentence-final, signaling that the model should output implementation code without explanatory text, markdown formatting, or conversational framing.

The prompt includes:

1. The target filename — tells the model what to name its output file and implicitly signals the expected export structure (the test file's imports are visible in the test cases, so the model can see what names are being imported).
2. The test cases in full — every mock setup, user interaction, and assertion is visible to the model. Nothing is hidden or requires external knowledge.
3. The "RETURN CODE ONLY" directive — constrains the output format to minimize parsing issues.

What is NOT in the prompt. Critically, the prompt does not include:

• Any description of the task in natural language — the model must infer what to build solely from the test cases.
• Any examples of correct implementations — this is zero-shot code generation from test specifications.
• Any hints about edge cases, expected patterns, or common pitfalls.
• Any context about the benchmark's purpose or structure.

This design choice is deliberate: it isolates the model's ability to parse test specifications and translate them into implementations, without assistance from natural language task descriptions that might provide additional semantic cues. The model's only source of information about what to build is the test code itself.

Automated evaluation. The generated code is evaluated by running the Jest test suite (the same test cases provided in the prompt) against the model's implementation. A problem is scored as "passed" if and only if all test cases pass. There is no partial credit — missing even one assertion in one test case counts as a failure. This binary scoring aligns with the benchmark's results-oriented philosophy: in practical software development, code either passes the tests or it doesn't.

The test execution environment. While the paper does not detail the exact evaluation infrastructure, the standard WebApp1K evaluation pipeline (established in Cui, 2024b) uses a Node.js environment with Jest and React Testing Library. The generated code is placed in the expected file location, dependencies are installed, and the test suite is executed. The 10-second timeout per test (visible in the test code as , 10000) prevents infinite loops or hanging API calls from blocking evaluation.

Reasoning Chain Recovery Method

The o1 models' reasoning chains are not visible through the OpenAI API — the API returns only the final output, not the intermediate chain-of-thought tokens. This creates a fundamental challenge for the paper's diagnostic goal: how can the authors analyze what the reasoning model "thought" when they cannot see the reasoning?

ChatGPT reenactment as a proxy. The paper's solution is to use ChatGPT (a non-reasoning model with visible output) as a reasoning chain simulator: "Since reasoning tokens are invisible in OpenAI API, we share reasoning steps obtained from ChatGPT reenactment, i.e. feeding the identical prompt to ChatGPT" (Section 1).

The procedure works as follows:

1. For a selected problem where o1-preview or o1-mini exhibited interesting behavior (dramatic success or dramatic failure), the authors take the exact prompt that was sent to the o1 model.
2. They feed this identical prompt to ChatGPT (the specific ChatGPT model version is not specified, but it is presumably a GPT-4 variant, likely gpt-4o or chatgpt-4o-latest based on the models listed in the results tables).
3. ChatGPT generates a visible chain-of-thought — a sequence of reasoning steps that the model produces before (or interleaved with) its code output. The authors examine these steps for evidence of how the model processes the test expectations.
4. The reasoning steps are reported in the paper as bullet-point summaries (e.g., "Refining test details → Investigating the scripts → Considering functionality → ...").

What the reenactment captures and what it misses. This method provides a simulation of what the o1 models' reasoning might have looked like, not a direct observation. Several caveats apply:

• Model differences: ChatGPT is a non-reasoning model, while o1 models use a specialized reasoning mechanism (likely involving reinforcement learning for chain-of-thought optimization, as suggested by the related work discussion of self-play and self-taught reasoning). The reasoning patterns of ChatGPT may differ systematically from those of o1 models.
• Post-hoc nature: The reenactment is generated after the authors know whether the o1 model succeeded or failed, introducing potential confirmation bias in interpreting the reasoning steps.
• Specificity: ChatGPT's reasoning steps are high-level summaries — e.g., "Refining the approach" — which the authors then interpret. The actual intermediate computation inside o1 models is likely more granular and may involve numerical optimization or tree search that ChatGPT does not replicate.

However, the reenactment method does capture something real: when ChatGPT, given the same prompt, produces reasoning steps that match the behavior pattern observed in o1 models (e.g., committing to frontend validation, splitting into two components), it provides evidence that the prompt itself contains cues that can derail reasoning, independent of the specific reasoning implementation. The reasoning chain structure — a sequence of planning, refinement, and implementation steps — is common across reasoning approaches, and the point where a step goes wrong is informative regardless of the exact reasoning algorithm.

Reporting conventions. The paper reports reasoning steps in two formats:

• Concise chain summaries using right-arrow notation: "Refining test details −→ Investigating the scripts −→ Considering functionality −→ ..." (Section 2.2). These list the sequence of reasoning step titles.
• Detailed step content for critical steps: quoted passages showing the exact reasoning text produced by ChatGPT, such as "I'm thinking about creating a TicketSubmission component with a 'Title' input and 'Submit' button. Submitting the form will trigger a POST request to '/api/tickets', validating the 'Title' field before submission" (Section 2.3.1).

The "blackened" convention. Throughout Sections 2 and 3, the paper uses the term "blackened" to indicate reasoning steps that are specifically relevant to the expectation being analyzed — for example, "steps specifically reasoning Edit is blackened" (Section 2.2). In the original document, these steps are presumably visually highlighted (bolded or colored). In the text, they serve as markers pointing the reader to the reasoning steps where the model engaged with the critical requirement.

The amplification hypothesis and reasoning chain evidence. The paper's central mechanistic claim — that the reasoning mechanism amplifies both success and failure — is supported by reasoning chain analysis from two contrasting cases:

Success case (Section 2.2, postEditing problem). The reasoning chain includes a step explicitly labeled "Editing content" and another labeled "Refining selector logic." These steps, when examined in detail, show the model considering how the test's fireEvent.change call targets an element by its text content. The model correctly infers that it must attach the string Edit to the form element (as placeholder text or visible label) so that the test's selector can find it. The reasoning process has successfully captured the implicit expectation and elaborated it into a correct implementation detail.

Failure case (Section 2.3.1, ticketSubmission problem). A ChatGPT reenactment of o1-preview's reasoning shows an early step — "Customer service improvement" — where the model writes: "I'm thinking about creating a TicketSubmission component with a 'Title' input and 'Submit' button. Submitting the form will trigger a POST request to '/api/tickets', validating the 'Title' field before submission." This commits to client-side validation (checking the title before making the API call). The next step — "Verifying form submission" — does not correct this error; instead, it elaborates it: "I'm thinking about how the form ensures 'Title' must be filled. It sends a POST request if 'Title' is entered, showing success or 'Title is required' based on the response status." The model is now reasoning about a hybrid pattern (check title first, but also handle server responses) without realizing that the test expects the API call to be made unconditionally and the error message to come only from the server response.

The paper's interpretation (Section 2.3.1) is that the derailing occurs because the model's pretraining prior about frontend validation best practices "preempts" the explicit test requirement. Once the reasoning chain commits to this wrong direction, subsequent verification steps derive their premises from neighboring derailed steps rather than from the original input tokens, making error recovery impossible. The reasoning process has amplified the initial miscomprehension into a systematically wrong implementation.

Duo-task raw format failure (Section 3.2). For the raw format of WebApp1K-Duo, the paper reports that the first reasoning step from a ChatGPT reenactment says: "To progress, the key task is creating components TaskA and TaskB in TaskA_B.js to ensure all tests are successfully passed." This commits to a multi-component architecture from the very beginning — the model has interpreted the two import statements as requiring two separate components, missing the semantic equivalence between default imports from the same module. The paper notes: "Comparing to the mistakes made in Sec. 2.3.1, the mistake in the above step covers a larger scope. It is reasonable to argue that mistakes made in large-scoped steps are more fatal and harder to correct." A planning-level error (wrong architecture) propagates through all subsequent reasoning, whereas an implementation-level error (wrong validation pattern) might be caught and corrected if later steps re-examine the original requirements. The reasoning chain's structure — early planning followed by elaboration — means that the scope of the first mistake determines whether recovery is possible.

Counterfactual evidence. The paper strengthens the amplification hypothesis by noting that non-reasoning models occasionally succeed on the raw format (Claude 3.5 achieves 0.32 pass@1 on the raw format, Table 6), while o1 models achieve 0.0. The non-reasoning models' occasional success suggests that the raw format is not impossible — the correct interpretation (one component, one default export) is reachable. But non-reasoning models do not have a systematic reasoning process that amplifies their initial interpretation, so when they happen to land on the correct architecture, they can produce a working solution. The o1 models' reasoning mechanism, by committing to a plan in the first step and then systematically elaborating it, makes the initial misinterpretation irreversible — there is no "happening to land on the correct architecture" because the reasoning process locks in the first interpretation.

Model Selection and Comparison Strategy

The paper evaluates six models across the two benchmarks, chosen to represent a cross-section of frontier capabilities as of late 2024.

Reasoning models:

• o1-preview: OpenAI's larger reasoning model, representing the strongest reasoning capability available at the time of the study.
• o1-mini: OpenAI's smaller, faster, and cheaper reasoning model, designed to provide strong reasoning performance at reduced cost.

Non-reasoning frontier models:

• gpt-4o-2024-08-06 (referred to as gpt-4o-2024-08-06 or chatgpt-4o-latest in different tables): OpenAI's multimodal flagship model, representing the strongest non-reasoning capability from the same organization.
• claude-3.5-sonnet: Anthropic's mid-tier model, which has shown strong performance on coding tasks in independent evaluations.
• deepseek-v2.5: DeepSeek's latest model at the time, representing strong open-weight alternatives.
• mistral-large-2: Mistral's largest model, another strong open-weight contender.

What this selection enables. The comparison set serves several analytical purposes:

1. Within-organization comparison: Comparing o1-preview and o1-mini against gpt-4o isolates the effect of the reasoning mechanism while controlling for the underlying organization's pretraining data and infrastructure. If o1 models outperform gpt-4o on single tasks but underperform on duo tasks, the difference is attributable to the reasoning mechanism rather than to different pretraining corpora.

2. Cross-organization comparison: Comparing o1 models against Claude 3.5 establishes whether the observed patterns are specific to OpenAI's implementation of reasoning or generalize across model families. The fact that Claude 3.5 (a non-reasoning model) achieves the SOTA on the normalized duo-task benchmark (0.679 vs. o1-mini's 0.667) suggests that non-reasoning models can be more robust on certain instruction-following tasks.

3. Performance ceiling and floor: The spread across models (from mistral-large-2's 0.449 to Claude 3.5's 0.679 on normalized WebApp1K-Duo) establishes the range of current capabilities, contextualizing the o1 models' position. On the single-task benchmark, the o1 models set a new ceiling (0.952); on the duo-task benchmark, they fall within the existing range, not at the top.

Budget constraint on pass@k. The paper reports only pass@1 for o1 models due to budget constraints. For non-reasoning models on WebApp1K, pass@1 results are taken from the prior WebApp1K paper (Cui, 2024b) or from independent evaluation. This means the comparison is fair in terms of metric (all models are evaluated at pass@1) but potentially understates what o1 models could achieve with multiple attempts. The paper does not explore whether o1 models' pass@k scaling behavior differs from non-reasoning models — a question left for future work with larger evaluation budgets.

No fine-tuning or prompt engineering. All models are evaluated in a zero-shot setting with the standard prompt template. No model-specific prompt optimization, few-shot example selection, or fine-tuning is performed. This ensures that the results reflect the models' out-of-the-box instruction-following capabilities, which is the relevant metric for most practical deployment scenarios where users will not craft per-task optimized prompts.

4. Key Insights and Innovations

Innovation 1: The Reasoning Mechanism as an Amplifier, Not an Unqualified Improvement

The paper's most conceptually distinctive contribution is its reframing of what reasoning models do. Before this study, the dominant narrative — both in OpenAI's release communications and in the broader research community — treated reasoning as a robustness enhancer: models that think step-by-step should make fewer careless errors, catch their own mistakes, and produce more reliable outputs than models that generate answers directly. This view is implicit in the chain-of-thought literature (Wang and Zhou, 2024; Lightman et al., 2023), in self-play fine-tuning approaches that use reasoning to convert weak performance into strong (Chen et al., 2024; Zhang et al., 2024), and in self-taught reasoning frameworks where models bootstrap their own capabilities (Zelikman et al., 2022, 2024). The shared assumption is that reasoning adds a corrective layer — the model might start down a wrong path, but the reasoning process should help it detect and recover from errors.

This paper challenges that assumption directly by introducing the amplification hypothesis: the reasoning mechanism does not just correct errors — it amplifies whatever direction the early reasoning steps establish. When those early steps correctly capture all test expectations, the amplification produces breakthrough performance (o1-preview's 0.952 on WebApp1K, lifting SOTA by 7%). But when those early steps commit to a misunderstanding, the amplification produces catastrophic failure (o1 models' 0.0 pass@1 on the WebApp1K-Duo raw format, where non-reasoning models like Claude 3.5 achieve 0.32).

This is not a marginal performance tradeoff. It is a qualitative difference in failure mode: non-reasoning models fail probabilistically on hard problems (sometimes they get it right, sometimes wrong), while reasoning models fail systematically when their first reasoning step commits to a wrong interpretation. The paper provides concrete mechanistic evidence for this claim through the reasoning chain reenactments:

• Success amplification (Section 2.2): On the postEditing problem, the reasoning chain dedicates explicit steps to processing the implicit expectation that the form element must carry the string Edit (as shown by reasoning steps like "Editing content" and "Refining selector logic" in the ChatGPT reenactment). The reasoning process systematically works through the requirement, producing an implementation that non-reasoning models consistently miss because they treat the Edit label as redundant with the form's semantic purpose.

• Failure amplification (Section 2.3.1): On the ticketSubmission problem, the reenactment shows the reasoning chain committing to client-side validation at step "Customer service improvement" with the text "validating the 'Title' field before submission." The subsequent "Verifying form submission" step reinforces this wrong direction rather than catching the error — it reasons about a hybrid pattern (check title first, handle server response after) without ever re-examining whether the original test expects the API call to be made unconditionally. The reasoning process has taken an initial misreading and elaborated it into a coherent but wrong implementation.

• Catastrophic amplification (Section 3.2): On the duo-task raw format, the first reasoning step commits to a multi-component architecture — "To progress, the key task is creating components TaskA and TaskB in TaskA_B.js" — a planning-level error that affects everything that follows. The paper explicitly contrasts this with the single-task failure case: mistakes at the planning level (architecture decisions) are "more fatal and harder to correct" than mistakes at the implementation level (validation patterns), because later reasoning steps derive their premises from the plan rather than from the original input tokens. The reasoning chain has no mechanism to revisit the architectural assumption because all subsequent steps are about how to build two components, not whether two components are needed.

This amplifier framing is significant beyond the specific benchmark results because it identifies a structural property of reasoning models that is independent of any particular implementation. Any reasoning system that proceeds by establishing a plan early and then elaborating it step-by-step will exhibit this amplification behavior. It does not matter whether the reasoning uses chain-of-thought prompting, tree search, reinforcement learning, or some future architecture — the amplification is inherent in the sequential-planning-then-elaboration structure. This means that improving reasoning models' robustness is not just a matter of better reasoning algorithms; it requires ensuring that the input comprehension phase (which happens before or at the start of reasoning) is sufficiently reliable that the amplification works in the right direction.

Prior work contrast: The self-play and self-taught reasoning literature (Chen et al., 2024; Zelikman et al., 2022) treats reasoning as a mechanism for converting weak signals into strong performance — the model generates reasoning chains, evaluates their outcomes, and learns from successes and failures. But these frameworks assume that the model can evaluate outcomes correctly, and that failures are distributed across reasoning steps in a way that allows learning. This paper shows that when the initial planning step is wrong, the entire reasoning chain becomes systematically wrong, making outcome-based learning ineffective — there are no partially-correct reasoning chains to learn from when the architectural premise is flawed from the start.

The amplification hypothesis thus represents a fundamental conceptual shift in how to think about reasoning models: from "reasoning adds robustness" to "reasoning adds systematicity, which improves performance when comprehension is correct and degrades it when comprehension is flawed." This reframing has direct practical implications for deployment: on tasks where instruction comprehension is the primary challenge (as in the results-oriented coding tasks studied here), reasoning models may be less reliable than non-reasoning models — not because their reasoning is worse, but because their systematicity makes them consistently wrong instead of occasionally right.

Innovation 2: The Distinction Between Exploratory Reasoning and Specification-Following as a Critical Task Taxonomy

The paper introduces a task categorization that cuts across the standard benchmark taxonomy in a novel way. Prior work organizes reasoning benchmarks by domain: math (Zhang et al., 2023), science (Rein et al., 2023), competitive programming (Mirzayanov, 2009), software engineering (Jimenez et al., 2024), function calling (Yan et al., 2024). The paper argues that a more important distinction for understanding reasoning model behavior is whether the task is explorational or results-oriented.

• Explorational tasks (math, science, competitive programming): The problem statement describes a scenario, and the model must discover the correct solution through logical deduction, mathematical manipulation, or algorithmic insight. The reasoning chain is genuinely additive — it produces new conclusions from premises. The model benefits from exploring multiple solution paths because the correct approach is not obvious from the problem statement alone.

• Results-oriented tasks (WebApp1K coding): The correct implementation is completely determined by explicit test specifications. The model does not need to discover anything novel; it needs to read the specification, identify every requirement, and produce output that exactly satisfies those requirements. The challenge is comprehensive attention to detail, not creative problem-solving. There is no benefit to exploring alternative approaches because only one approach passes the tests.

This distinction is not a minor relabeling — it identifies why reasoning helps in some domains and hurts in others. In explorational tasks, the reasoning process adds value by generating intermediate conclusions that are not directly derivable from the input alone. A math problem with the statement "Prove that..." does not contain the proof within the statement — the model must construct the proof, and reasoning helps explore the construction space. But in results-oriented tasks, all the information needed to produce the correct output is already present in the input (the test cases). There is nothing to discover; there are only requirements to notice and satisfy. The reasoning process in this regime is not adding new information — it is reorganizing and prioritizing existing information. And if it deprioritizes or overlooks a requirement during this reorganization, the output fails.

The paper supports this taxonomy with its two central results. On explorational tasks (consistent with the broader o1 release results on math and science benchmarks), reasoning models achieve SOTA. On the paper's results-oriented tasks:

• Single-task: reasoning models excel because the specification is short enough that all requirements are typically captured in the reasoning chain, and the systematic elaboration produces correct implementations that non-reasoning models miss due to their less-thorough processing of implicit requirements.
• Duo-task: reasoning models degrade because longer inputs increase the probability that a requirement is missed or deprioritized early in the reasoning chain, and once missed, the systematic elaboration cannot recover it.

Prior work contrast: The chain-of-thought literature (Wang and Zhou, 2024; Lightman et al., 2023) has focused overwhelmingly on explorational tasks — math word problems, logical reasoning, scientific QA. Success on these tasks has been interpreted as evidence that reasoning models are generally more capable. This paper demonstrates that this interpretation overgeneralizes: the same reasoning mechanism that helps on math word problems can hurt on specification-following coding tasks. The task taxonomy explains why without invoking model-specific failures — it is a property of the task structure, not a bug in a particular reasoning implementation.

The significance of this innovation extends beyond the paper's empirical results. It provides a conceptual framework for reasoning model evaluation that future benchmarks should adopt. Rather than asking "how good are reasoning models at benchmark X?", researchers should ask "is benchmark X explorational or results-oriented, and how does the reasoning mechanism interact with that task structure?" This reframing could explain apparent contradictions in the literature — for example, why some studies find that chain-of-thought improves code generation (on explorational algorithmic problems) while others find it degrades performance (on specification-heavy implementation tasks). The task taxonomy predicts this divergence and provides a testable framework for future work.

Innovation 3: The Planning-Step Irreversibility Hypothesis as a Mechanistic Explanation for Catastrophic Failure

The paper's qualitative analysis of reasoning chain reenactments yields a specific mechanistic hypothesis about when reasoning failures become catastrophic rather than recoverable. The hypothesis, which I will call the planning-step irreversibility hypothesis, states: the scope of the first mistaken reasoning step determines whether the error can be corrected by subsequent steps. Errors at the implementation level (how to validate a form, how to display an error message) can potentially be caught and corrected by later reasoning steps that re-examine the original requirements. But errors at the planning level (how many components to build, what architecture to use) are irreversible because all subsequent reasoning steps derive their premises from the plan, not from the original input.

The paper develops this hypothesis through a comparison of two failure cases:

Implementation-level error (Section 2.3.1, ticketSubmission): The reasoning chain commits to frontend validation at an intermediate step ("Customer service improvement"). This is wrong but scoped — it affects how the form submission logic works, but does not change the overall component architecture (one component, one export, one form). The subsequent "Verifying form submission" step still reasons within the correct architectural framework and could, in principle, correct the validation logic if it re-examined the original test expectations. The paper notes this as a counterexample to the idea that reasoning always produces correct results, but also as a recoverable class of error — the reasoning was wrong, but not catastrophically so, and non-reasoning models sometimes make the same error.

Planning-level error (Section 3.2, duo-task raw format): The very first reasoning step commits to "creating components TaskA and TaskB" — a multi-component architecture. This error is unscoped — it determines the structure that all subsequent reasoning steps elaborate. Every later step is about how to implement two components, not whether two components are necessary. The paper explicitly argues: "Comparing to the mistakes made in Sec. 2.3.1, the mistake in the above step covers a larger scope. It is reasonable to argue that mistakes made in large-scoped steps are more fatal and harder to correct."

The evidence for irreversibility is the 0.0 pass@1 result on the raw format (Table 6) — every single o1 model attempt across all 1,000 duo tasks in the raw format fails, while non-reasoning models occasionally succeed (Claude 3.5 achieves 0.32). This is not a matter of reasoning models being "sometimes wrong" — they are uniformly wrong because the first-step planning error is deterministic given the prompt structure, and the reasoning process cannot recover from it.

Why this is a conceptual advance: The planning-step irreversibility hypothesis provides a mechanism for understanding when reasoning helps vs. hurts that is more precise than the broad explorational/results-oriented task taxonomy. It explains within the results-oriented category why some failures are partial (single-task edge cases where an implementation detail is wrong) and others are total (raw-format duo tasks where the architecture is wrong). It also explains the paper's counterexample in Section 2.3.1 — on the ticketSubmission problem, the o1 model sometimes commits to frontend validation (an implementation-level error) and fails, but the failure mode is not universal across all attempts because the planning-level reasoning (one component, one form) is still correct. The model gets the architecture right but the validation logic wrong, which is a recoverable class of error, and indeed non-reasoning models also exhibit this pattern.

Prior work contrast: The chain-of-thought literature has studied error propagation in reasoning chains (e.g., Lightman et al., 2023's process supervision approach, which aims to catch errors at intermediate steps), but it has not distinguished between scoped errors (which can be caught by later steps examining the original input) and unscoped errors (which define the framework within which later steps operate). This distinction is novel and has practical implications: to make reasoning models robust on results-oriented tasks, one should focus on preventing planning-level errors (perhaps by verifying the architectural plan against the specification before proceeding to implementation) rather than on catch-all error correction throughout the chain.

The significance of this hypothesis is both diagnostic and prescriptive. Diagnostically, it explains the paper's most striking result (0.0 vs. 0.32 on the raw format) in mechanistic terms that can be tested in future work — for example, by deliberately inducing planning-level vs. implementation-level errors and measuring recovery rates. Prescriptively, it suggests that reasoning model training should emphasize getting the first step right (architecture, planning, requirement identification) even at the cost of making later steps less thorough, because first-step errors are unrecoverable while later-step errors might self-correct.

Innovation 4: Catastrophic Failure on Syntactically Valid but Semantically Non-Obvious Inputs as a Reasoning Model Vulnerability

The 0.0 pass@1 result on the WebApp1K-Duo raw format (Table 6) is not merely a low score — it is a qualitative finding about a vulnerability class that distinguishes reasoning models from non-reasoning models. The paper identifies that the raw format exploits a specific combination of JavaScript module semantics — two default imports from the same module, which are semantically equivalent to one import but syntactically appear to demand two separate components — and that reasoning models are uniquely vulnerable to this syntactic misdirection.

What makes this an innovation rather than just a benchmark result is the diagnostic logic that isolates the vulnerability. The paper shows that:

1. The raw and normalized formats are semantically equivalent — both demand a single component with a single default export.
2. Non-reasoning models sometimes recognize this equivalence (Claude 3.5 achieves 0.32 on the raw format), indicating that the equivalence is discoverable from the prompt.
3. Reasoning models never recognize this equivalence (0.0 across all attempts), indicating that their reasoning process systematically interprets the syntactic surface structure (two imports → two components) without examining the underlying semantics.
4. Under the normalized format — where the syntactic structure matches the semantic requirement (one import → one component) — reasoning models recover to near-SOTA performance (0.652–0.667 vs. Claude 3.5's 0.679, Table 7).

This pattern — systematic failure on a syntactically misleading input that is semantically equivalent to a straightforward input — identifies a vulnerability class: reasoning models may be more susceptible than non-reasoning models to inputs where the surface structure suggests an incorrect interpretation that must be overridden by deeper semantic analysis. The paper's explanation, via the amplification hypothesis, is that reasoning models commit to the surface interpretation in their first planning step and then systematically elaborate it, while non-reasoning models sometimes "luck into" the correct interpretation because their less-systematic processing allows occasional bypass of the misleading surface cues.

Prior work contrast: The literature on adversarial examples and robustness typically focuses on inputs that are designed to fool models through imperceptible perturbations or distribution shift. The WebApp1K-Duo raw format is different: it is a syntactically valid, semantically equivalent alternative representation of the same task that happens to be less common in training data. It is not an adversarial example — it is a legitimate (if unusual) way to write a test file that a human developer would recognize as equivalent to the normalized format after a moment's thought. The fact that reasoning models catastrophically fail on this legitimate variant, while non-reasoning models handle it some of the time, reveals a brittleness in how reasoning models process input structure.

The significance of this finding extends beyond JavaScript module syntax. It suggests a general principle: reasoning models are vulnerable to any input where the obvious first interpretation is wrong and must be overridden by deeper analysis. In the raw format, the obvious interpretation (two imports → two components) is wrong, and the correct interpretation (both imports reference the same default export → one component) requires understanding that default imports are name-agnostic and that a module can have only one default export. In other domains, analogous patterns might arise: legal language where common-sense interpretation conflicts with technical definitions, mathematical notation where conventional reading differs from the formal semantics, or API documentation where example usage patterns conflict with the specification's fine print. This vulnerability class is not specific to JavaScript or coding tasks — it is a general property of any domain where surface patterns can mislead.

The innovation here is establishing that this vulnerability is amplified by the reasoning mechanism itself, not mitigated by it. This runs counter to the intuitive expectation that a model that "thinks more carefully" should be less susceptible to surface-level misinterpretation. The paper's evidence suggests the opposite: systematic thinking amplifies the initial misinterpretation, making the failure universal rather than probabilistic.

5. Experimental Analysis

Evaluation Methodology

• Dataset. The paper uses two benchmarks: WebApp1K (Cui, 2024b), consisting of 1,000 single-task React coding problems organized into 20 categories with 50 tasks each, and WebApp1K-Duo, a newly constructed benchmark of 1,000 duo-task problems formed by randomly pairing two atomic tasks from the same WebApp1K category, presented in two format variants (raw and normalized). All problems require implementing React components that pass Jest test cases involving mocked API interactions and user event simulations, with single-task problems having exactly 2 test cases (one success, one failure) and duo-task problems having exactly 4 test cases (success and failure for each of the two constituent tasks).

• Base models. Six frontier models are evaluated: o1-preview and o1-mini (OpenAI's reasoning models, the primary subjects of study), gpt-4o-2024-08-06 / chatgpt-4o-latest (OpenAI's non-reasoning flagship), claude-3.5-sonnet (Anthropic), deepseek-v2.5, and mistral-large-2. The reasoning models are chosen as the evaluation targets; the non-reasoning models serve as comparison points to isolate the effect of the reasoning mechanism while controlling for organizational pretraining differences (within-OpenAI: o1 vs. gpt-4o) and for frontier capability broadly (cross-organization: o1 vs. Claude 3.5, DeepSeek, Mistral).

• Metrics. The sole metric is pass@1: the fraction of the 1,000 benchmark problems for which the model's single generated implementation passes all test cases on the first attempt. Evaluation uses the standard Jest testing framework with React Testing Library; a problem is scored as passed if and only if every test case in the problem's test file executes without failure (binary grading, no partial credit). The paper notes that for o1 models, only pass@1 is reported "due to budget constraints," while non-reasoning model pass@1 results are drawn from prior work (Cui, 2024b) or independent evaluation under the same protocol.

• Baselines. The non-reasoning frontier models (gpt-4o, claude-3.5-sonnet, deepseek-v2.5, mistral-large-2) serve as comparative baselines rather than methodological baselines. There is no ablation of the reasoning mechanism itself (e.g., comparing o1 with chain-of-thought disabled against o1 with reasoning enabled) because the paper studies externally accessible models whose internal reasoning configuration cannot be modified. The comparison is therefore between model families, not between configurations of the same model.

• Generation budget / compute accounting. All models receive one generation attempt per problem (pass@1), so the generation budget is uniform across models at 1 generation per problem × 1,000 problems = 1,000 generations per benchmark. The paper does not measure or compare FLOPs, token counts, inference time, or API costs across models — compute accounting is limited to counting generation attempts, not computational resources consumed. This is a significant simplification: the reasoning models' internal chain-of-thought tokens consume additional compute that is not accounted for in the pass@1 comparison, meaning the o1 models may be using substantially more inference-time computation than the non-reasoning models to produce their single generation.

• Cross-validation / statistical protocol. No cross-validation or statistical significance testing is reported. All results are simple pass@1 fractions over the full 1,000-problem benchmark. There is no error bar, confidence interval, or variance estimate for any reported score, making it impossible to assess whether differences between models (e.g., o1-mini's 0.667 vs. Claude 3.5's 0.679 on WebApp1K-Duo normalized) are statistically meaningful or within sampling noise. The paper also does not report how many problems within each benchmark were used for qualitative analysis vs. held out, leaving open the possibility that the problems selected for deep-dive analysis (Sections 2.2, 2.3, 3.2, 3.3) were chosen post-hoc based on knowing the outcomes.

Main Quantitative Results

Single-Task Benchmark: Reasoning Models Set New SOTA

Headline result. On WebApp1K (Table 2), o1-preview achieves a pass@1 of 0.952 and o1-mini achieves 0.939, representing a 7% absolute improvement over the prior SOTA held by gpt-4o-2024-08-06 at 0.885. The full ranking:

The gap between the two reasoning models (0.013) is smaller than the gap between the lower reasoning model and the best non-reasoning model (0.939 - 0.885 = 0.054), indicating that the reasoning mechanism itself, rather than model scale (o1-preview vs. o1-mini), is the primary driver of the improvement.

Novel capability. The paper reports that "the two o1 models unlock a total of 16 challenges never solved by previous non-reasoning models" (Section 2.1). This is not quantified as a fraction (16/1000 = 1.6% of the benchmark), so most of the 7% SOTA improvement comes from solving problems that non-reasoning models sometimes solve, but more consistently — not from unlocking entirely novel problem classes. The 16 "never solved" problems represent a small but meaningful capability expansion at the hardest end of the single-task distribution.

Duo-Task Raw Format: Catastrophic Failure of Reasoning Models

Headline result. On WebApp1K-Duo raw format (Table 6), both o1-preview and o1-mini achieve pass@1 of 0.0 — they fail every single one of the 1,000 duo tasks. In contrast, non-reasoning models achieve non-zero, albeit low, pass rates:

The 0.0 result is the paper's most striking quantitative finding. It is not merely a low score — it is a qualitative collapse: reasoning models, which achieved the highest scores on single tasks, cannot solve a single duo task in the raw format, while Claude 3.5 solves nearly one-third of them (0.32). This represents a complete inversion of the single-task ranking: Claude 3.5 goes from 4th place on single-task (0.881) to 1st place on duo-task raw format (0.32), while o1 models go from 1st and 2nd (0.952, 0.939) to tied-for-last (0.0, 0.0).

Interpretation. The raw format's double-default-import pattern (Section 3.2) triggers a planning-level error in the reasoning models' first reasoning step (committing to a multi-component architecture) that is unrecoverable. Non-reasoning models lack this systematic planning step, so they occasionally avoid the misinterpretation.

Duo-Task Normalized Format: Reasoning Models Recover But Do Not Lead

Headline result. On WebApp1K-Duo normalized format (Table 7), all models improve substantially over their raw-format scores, and the o1 models recover to competitive but not leading performance:

Claude 3.5 achieves SOTA at 0.679, with o1-mini trailing by 0.012 and o1-preview by 0.027. The gap is small (within 2-3 percentage points) but consistent: both reasoning models score below Claude 3.5, reversing the single-task ordering where both o1 models scored above Claude 3.5 (0.952 and 0.939 vs. 0.881).

The format intervention effect. Comparing raw vs. normalized scores for the same models:

• Claude 3.5: 0.32 → 0.679 (+0.359) — substantial improvement, but raw-format was already solvable some of the time.
• chatgpt-4o-latest: 0.026 → 0.531 (+0.505) — massive improvement, suggesting the raw format was primarily a syntactic parsing barrier.
• o1-mini: 0.0 → 0.667 (+0.667) — complete recovery from catastrophic failure to near-SOTA.
• o1-preview: 0.0 → 0.652 (+0.652) — same pattern.

The o1 models show the largest absolute gain from the format normalization because they start from zero. But they do not reach the top of the leaderboard, indicating that even when the syntactic misdirection is removed, duo tasks still pose challenges (longer input, more requirements) that reasoning models handle slightly less well than Claude 3.5 — consistent with the amplification hypothesis: if any requirement is missed from the longer input, the reasoning process systematically fails to include it.

Performance Degradation from Single-Task to Duo-Task

Although the paper does not tabulate this comparison directly, we can compute the degradation by comparing WebApp1K single-task pass@1 (Table 2) against WebApp1K-Duo normalized pass@1 (Table 7) for each model:

• o1-preview: 0.952 → 0.652 (−0.300)
• o1-mini: 0.939 → 0.667 (−0.272)
• gpt-4o / chatgpt-4o-latest: 0.885 → 0.531 (−0.354)
• claude-3.5-sonnet: 0.881 → 0.679 (−0.202)
• deepseek-v2.5: 0.834 → 0.49 (−0.344)
• mistral-large-2: 0.780 → 0.449 (−0.331)

Claude 3.5 shows the smallest degradation (−0.202), while the o1 models show intermediate degradation (−0.272 to −0.300) — larger than Claude 3.5's but smaller than the other non-reasoning models. This suggests that while o1 models are more robust to the complexity increase than most non-reasoning models, Claude 3.5 is uniquely robust, retaining more of its single-task capability when moving to duo tasks. This pattern is consistent with the paper's claim that instruction adherence is the key bottleneck: Claude 3.5 may have stronger base-model instruction-following capability, allowing it to handle longer specifications with fewer missed requirements.

Qualitative Failure Analysis: Ignored Expectation Under Normalized Format

The paper provides one concrete example (Section 3.3) of where o1-preview fails on the normalized format despite the syntactic issue being resolved. On the addComment_retrieveAllBlogPosts problem (Table 10), o1-preview passes three of four test cases but fails the fourth — the test expecting Internal Server Error to be displayed when the blog post retrieval API returns a 500 status. The output code "neither attempt to catch the 500 error nor print out the Internal Server Error string." The ChatGPT reenactment shows a reasoning chain that never specifically mentions the need to catch internal server errors:

Crafting the component → Laying out the requirements →
Importing dependencies → Breaking down the code →
Setting up the app → Testing a post functionality →
Testing API integration

The paper notes that o1-preview's "inherent coding ability is solid, because it solves the retrieveAllBlogPosts problem when evaluated under the single-task benchmark" — meaning the model is capable of implementing this exact error-handling logic when the retrieveAllBlogPosts task is presented in isolation. The fact that it fails when the same task is paired with addComment is attributed to "failure to pick up the expectation from input tokens, possibly due to length constraint" (Section 3.3). This provides direct evidence for the paper's central hypothesis: the degradation is not due to the model being unable to implement the required logic, but due to the requirement being overlooked when the input is longer and contains multiple tasks' worth of test specifications.

No quantitative frequency for this failure mode is reported — the paper provides one example but does not state how many of the o1 models' failures on the normalized duo benchmark follow this "ignored expectation" pattern versus other error types.

Ablation Studies and Robustness Checks

The paper does not contain formal ablation studies in the traditional sense — no systematic manipulation of model components, prompt structure, or evaluation parameters with controlled comparisons. However, several implicit ablations can be extracted from the experimental design:

Single-task vs. duo-task comparison: This functions as an input-length and task-count ablation. Moving from 2 test cases to 4 test cases while preserving task semantics allows measuring the effect of specification length and implementation scope on model performance. The result is that all models degrade, but Claude 3.5 degrades least (−0.202 pass@1) and o1 models show moderate degradation (−0.272 to −0.300). No intermediate complexity levels (3 test cases, 6 test cases) are tested, so the relationship between input length and degradation cannot be characterized as linear, sublinear, or having a threshold.

Raw format vs. normalized format: This functions as a prompt structure ablation for duo tasks. The two formats are semantically equivalent but syntactically different, allowing isolation of the effect of syntactic surface structure on model comprehension. The result is dramatic for reasoning models (0.0 → ~0.66) and substantial for non-reasoning models (~0.02–0.32 → ~0.45–0.68), confirming that prompt syntax significantly impacts performance independent of task semantics. No intermediate format variants (e.g., partial normalization) are tested.

Reasoning vs. non-reasoning within the same organization: Comparing o1-preview and o1-mini against gpt-4o / chatgpt-4o-latest provides a within-organization control for the reasoning mechanism, since all models are developed by OpenAI with likely overlapping pretraining data and infrastructure. On single-task, reasoning models outperform (+0.054 to +0.067 over gpt-4o). On duo-task normalized, o1-mini outperforms chatgpt-4o-latest (+0.136) while o1-preview similarly outperforms (+0.121). On duo-task raw format, chatgpt-4o-latest scores 0.026 while both o1 models score 0.0 — the one regime where the reasoning mechanism underperforms the non-reasoning counterpart from the same organization.

Single-task isolation as a capability check: The paper verifies that o1-preview solves retrieveAllBlogPosts correctly when it appears as a single task (Section 3.3), establishing that the failure on the duo version is not due to inability to implement the required error-handling logic, but due to the requirement being missed when embedded in a longer specification. This is not a controlled ablation but a diagnostic check that supports the paper's causal claim.

Model scale within reasoning family: Comparing o1-preview against o1-mini provides a rough scale ablation for reasoning models. On single-task, o1-preview leads by 0.013 (0.952 vs. 0.939). On duo-task normalized, o1-mini leads by 0.015 (0.667 vs. 0.652) — a reversal, though the differences are small in both directions (around 1.3–1.5 percentage points). On duo-task raw format, both score 0.0. This suggests that within the reasoning model family, scale does not substantially affect performance on these benchmarks — the reasoning mechanism's behavior (amplification) dominates any scale effects. However, with only two model sizes and no information about their parameter counts, this inference is tentative.

Missing ablations that would have strengthened the paper:

• Pass@k for reasoning models: Measuring pass@k (e.g., pass@10, pass@100) would reveal whether reasoning models' failures are consistent across multiple attempts or whether resampling can recover from first-step planning errors. If pass@k rises substantially with k on the raw format, it would suggest that the planning error is probabilistic rather than deterministic per model, contradicting the strong version of the amplification hypothesis.
• Prompt structure variations: Testing whether adding explicit hints (e.g., "Note: only one default export is allowed per module") to the prompt eliminates the raw-format failure would directly test whether the failure is due to the reasoning mechanism's inherent planning behavior or due to the prompt not providing sufficient disambiguating information.
• Task ordering effects: For duo tasks, the test cases for Task A appear before Task B in the prompt. Testing whether reversing the order changes which expectations are missed would probe whether the "ignored expectation" failure is due to recency effects, attention decay over long contexts, or other positional biases.
• Component-level evaluation: Measuring whether models more frequently fail expectations from the second constituent task than the first would reveal whether the degradation is symmetric or biased toward later parts of the prompt.
• Difficulty-controlled comparisons: The paper does not bin single-task problems by difficulty (e.g., by non-reasoning model pass rates) and then examine whether o1 models' advantage is concentrated in easy, medium, or hard subsets. Such binning would reveal whether the reasoning mechanism primarily helps on tasks where non-reasoning models already have some success (amplifying partial capability) or on tasks where non-reasoning models consistently fail (unlocking novel capability).

Critical Assessment

Claim: "Reasoning models deliver SOTA results for WebApp1K, a single-task benchmark"

This claim is directly supported by Table 2, where o1-preview (0.952) and o1-mini (0.939) surpass all non-reasoning models, with the best non-reasoning model (gpt-4o-2024-08-06) at 0.885. However, the following qualifications apply:

• Pass@1 only: The paper reports only pass@1 for o1 models, citing budget constraints. If non-reasoning models were also evaluated only at pass@1 (which they appear to be, based on the prior WebApp1K paper's methodology), then the comparison is fair for pass@1. But it is unknown whether non-reasoning models with pass@k (k > 1) might match or exceed o1 pass@1, which is the relevant comparison for deployment scenarios where retries are feasible. The claim "SOTA" is therefore specific to the pass@1 metric on this benchmark.
• The 7% lift may overstate practical improvement: The prior SOTA (0.885) already solves nearly 9 in 10 problems. The remaining unsolved ~12% of problems are presumably the hardest, and o1 models solve roughly half of those (lifting to 0.952 solves an additional ~67 problems). This is meaningful but incremental — most of the benchmark was already solvable.
• No difficulty breakdown: Without binning by difficulty, we cannot distinguish whether o1 models excel at easy problems (where amplification of correct comprehension helps), hard problems (where the reasoning mechanism discovers solutions non-reasoning models miss), or both. The qualitative examples in Section 2.2–2.3 focus on problems where non-reasoning models systematically fail, suggesting o1's advantage is on problems with atypical or implicit requirements — but no quantitative evidence is provided for this claim across the full benchmark.

Claim: "The o1 model performances decline significantly [on WebApp1K-Duo], falling behind Claude 3.5"

This claim is supported by Table 7 (normalized format), where Claude 3.5 (0.679) leads o1-mini (0.667) and o1-preview (0.652), and overwhelmingly by Table 6 (raw format), where o1 models score 0.0 while Claude 3.5 scores 0.32. However:

• "Decline significantly" is relative to what baseline? The single-task benchmark and the duo-task benchmark are different sets of problems — they do not share tasks, so it is not possible to directly measure "decline" on identical problems. The paper constructs duo tasks by pairing single tasks, but the pairing itself changes the problem (the model must now implement both tasks in one component), so the duo task is not the same problem as either constituent single task. The "decline" is inferred from comparing pass@1 rates across different problem distributions, not from measuring the same problems at different complexity levels. A within-problem comparison (e.g., measuring o1-preview's pass rate on the 1,000 individual constituent tasks that make up the 1,000 duo tasks, and comparing to the duo-task pass rate on those same pairings) would provide a direct measure of degradation but is not reported.
• The raw-format 0.0 may be specific to JavaScript module semantics: The catastrophic failure on the raw format is triggered by a very specific syntactic pattern (double default import from the same module). Other coding tasks with different syntactic properties might not exhibit this vulnerability. The paper does not test whether analogous syntactic misdirections in other languages or frameworks produce similar reasoning-model collapses, limiting the generalizability of the 0.0 finding.
• Claude 3.5's lead on the normalized format is small and potentially not statistically significant: With 1,000 problems and no reported confidence intervals, a difference of 0.679 vs. 0.667 (12 problems out of 1,000) could be within sampling noise. The paper does not provide the information needed to assess this.

Claim: "They consistently fail when confronted with atypical yet correct test cases, a trap non-reasoning models occasionally avoid"

This claim has two parts: (1) reasoning models consistently fail on atypical test cases, and (2) non-reasoning models occasionally avoid these failures. The evidence for each:

• Consistency of reasoning model failure: Supported for the duo-task raw format by the 0.0 pass@1 (Table 6) — 1,000 out of 1,000 failures is maximally consistent. Supported for the ticketSubmission frontend-vs-backend validation trap by the counterexample discussion in Section 2.3.1, where the paper notes o1-preview sometimes falls into the same trap as non-reasoning models (committing to frontend validation). However, the single-task pass@1 of 0.952 for o1-preview means that across the full benchmark, o1-preview avoids traps on 95.2% of problems — so "consistently fail" overstates the frequency. The claim applies to specific subcategories of atypical cases, not to the benchmark as a whole.
• Non-reasoning models occasionally avoiding failures: Supported for the raw format by Claude 3.5's 0.32 (Table 6) and for the frontend-validation trap by the earlier non-reasoning model results cited from Cui (2024a, 2024b). But the paper does not quantify "occasionally" — we do not know what fraction of atypical test cases non-reasoning models avoid, or whether there are atypical cases where reasoning models succeed and non-reasoning models consistently fail (the success case in Section 2.2, where non-reasoning models "forget to explicitly attach the Edit string," is one such example, but its frequency across the benchmark is not quantified).

The stronger and better-supported version of this claim would be: On problems where the correct implementation requires overriding a strong pretraining prior or noticing an implicit requirement from test selectors, reasoning models succeed at higher rates than non-reasoning models when they notice the requirement (because amplification ensures systematic implementation), but fail universally when they miss the requirement in early reasoning (because the reasoning chain cannot recover overlooked information). The net effect depends on the base model's instruction-following reliability — high on short inputs, degrading on longer inputs.

Claim: "The performance variability is due to instruction comprehension... the reasoning mechanism boosts performance when all expectations are captured, meanwhile exacerbates errors when key expectations are missed"

This is the paper's central mechanistic hypothesis. The evidence is:

Evidence for "boosts when expectations are captured":

• Qualitative analysis of the postEditing success case (Section 2.2) shows the reasoning chain explicitly processing the implicit Edit string requirement and producing correct code.
• The single-task benchmark result (0.952 for o1-preview, Table 2) is consistent with reasoning amplifying correct comprehension — but does not directly demonstrate it, since we cannot observe o1-preview's actual reasoning chain and cannot verify that the high pass@1 is caused by amplification rather than by other differences between o1 and non-reasoning models (e.g., better pretraining, more parameters, different training objectives).

Evidence for "exacerbates errors when expectations are missed":

• Qualitative analysis of the ticketSubmission counterexample (Section 2.3.1) shows a reasoning chain committing to the wrong validation strategy and subsequent steps reinforcing rather than correcting it.
• The duo-task raw format 0.0 result (Table 6) is the strongest evidence — the planning error is triggered by the prompt syntax, and the reasoning mechanism amplifies it into universal failure.
• The duo-task normalized case study (Section 3.3) shows the retrieveAllBlogPosts error-handling requirement being overlooked in a longer input.

Limitations of the evidence:

• All reasoning chain evidence comes from ChatGPT reenactments, not from the actual o1 model reasoning (which is not accessible via the API). ChatGPT is a non-reasoning model and may produce different reasoning patterns than o1. The reenactments are illustrative but do not constitute direct evidence about o1's internal processing.
• The sample size for qualitative analysis is extremely small: three specific problems discussed in detail (postEditing success, ticketSubmission counterexample, addComment_retrieveAllBlogPosts failure), plus the raw-format first-step analysis. These are selected by the authors to illustrate their hypothesis, raising the possibility of cherry-picking. We do not know whether other success or failure cases would show different patterns, or how representative these examples are of the broader error distribution.
• No quantitative correlation is shown between reasoning chain properties (e.g., whether early steps mention all test expectations) and task success. Such an analysis would require access to o1's actual reasoning traces at scale, which is not possible with the current API, but the paper does not acknowledge this as a fundamental limitation of its evidential basis.

What would strengthen the claim:

• A systematic comparison of o1-preview's pass@1 on single-task problems binned by whether the task contains an "atypical" requirement (as judged by human annotators or by non-reasoning model pass rates), versus problems with only "typical" requirements. If amplification helps specifically on atypical problems, we would expect a larger o1 vs. non-reasoning gap on the atypical subset.
• Evaluation of an intermediate model (e.g., o1 with reasoning depth limited, if such control were possible) to establish a dose-response relationship between reasoning intensity and the amplification pattern.
• A larger-scale analysis of failure modes on the duo-task normalized benchmark, categorizing whether each o1 failure is due to a missed requirement (amplification of omission), a misinterpreted requirement (amplification of error), or an implementation error despite correct requirement capture (not amplification). The paper's hypothesis predicts that the first two categories should dominate. Currently, only one example of each pattern is provided.

Overall Assessment: What Has Been Demonstrated vs. What Is Hypothesized

The paper convincingly demonstrates a set of empirical facts:

1. o1 models achieve higher pass@1 than non-reasoning models on WebApp1K single tasks (Table 2). This is a direct measurement and is not in dispute.
2. o1 models get 0.0 pass@1 on the WebApp1K-Duo raw format, while non-reasoning models achieve non-zero pass rates (Table 6). This is also a direct measurement.
3. The raw format's double-default-import pattern is semantically equivalent to the normalized format (Table 8), yet produces dramatically different outcomes for reasoning models (0.0 vs. 0.652–0.667). This establishes that o1 models are sensitive to prompt syntax in ways that affect their architectural planning.
4. o1 models trail Claude 3.5 on the normalized duo-task format (Table 7), despite leading on single tasks — the performance ordering across models changes between benchmarks.

The paper hypothesizes a mechanistic explanation (the amplification hypothesis) that is consistent with these facts. The hypothesis has genuine explanatory power: it accounts for why o1 models excel on single tasks (short inputs → comprehensive requirement capture → amplification produces thorough implementations), why they catastrophically fail on the raw format (syntactic misdirection → first-step planning error → amplification makes error universal), and why they trail on the normalized format (longer inputs → occasionally missed requirements → amplification cannot recover missing information). The qualitative reasoning chain reenactments provide illustrative support but do not constitute gold-standard evidence because they come from a different model (ChatGPT) and a tiny, author-selected sample.

The key gap is that the paper does not establish a causal link between the hypothesized mechanism and the observed outcomes. The amplification hypothesis makes testable predictions — e.g., that on duo tasks where both constituent single tasks were individually solvable by o1-preview, the duo failure rate should be explained primarily by missed requirements rather than by implementation interference between tasks — but these predictions are not tested. The paper's contribution is best characterized as: (1) documenting a striking and previously unreported performance pattern in reasoning models on specification-following coding tasks, and (2) proposing a plausible mechanistic hypothesis grounded in qualitative case analysis, which future work with access to actual reasoning traces (or controlled reasoning depth ablation) could test.

Genuine Weaknesses

• No confidence intervals or statistical tests: All results are point estimates (pass@1 over 1,000 problems). With no variance information, differences of 1–3 percentage points between models (e.g., 0.667 vs. 0.679) are uninterpretable. This is a basic omission for empirical work.
• Pass@1 only, no pass@k: The budget constraint is understandable but limits the conclusions. If o1 models' failures on the raw format are deterministic (as the amplification hypothesis implies), pass@k would remain 0.0 even at high k. If failures are probabilistic, pass@k would rise — which would contradict the strong amplification hypothesis. This is a directly informative experiment that is not run.
• Tiny qualitative sample: Three problems analyzed in detail, selected by authors who know the outcomes. This is insufficient to establish representativeness of the failure patterns, and the use of ChatGPT reenactments rather than actual o1 reasoning traces undermines the mechanistic specificity of the claims.
• No difficulty or error categorization across the full benchmark: Without systematically classifying the 1,000 duo-task problems (or at minimum the failures) by error type, the paper cannot quantify what fraction of o1 failures are due to the hypothesized amplification-of-omission vs. other causes. The single addComment_retrieveAllBlogPosts example (Section 3.3) is compelling but anecdotal.
• No within-problem degradation measurement: The "decline" from single-task to duo-task is inferred across different problem distributions. A direct within-problem comparison (measuring o1-preview's accuracy on individual constituent tasks and on their paired duo version) would provide a cleaner estimate of the cost of combining tasks, but is not reported.
• Prompt sensitivity not systematically explored: The raw-vs-normalized comparison establishes that prompt syntax matters dramatically, but only one syntactic property (export name normalization) is varied. We do not know whether other prompt variations — e.g., explicit reminders about single default exports, reordering test cases, adding comments — would mitigate or exacerbate the failure. This limits the practical implications: we know one thing that causes catastrophic failure but not how to prevent it short of avoiding that specific syntactic pattern.
• No information about o1's actual compute usage: The paper notes that o1 models use internal reasoning tokens but does not account for this in any cost or compute comparison. If o1-preview uses substantially more inference-time compute than Claude 3.5 to produce its single generation, then the pass@1 comparison is not compute-matched, and the fact that o1-mini trails Claude 3.5 on the normalized format despite likely using more compute is an even stronger negative result than the raw numbers suggest. Conversely, if o1 models are compute-competitive, that would strengthen the positive single-task result. The missing compute accounting makes both the positive and negative results harder to interpret in practical terms.

6. Limitations and Trade-offs

6.1 Reasoning Chain Analysis Relies on Proxy Reenactments, Not Direct Observation

The assumption or constraint. The paper's central mechanistic claims — the amplification hypothesis, the planning-step irreversibility hypothesis, the claim that early-chain errors preempt later correction — depend entirely on observing and interpreting the reasoning models' internal chain-of-thought. However, "reasoning tokens are invisible in OpenAI API" (Section 1), so the authors must use a proxy: "we share reasoning steps obtained from ChatGPT reenactment, i.e. feeding the identical prompt to ChatGPT" (Section 1). This means every reasoning chain presented as evidence of o1 behavior — the postEditing success chain in Section 2.2, the ticketSubmission counterexample chain in Section 2.3.1, the raw-format first-step error in Section 3.2, the addComment_retrieveAllBlogPosts omission chain in Section 3.3 — was generated by ChatGPT, a non-reasoning model, not by o1-preview or o1-mini.

The consequence. We cannot know whether the actual o1 reasoning chains exhibit the same structure, the same error patterns, or the same step-by-step planning behavior as the ChatGPT reenactments. Several specific concerns arise:

• ChatGPT is a non-reasoning model (presumably gpt-4o or chatgpt-4o-latest based on the results tables). Its chain-of-thought generation process may differ systematically from o1's reasoning mechanism, which likely involves reinforcement learning for chain-of-thought optimization (as suggested by the related work discussion of self-play and self-taught reasoning in Section 4). The reenactment captures what a non-reasoning model "thinks" when asked to produce code for the same prompt, not what the reasoning-optimized model thinks.
• The reenactments are post-hoc and outcome-informed: the authors selected which problems to analyze after knowing whether o1 succeeded or failed. This introduces potential confirmation bias — reasoning chains that match the authors' hypothesis may have been selected, while reasoning chains that contradict it may not have been examined or reported.
• Only four specific problems receive detailed reasoning chain analysis across the entire paper (two in Section 2, one counterexample also in Section 2, one in Section 3). With 3,000 total problems evaluated (1,000 single-task + 1,000 duo-task raw + 1,000 duo-task normalized), this represents a 0.13% qualitative sample, selected by authors who know the outcomes.
• The reenactments cannot verify a key prediction of the amplification hypothesis: that the o1 model's actual first reasoning step commits to a multi-component architecture on the raw format, and that this commitment is universal across all 1,000 raw-format failures. We have one illustrative ChatGPT reenactment of this error pattern, but no evidence that o1-preview and o1-mini actually follow this reasoning pattern internally.

What evidence exists in the paper. The ChatGPT reenactments are presented in Sections 2.2, 2.3, 2.3.1, 3.2, and 3.3. The paper reports the sequence of reasoning step titles (using → notation) and, for critical steps, quotes the detailed reasoning text. For example, Section 3.2 reports the first planning step from a ChatGPT reenactment: "To progress, the key task is creating components TaskA and TaskB in TaskA_B.js to ensure all tests are successfully passed." However, nowhere does the paper directly observe or quote actual o1 reasoning traces — because the API does not expose them.

Mitigation status. The paper does not attempt to mitigate this limitation. It does not:

• Acknowledge the gap between ChatGPT reenactments and actual o1 reasoning as a threat to the validity of its mechanistic claims.
• Discuss whether ChatGPT's reasoning patterns are expected to generalize to o1's reasoning mechanism, or note known differences between the model families.
• Attempt systematic validation of the reenactment approach (e.g., comparing ChatGPT reenactments against visible reasoning from other reasoning models, or against o1's behavior on problems where the reasoning can be inferred from output structure).
• Expand the qualitative sample beyond four problems.
• Flag this as a direction for future work once o1 reasoning traces become accessible.

This limitation affects the paper's central intellectual contribution — the amplification hypothesis — more than it affects the empirical results. The quantitative findings (pass@1 rates in Tables 2, 6, 7) stand independent of the reasoning chain analysis. But the mechanistic explanation for why those patterns occur rests on evidence that is, at present, indirect and dramatically undersampled relative to the scope of the claims.

6.2 Difficulty Estimation Cost Is Implicit and Unaccounted for in the Performance Numbers

The assumption or constraint. The paper treats the single-task benchmark as the baseline and the duo-task benchmark as the complex condition, but it does not address how a practitioner deploying these models would know, in advance, whether a given coding task falls into the "easy" regime (where o1 reasoning models excel) or the "hard" regime (where they fail catastrophically on certain prompt formats or trail non-reasoning models on longer specifications). The entire diagnostic framework — "use reasoning models on short, single-task specifications; consider non-reasoning models for longer, multi-task specifications or when prompt syntax may be misleading" — requires estimating task difficulty and prompt complexity before selecting a model.

The consequence. A practitioner reading this paper learns that o1 models achieve 0.952 on single tasks but 0.0 on duo tasks with certain prompt formats, but receives no guidance on how to predict which regime a new, unseen task falls into without running the model and observing failure. The difficulty is not inherent in the task semantics — it depends on:

• Input length: The duo-task normalized format has 4 test cases instead of 2, but many real-world specifications are longer still. At what point does the amplification of missed expectations become the dominant failure mode?
• Syntactic properties: The raw format triggers catastrophic failure, the normalized format does not. But other syntactic patterns (unconventional import structures in other languages, unusual test organization patterns, atypical framework usage) may create similar "syntactic misdirection" vulnerabilities that are not identified by this paper because only one syntactic variable (export name normalization in JavaScript) was manipulated.
• Requirement atypicality: The paper identifies cases where pretraining priors (frontend validation best practices) conflict with test expectations (backend validation). But which requirements are "atypical" depends on the model's training distribution, which is unknown to practitioners and varies across models.

Without a predictive model of task difficulty, the practitioner's only option is trial and error: try o1, see if it fails, try Claude 3.5 if it does. This consumes inference budget — potentially substantial given that o1 models use internal reasoning tokens whose cost is not accounted for in this paper — and introduces latency into the development workflow.

What evidence exists in the paper. The difficulty estimation problem is implicit throughout the paper but never directly addressed. The evidence for its importance comes from the dramatic performance variation across conditions: 0.952 on single-task (Table 2) vs. 0.0 on duo-task raw format (Table 6) vs. 0.652–0.667 on duo-task normalized format (Table 7). These are outcomes observed after evaluation — not predictions that could be made before running the model. The paper does not:

• Propose or evaluate any method for predicting, from a prompt alone, whether a reasoning model will succeed or fail.
• Analyze what properties of the duo-task raw format (beyond "it has double default imports") cause the catastrophic failure, in a way that would let practitioners scan their own prompts for similar risks.
• Measure how the failure probability scales with prompt length or task count between the extremes of 2 test cases (single-task) and 4 test cases (duo-task).
• Discuss the practical workflow implications of the fact that the same model can be either the best or the worst choice depending on subtle prompt properties that are not obvious without prior evaluation.

Mitigation status. Not addressed. The paper concludes by recommending "meticulous adherence to instructions" via "top-notch base model and SFT" (Section 5), which is a direction for model developers, not a practical strategy for model users facing an existing deployment decision. The difficulty estimation problem — knowing when to deploy a reasoning model vs. a non-reasoning model for a given coding task — is left entirely to future work.

6.3 Single Benchmark Domain with One Framework and One Programming Language

The assumption or constraint. All experiments use the WebApp1K benchmark family, which is restricted to React web application components written in JavaScript and tested with Jest and React Testing Library. The paper explicitly notes these characteristics (Section 1): "The problem is less explorational and more results-oriented than other benchmarks. The specific instructions are laid out in the form of test setup and expectations. No external knowledge is required to complete the task, since React is a prominent framework with sufficient code circulating on Internet for a decade." While the paper positions this specificity as a feature (controlled setting, no knowledge confounds), it also means that every finding — the amplification pattern, the sensitivity to import syntax, the frontend-vs-backend validation trap — is observed in exactly one programming language, one framework, one testing paradigm, and one task format (test-driven component implementation).

The consequence. Practitioners working in different technical stacks cannot assume the paper's findings transfer. Several aspects of the WebApp1K setting may be unusually favorable or unfavorable to reasoning models in ways that would not generalize:

• JavaScript module syntax: The raw-format failure (0.0 pass@1 for o1 models, Section 3.2) depends specifically on the semantics of default vs. named exports in ES6 modules, and on the fact that default imports are name-agnostic. In languages without this syntactic property (Python, Java, Go, Rust), the exact failure mode cannot occur. However, analogous syntactic "misdirection" patterns may exist — e.g., Python's import X from Y vs. from Y import X vs. from Y import X as Z — and the paper provides no framework for identifying which syntactic patterns in other languages might trigger similar planning-step errors in reasoning models.
• Test-driven specification: WebApp1K provides the entire task specification as executable test code. This is a specific interaction paradigm — the model reads tests, infers requirements, and writes implementation code. Many real-world coding tasks involve natural language specifications, partial specifications, or iterative refinement through conversation, none of which are tested here. The paper's findings about instruction comprehension (how models capture expectations from the prompt) may not transfer to settings where expectations are communicated through dialogue rather than through formal test assertions.
• Component scope: WebApp1K tasks produce 40–50 line React components implementing single atomic actions. Real-world software engineering involves much larger modules, multi-file architectures, existing codebases that must be modified rather than written from scratch, and complex dependencies between components. The duo-task benchmark begins to probe multi-task integration but still operates at a scale (two atomic tasks in one component) that is tiny relative to production code.
• Mocked API interactions: All tasks use fetchMock to simulate backend API calls with predetermined success and failure responses. The model does not need to design API contracts, handle real network behavior, or integrate with actual backend systems. The frontend-vs-backend validation trap (Section 2.3) is the closest the benchmark comes to testing architectural reasoning, and it is one of only a few such edge cases across 1,000 single-task problems.

What evidence exists in the paper. The paper acknowledges the framework specificity implicitly through its benchmark descriptions, but does not discuss generalizability as a limitation. The benchmark itself is React-only by design (Cui, 2024b). Section 5 mentions future plans to "incorporate more frameworks (e.g. Vue) and languages (e.g. Python) to increase the benchmark coverage," but this is a direction for benchmark expansion, not a validation that the current findings transfer.

Mitigation status. Not addressed in the current paper. The Vue and Python extensions are listed as future work (Section 5), but even if implemented, they would not retroactively validate the claims made here about o1 reasoning model behavior. The paper's findings should be understood as specific to test-driven React component implementation in JavaScript until replication in other settings is conducted.

6.4 No Pass@k Results and No Compute-Matched Comparison Against Non-Reasoning Models

The assumption or constraint. The paper evaluates o1 models using only pass@1, citing budget constraints: "Due to budget constraints, we only obtained pass@1 results for the o1 models" (Section 2.1). This means we observe what o1 models can do in a single attempt, but not how their performance scales with multiple attempts. More importantly, the paper does not measure or account for the inference-time computation cost of the o1 models' internal reasoning process. The reasoning models generate internal chain-of-thought tokens before producing their output — tokens that consume compute, increase latency, and are billed to the user — but this cost is entirely invisible in the pass@1 comparison against non-reasoning models.

The consequence. Two distinct interpretability problems arise:

First, the pass@1 comparison may be unfair in opposite directions depending on the use case. For deployment scenarios where retries are feasible, pass@k is the relevant metric: a model that achieves 0.3 pass@1 but 0.9 pass@10 may be more useful than a model that achieves 0.4 pass@1 but only 0.5 pass@10. The paper cannot speak to this because pass@k for o1 models is unknown. On the single-task benchmark, where o1-preview achieves 0.952, pass@k scaling is unlikely to change the conclusion (the ceiling is already near 1.0). But on the duo-task normalized format (where o1-mini at 0.667 trails Claude 3.5 at 0.679), the ranking could reverse at pass@k if o1 models' failures are more independent across attempts than non-reasoning models' failures. The amplification hypothesis predicts the opposite — that o1 failures should be more correlated across attempts (systematic errors in reasoning chains) — but this prediction is untestable without pass@k data.

Second, and more fundamentally, the pass@1 comparison is not compute-matched. The o1 models' reasoning process consumes additional inference-time computation that is not counted in the pass@1 metric. A single o1-preview generation may cost substantially more in FLOPs, tokens, latency, and API dollars than a single Claude 3.5 generation. If o1-preview uses, say, 10× the inference compute of Claude 3.5 to produce its single answer, then a fair comparison would give Claude 3.5 10 attempts (pass@10) against o1-preview's pass@1 — and the outcome of that comparison is unknown because neither o1's reasoning cost nor non-reasoning models' pass@k curves are reported. The paper's headline finding — o1 models set SOTA on single-task — may not hold in a compute-matched comparison if non-reasoning models can achieve similar or better accuracy by spending the same compute budget on multiple samples.

Similarly, the raw-format failure (0.0 for o1 models) may be less dramatic in a compute-matched comparison if o1 models are using much more compute per attempt — a compute-matched comparison might give non-reasoning models enough attempts to achieve higher pass rates than the 0.32 reported for Claude 3.5 at pass@1, widening the gap further.

What evidence exists in the paper. The paper explicitly states the pass@1 constraint for o1 models (Section 2.1) but does not discuss its implications for the fairness or interpretability of the comparisons. The paper provides no data on:

• o1 model pass@k for any k > 1.
• The number of internal reasoning tokens generated by o1 models per problem (the API does not expose this, but it affects cost and latency).
• The per-query cost or wall-clock time for any model, reasoning or non-reasoning.
• The non-reasoning models' pass@k curves on the same benchmarks, which would allow readers to estimate how much non-reasoning model performance improves with additional attempts.

Mitigation status. The paper acknowledges the budget constraint but does not treat it as a limitation affecting the validity of the claims — it is presented as a practical constraint, not a methodological concern. The paper does not compute or discuss compute-matched comparisons, does not propose methods for estimating o1 reasoning cost, and does not flag the absence of compute-matched comparisons as an area for future work. This omission is significant because the practical deployment question is not "which model is best at pass@1 regardless of cost?" but "which model provides the best accuracy per unit of inference compute or per dollar?" — and the paper cannot answer this question with the data it provides.

6.5 No Statistical Rigor: Point Estimates Without Variance, Confidence Intervals, or Significance Tests

The assumption or constraint. All results are reported as single point estimates — pass@1 fractions over the 1,000-problem benchmarks (e.g., 0.952, 0.667, 0.679) — with no accompanying measures of uncertainty. The paper does not report confidence intervals, standard errors, or statistical significance tests for any comparison between models.

The consequence. Several of the paper's interpretive claims — particularly those involving small numerical differences between models — are uninterpretable because the reader cannot distinguish signal from noise:

• On the duo-task normalized format (Table 7), Claude 3.5's 0.679 leads o1-mini's 0.667 by 0.012. With 1,000 test problems, the standard error of a proportion at ~0.67 is approximately sqrt(0.67 × 0.33 / 1000) ≈ 0.015. The observed difference (0.012) is smaller than one standard error, meaning we cannot reject the null hypothesis that the two models have equal underlying capability on this benchmark. The paper's claim that o1 models "fall behind Claude 3.5" (Section 3.1) on this format is statistically unsupported on the provided data.
• The assertion that o1-preview (0.952) and o1-mini (0.939) represent a meaningful difference on single-task (Table 2) similarly lacks statistical support — a 0.013 difference on 1,000 problems has a standard error of approximately sqrt(0.945 × 0.055 / 1000) ≈ 0.007, making the difference about 1.8 standard errors — potentially significant but not reported as such.
• The claim that the o1 models "unlock a total of 16 challenges never solved by previous non-reasoning models" (Section 2.1) could be an artifact of finite sampling: with non-reasoning model pass@1 around 0.88, roughly 120 problems are unsolved by the best non-reasoning model. If o1-preview solves 48 of those (lifting pass@1 from 0.88 to 0.952, a gain of 72 problems, some of which were previously solved by other non-reasoning models and some not), the 16 "never solved" problems are a subset. Whether this subset represents genuinely novel capability or statistical fluctuation in which specific problems happen to be solved in the finite sample cannot be determined without uncertainty quantification.
• The paper does not report whether the 1,000 problem sets for WebApp1K and WebApp1K-Duo are the same as, overlap with, or are independent of the sets used in prior work (Cui, 2024a, 2024b) to establish baseline pass@1 rates. If the non-reasoning model pass@1 values in Tables 2, 6, and 7 are drawn from prior publications rather than re-evaluated on the exact same problem instances used for o1 evaluation, differences in problem difficulty distributions could confound the comparisons.

What evidence exists in the paper. No uncertainty quantification of any kind appears. All numbers are point estimates. The paper does not discuss sample size, statistical power, or the interpretation of small numerical differences. This is particularly concerning for the duo-task normalized comparison (Table 7), where the ranking between Claude 3.5, o1-mini, and o1-preview is central to the paper's narrative about reasoning model degradation but the numerical gaps are within plausible sampling noise.

Mitigation status. Not addressed. The paper reports results as if precision is perfect and any numerical difference is meaningful. This is a basic methodological omission for empirical work making comparative claims. Future work should report confidence intervals (e.g., binomial proportion confidence intervals for pass@1) and, where comparisons between models are central to claims, appropriate hypothesis tests (e.g., McNemar's test for paired pass/fail outcomes on the same problem set, if models are evaluated on identical problem instances).

6.6 The Hardest Problems Are Fundamentally Unsolved Regardless of Model or Strategy

The assumption or constraint. The paper's core finding is that reasoning models exhibit highly variable performance depending on task complexity and prompt format — excelling on single tasks, catastrophically failing on the duo-task raw format, and trailing non-reasoning models on the duo-task normalized format. However, even the best model × format combination — Claude 3.5 on the duo-task normalized format — achieves only 0.679 pass@1. This means that across any combination of model and prompt format evaluated in this paper, at least 32.1% of the duo-task benchmark remains unsolved.

The consequence. There exists a substantial fraction of duo-task problems — likely those with the most demanding combinations of implicit requirements, atypical API patterns, and complex state management across the two constituent tasks — that no current model can solve in a single attempt, regardless of whether it uses reasoning or not. The paper provides no analysis of what distinguishes these 321+ unsolved problems from the ~679 that Claude 3.5 solves, nor does it investigate whether the failure modes on these hardest problems are the same as or different from the failure modes on easier problems. Several specific knowledge gaps result:

• Are the unsolved problems harder versions of the same challenges, or qualitatively different challenges? The paper identifies specific failure patterns — missed expectations from longer inputs (Section 3.3), planning-step errors from syntactic misdirection (Section 3.2), pretraining prior overriding test requirements (Section 2.3.1). But these failure patterns are illustrated on problems that some model can solve (Claude 3.5 solves the raw-format duo tasks 32% of the time; o1-preview solves the ticketSubmission single task 95.2% of the time). The problems that no model solves under any format may exhibit fundamentally different challenges — e.g., requiring multi-step reasoning that exceeds all models' capabilities, or involving test specifications that are genuinely ambiguous — and the paper's diagnostic framework may not apply to them.
• What is the ceiling for test-time compute scaling on the hardest problems? The paper does not measure pass@k for any model, so we do not know whether the unsolved problems would yield to repeated sampling (suggesting probabilistic failure from near-capability) or remain unsolved even at high k (suggesting fundamental capability gaps). The amplification hypothesis predicts that reasoning model failures on the hardest problems should be systematic rather than probabilistic — if the reasoning chain misses a requirement, it will miss it on every attempt. But this prediction cannot be tested without pass@k data, and for non-reasoning models, the distinction between probabilistic and systematic failure on the hardest problems is similarly unknown.
• Is further progress on this benchmark gated by better instruction following, better reasoning, or both? The paper argues that "the coding success of reasoning models hinges on the top-notch base model and SFT to ensure meticulous adherence to instructions" (Section 5). If this is correct, then improving instruction adherence should close the gap between current SOTA (0.679) and the theoretical ceiling (1.0). But if the hardest problems require capabilities that go beyond instruction following — e.g., debugging complex state interactions between the two tasks, or inferring requirements that are genuinely underspecified by the tests — then instruction adherence improvements alone will not suffice.

What evidence exists in the paper. The performance ceiling is visible in every results table: the best single-task pass@1 is 0.952 (Table 2), meaning ~5% of single tasks are unsolved by o1-preview; the best duo-task raw-format pass@1 is 0.32 (Table 6), meaning 68% are unsolved; the best duo-task normalized pass@1 is 0.679 (Table 7), meaning 32.1% are unsolved. The paper does not analyze these unsolved subsets, does not bin problems by the number of models that can solve them, and does not categorize the failure modes on problems that resist all models. The qualitative analysis exclusively examines problems where at least one model succeeds (single-task) or where a specific syntactic property drives dramatic failure (raw-format duo), leaving the hardest tier of problems entirely unexamined.

Mitigation status. Not addressed. The paper's diagnostic focus is on variability — why the same model succeeds in one condition and fails in another — rather than on the absolute ceiling of what is achievable. The unsolved ~32% of duo tasks and ~5% of single tasks are acknowledged only implicitly through the pass@1 numbers. Understanding what makes these problems uniquely hard, and whether reasoning models provide any advantage over non-reasoning models on them, is a central open question for future work that the paper does not flag.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model, training method, or benchmark — it changes the conceptual framing through which the field understands reasoning models. Before this work, the release of OpenAI's o1 models was accompanied by a narrative of unqualified improvement: reasoning models achieved SOTA on math, science, and competitive programming benchmarks (OpenAI, 2024; Zhang et al., 2023; Rein et al., 2023; Mirzayanov, 2009), and the implicit assumption was that the reasoning mechanism makes models more robust in general. This paper provides a sharp counterpoint: reasoning is not a universal robustness enhancer. It is an amplifier — it systematically elaborates the direction established in early reasoning steps, whether that direction is correct or catastrophically wrong.

The magnitude of this contribution is best characterized as a diagnostic reframing, not a paradigm shift. The paper does not overturn the value of reasoning models — on single-task WebApp1K, o1-preview's 0.952 pass@1 lifts SOTA by 7% and unlocks 16 previously unsolved problems (Table 2), results that any practitioner would call impressive. Rather, it establishes boundary conditions for when reasoning helps and when it hurts, drawing those boundaries with unusual precision through the contrast between the raw and normalized duo-task formats (0.0 vs. 0.652–0.667 for o1-preview, Tables 6 and 7). The fact that a syntactically valid but semantically misleading prompt structure can cause reasoning models to fail every single problem — while non-reasoning models solve nearly a third — is not a marginal effect. It is a qualitative vulnerability that changes how the field should think about deploying reasoning models on real-world tasks.

The paper resolves a latent tension in the reasoning model literature that prior work could not articulate. The chain-of-thought and self-play literatures (Wang and Zhou, 2024; Lightman et al., 2023; Chen et al., 2024; Zelikman et al., 2022, 2024) treat reasoning as a mechanism for exploring solution space and catching errors — a corrective layer on top of base model capabilities. The instruction-following literature, by contrast, has focused on how well models attend to explicit specifications, often finding that longer or more complex specifications degrade performance. These two lines of work have not been in direct dialogue because they address different task types (explorational reasoning vs. specification-following). This paper bridges them by demonstrating that the reasoning mechanism's value depends on task structure: on explorational tasks, reasoning adds value by discovering non-obvious solution paths; on specification-following tasks, reasoning amplifies whatever subset of the specification the model successfully internalizes, making it both better when comprehension is complete and worse when comprehension fails. This reconciles why o1 models can simultaneously set SOTA on some benchmarks and collapse to zero on others.

The paper also identifies which research directions become more attractive, and which become less so. More attractive: work on instruction comprehension and faithful specification encoding, especially at the boundary between input parsing and the first reasoning step; work on verifier or critic mechanisms that check early reasoning plans against the original input before elaboration proceeds; work on difficulty estimation from prompt properties alone, enabling adaptive model selection. Less attractive: work that assumes reasoning depth alone will solve robustness problems — the paper shows that deeper reasoning on a flawed premise produces more-elaborated errors, not self-correction. This redirects investment from "better search over reasoning chains" toward "higher-fidelity encoding of constraints into the reasoning chain's initial state."

Follow-Up Research This Work Enables

Direct measurement of o1 reasoning traces to validate or refute the amplification hypothesis. The paper's central mechanistic claim — that o1 models commit to an interpretation in early reasoning steps and then systematically elaborate it without revisiting the original input — is supported only by ChatGPT reenactments, not by direct observation of o1's internal reasoning. The OpenAI API currently hides reasoning tokens, but if future API versions or open-source reasoning models expose chain-of-thought traces, a direct test becomes possible: for each of the 1,000 raw-format duo tasks where o1 models score 0.0, does the first reasoning step consistently commit to a multi-component architecture? For the normalized-format duo tasks where o1 models fail (~33% of the time), does the reasoning chain contain a step that explicitly processes each of the four test cases' expectations, and do failures correspond to cases where an expectation is never mentioned? A strong test of irreversibility would examine whether later reasoning steps ever reintroduce expectations that were absent from early steps — the amplification hypothesis predicts they do not. This experiment requires access to reasoning traces at scale (hundreds to thousands of problems) and systematic annotation of whether each test expectation appears in the reasoning chain, ideally blinded to the pass/fail outcome.

Systematic manipulation of the planning step to test causal intervention. If the raw-format failure is caused by a first-step planning error (committing to two components instead of one), then a prompt intervention that corrects or constrains the first step should eliminate the failure. A clean experiment: take the raw-format duo-task prompt and prepend a sentence like "Note: the test file imports from './TaskA_B' twice, but both imports reference the same default export — only one component is needed." Then measure o1-preview pass@1. If performance rises from 0.0 to near the normalized-format level (~0.65), this establishes a causal link between the first-step planning scope and the catastrophic failure. If performance remains low, the failure mechanism is not (or not only) the first-step misinterpretation — it may involve a more general difficulty with the double-default-import syntax that affects later reasoning steps as well. Variants of this intervention could test different hypotheses: adding the hint after the test cases (to see if later reasoning steps can incorporate new information), providing the hint in different phrasings (to test robustness of the intervention), or providing an incorrect hint (e.g., "two separate components are needed") to verify that the model can be pushed in either direction. This experiment is immediately feasible with the current o1 API — it requires no access to reasoning traces, only systematic prompt variation and pass@1 measurement.

Difficulty prediction from prompt properties to enable adaptive model routing. The paper demonstrates but does not solve the practical problem of knowing in advance whether a reasoning model will help or hurt on a given task. A concrete follow-up: train a lightweight classifier (potentially a fine-tuned small LM or a feature-based logistic regression) to predict, from prompt features alone, whether o1-preview will succeed on a given WebApp1K or WebApp1K-Duo problem. Candidate features include: prompt token count, number of test cases, number of fireEvent calls (a proxy for interaction complexity), number of expect statements, presence of double-default-import patterns (detectable via regex), number of distinct mocked API endpoints, and maximum nesting depth of mock response objects. The training labels would be o1-preview pass/fail outcomes on a held-out subset of the benchmark. If the classifier achieves non-trivial accuracy (significantly above the base rate), it provides a practical tool for deciding when to route a task to o1 vs. Claude 3.5 vs. a cheaper non-reasoning model — a model selection decision that currently requires trial and error. The paper's finding that prompt syntax (raw vs. normalized format) has enormous causal effect on o1 performance suggests that syntactic features should be highly predictive, making this a feasible near-term project. The harder question is whether the classifier transfers to coding tasks outside the WebApp1K distribution, which would require evaluation on an independently constructed test set.

Pass@k scaling analysis for reasoning vs. non-reasoning models under compute-matched budgets. The paper's pass@1 comparison is not compute-matched because o1 models consume additional inference compute for their internal reasoning tokens. A rigorous follow-up would: (1) measure or estimate the total inference FLOPs or API cost for one o1-preview generation and one Claude 3.5 generation on the same WebApp1K-Duo problems; (2) compute the ratio R = (o1 generation cost) / (Claude 3.5 generation cost); (3) measure pass@k for Claude 3.5 for k values up to ~R, establishing how much Claude 3.5 improves with a compute budget equivalent to one o1-preview generation; (4) compare o1-preview pass@1 against Claude 3.5 pass@R. This experiment directly tests whether the o1 models' single-task SOTA (0.952) holds under compute-matched comparison — if Claude 3.5 at pass@R approaches or exceeds 0.952, then the reasoning mechanism's advantage is partly or wholly attributable to spending more inference compute rather than to more efficient use of compute. Conversely, if o1-preview pass@1 exceeds Claude 3.5 pass@R by a wide margin, the reasoning mechanism provides genuine compute efficiency gains on single tasks. On the duo-task normalized format, the comparison would reveal whether Claude 3.5's lead (0.679 vs. 0.652–0.667) widens or narrows under compute matching — the paper's current claim that o1 models "fall behind" Claude 3.5 could be either strengthened (if Claude 3.5 wins even more decisively when given equal compute) or reversed (if o1's per-attempt advantage emerges when Claude 3.5's multiple attempts are accounted for). This experiment is gated by cost transparency — it requires knowing o1's per-generation cost in tokens or dollars, which OpenAI may or may not expose at sufficient granularity.

Cross-language and cross-framework replication to test the generality of syntactic misdirection vulnerability. The raw-format failure depends on JavaScript ES6 module semantics (name-agnostic default imports, single default export per module). Are reasoning models vulnerable to analogous syntactic misdirection in other languages? A direct replication: construct a Python equivalent where the test file imports the same function twice under different names (valid in Python via from module import func as name1 and from module import func as name2) and the correct implementation is a single function satisfying both sets of test expectations. Measure o1-preview pass@1 on this Python variant. If performance is near-zero (analogous to the raw format), the vulnerability is not JavaScript-specific — it is a general property of how reasoning models process import syntax to infer architectural requirements. If performance is high, the vulnerability is JavaScript-specific, perhaps due to relative frequency of the double-default-import pattern in pretraining data vs. the Python equivalent. Further variants could test: Java (with its explicit class-per-file convention, which might make multi-file architectures seem even more natural), Rust (with its module system's use statements), and frameworks other than React (Vue single-file components, Svelte, Angular). This replication program would establish whether the planning-step irreversibility hypothesis generalizes across ecosystems or is an artifact of JavaScript's specific syntactic properties. The experiment is immediately feasible — it requires implementing test harnesses for each language/framework, constructing prompts analogous to the WebApp1K-Duo format variants, and evaluating o1 models at pass@1.

Error categorization at scale to quantify the prevalence of "ignored expectation" vs. other failure modes. The paper identifies one specific failure mode on the duo-task normalized format: o1-preview overlooks the Internal Server Error expectation from the retrieveAllBlogPosts task when it is paired with addComment (Section 3.3). But the paper provides no information about how many of o1-preview's ~348 failures on the normalized duo benchmark follow this pattern vs. other patterns (e.g., correctly noticing all expectations but producing implementation bugs, or noticing expectations but failing to integrate them without interference between the two tasks). A systematic follow-up would: take all 1,000 duo-task problems in the normalized format, collect o1-preview's generated code for each, and for every failure, categorize the root cause by comparing the generated code against the test expectations. Categories would include: (a) missing expectation (a test assertion has no corresponding logic in the implementation), (b) misinterpreted expectation (logic exists but is wrong, e.g., frontend validation vs. backend validation), (c) implementation bug despite correct expectation capture (logic exists and is correct in intent but contains a coding error), and (d) task interference (logic for Task A breaks Task B, or vice versa). The paper's amplification hypothesis predicts that category (a) should dominate on duo tasks, and that category (a) failures should be more common for the second task in the prompt (recency or attention decay effects). This experiment requires manual annotation or a reliable automatic classifier for failure categories — a non-trivial methodological challenge, but one that would transform the paper's anecdotal evidence into a quantitative characterization of reasoning model failure modes.

Practical Applications and Downstream Use Cases

Single-task test-driven development with reasoning models for high-reliability components. For development workflows where individual components have clear, testable specifications and the specification length is short (2–4 test cases, analogous to WebApp1K single tasks), o1-preview achieves 0.952 pass@1 — nearly 19 in 20 components implemented correctly on the first attempt. This is directly applicable to teams practicing test-driven development (TDD) for React applications: a developer writes the test file, feeds it to o1-preview with the Generate Component.js to pass the tests below: ... RETURN CODE ONLY prompt template, and receives a working implementation ~95% of the time without iteration. The time savings relative to manual implementation of a ~45-line component (the typical WebApp1K output size) are modest per component but compound across large codebases. The key practical caveat from this paper is format sensitivity: the test file must use normalized import patterns (single import, single component name) — if the test file uses the raw format's double-default-import pattern, o1-preview fails 100% of the time, and the developer would be better served by Claude 3.5 (0.32 pass@1 on that format). This creates a concrete workflow rule: standardize test file imports before sending to o1, or route raw-format tests to Claude 3.5.

Automated regression test generation for reasoning model evaluation in CI/CD pipelines. The WebApp1K-Duo normalized format establishes that even the best models (Claude 3.5 at 0.679) fail on ~32% of duo-task problems, and reasoning models trail slightly (0.652–0.667). For teams using LLMs in automated code generation pipelines, this provides a calibration point: if a task involves integrating two independent pieces of functionality into one component, expect the best available model to succeed on roughly 2 in 3 attempts. A CI/CD pipeline that auto-generates code and runs tests could use this benchmark's structure to continuously evaluate deployed models: maintain a bank of ~100–200 duo-task-style test specifications (not from the published benchmark, to avoid contamination), run the current model against them on each model update, and track pass@1 over time. A sudden drop in duo-task pass@1 relative to single-task pass@1 would flag the kind of instruction-comprehension degradation the paper documents, triggering an alert before the model is deployed to production. This is a direct operationalization of the paper's diagnostic framework: use the single-task/duo-task performance gap as a canary for instruction-following reliability.

Model selection based on task complexity estimation for cost-optimized coding assistants. The paper's finding that model rankings invert between single-task (o1-preview > Claude 3.5, Table 2) and duo-task normalized (Claude 3.5 > o1-preview, Table 7) creates a concrete decision rule for coding assistant products that offer multiple model backends. If the user's prompt contains a single test file with exactly two test cases and the imports follow a normalized pattern, route to o1-preview (expected success ~95%) rather than Claude 3.5 (~88%). If the prompt contains four test cases across two functional areas or uses non-standard import patterns, route to Claude 3.5 (expected success ~68% on normalized, ~32% on raw) rather than o1-preview (~65% on normalized, ~0% on raw). The complexity heuristic can be automated: count the number of test( blocks in the prompt (a proxy for task count), check for double-default-import patterns via regex, and estimate total prompt token count. This routing logic can be implemented in a few dozen lines of code and backed by the specific performance numbers in Tables 2, 6, and 7. The cost savings come from avoiding o1-preview queries on raw-format duo tasks (where it fails with certainty, wasting the query cost) and from using the cheaper Claude 3.5 on normalized duo tasks (where it actually outperforms the more expensive o1 models). The missing piece is relative per-query pricing, which the paper does not provide but which practitioners can plug in from their API bills to compute expected cost per successful component generation for each routing strategy.

Benchmark-driven prompt format linting for LLM code generation tools. The raw-format result (0.0 pass@1 for o1 models, Table 6) is so stark that it justifies building a "prompt linter" for code generation — a tool that scans a test file before it is sent to a reasoning model and flags syntactic patterns known to trigger catastrophic failure. The linter would detect double-default-imports from the same module and automatically rewrite them to the normalized format (single import, unified component name) before the prompt reaches the model. Given that this single transformation raises o1-preview pass@1 from 0.0 to 0.652, the ROI is essentially infinite for any pipeline that might encounter raw-format prompts. The linter could be extended with additional rules as the community identifies more syntactic patterns that cause reasoning model collapse — e.g., import patterns in other languages, specific mock setup structures, or test organization conventions that conflict with model pretraining priors. This is a directly implementable tool with a clear specification (input: test file, output: normalized test file or warning) and a validation set (the WebApp1K-Duo raw-format problems, which test that the normalization eliminates the 0.0 failure mode).