Premise Order Matters in Reasoning with Large Language Models

🎯 Pitch

Simply reordering the premises in a logic puzzle—without changing a single fact—can slash LLM accuracy by over 30%, because models latch onto the order they see rather than the underlying logical structure. Even top models like GPT‑4‑turbo fail on 10% of math problems they originally solved correctly when the problem statements are permuted.

1. Executive Summary

This paper studies how the ordering of premises affects LLM reasoning performance across deductive logic and mathematical problem-solving, evaluating GPT-4-turbo, GPT-3.5-turbo, PaLM 2-L, and Gemini 1.0 Pro on a custom logical reasoning benchmark derived from SimpleLogic and a newly constructed benchmark called R-GSM (reordered GSM8K). The central finding is that LLMs are brittle to premise order—presenting premises in the forward order (the order aligning with the ground-truth proof, where each rule applied in a derivation step appears sequentially in the problem text; τ = 1) yields the best performance, while permuting to a shuffled order (τ ≈ 0) or backward order (τ = -1, the reverse of forward) causes accuracy drops exceeding 30% on logical reasoning tasks with 12 relevant rules, and up to a 35% relative degradation on R-GSM problems the model initially solved correctly. The paper establishes that this ordering effect is not merely a curiosity but a systematic frailty amplified by longer reasoning chains, irrelevant distracting premises, and models' tendency to process information left-to-right rather than reasoning back-and-forth across the problem text—establishing that LLMs fail on logically equivalent reorderings even when the underlying task is unchanged, with the performance gap widening as problem complexity increases.

2. Context and Motivation

The Core Problem: Premise Order Shouldn't Matter, But It Does

This paper addresses a specific, well-defined brittleness in LLM reasoning: the ordering of premises dramatically affects reasoning accuracy, even though the logical task is invariant under permutation. Consider a set of deductive reasoning premises — facts and rules that together entail a conclusion. In formal logic, the conclusion follows regardless of the order in which those premises are presented. A reasoner (human or machine) that truly understands the logical relationships should derive the same correct conclusion whether the premises appear as A → B, B → C, A or as B → C, A → B, A. The logical content is identical; only the surface presentation differs.

The paper demonstrates that LLMs violate this invariance in a systematic and severe way. The opening example in Section 1 makes this concrete: given the three-statement modus ponens chain above, LLMs perform substantially worse when "If B then C" appears before "If A then B" — a purely superficial reordering that changes nothing about the underlying deduction. On logical reasoning problems with 12 relevant rules, permuting from the forward order (τ = 1, premises aligned with the proof sequence) to shuffled orders (τ ≈ 0) causes accuracy drops exceeding 30 percentage points across multiple state-of-the-art models (Figure 3). On the R-GSM mathematical reasoning benchmark, Table 2b shows that across the subset of problems each model initially solved correctly, reordering causes GPT-4-turbo to fail on 10.1% of them, PaLM 2-L on 12.1%, Gemini 1.0 Pro on 25.4%, and GPT-3.5-turbo on 35.1%.

The problem is not that LLMs are weak reasoners in absolute terms — GPT-4-turbo achieves 96.5% accuracy with forward-ordered premises at 12 rules (Table 6a). The problem is that this strong performance collapses under a transformation that is semantically vacuous, revealing that the models' reasoning process is shallowly tied to surface-level text ordering rather than grounded in robust logical understanding.

Why This Matters: Real-World and Theoretical Significance

The significance of this finding operates on multiple levels:

Practical deployment reliability. LLMs are increasingly deployed for reasoning tasks where premise order is intrinsically arbitrary: legal document analysis (clauses can appear in any sequence), contract review, policy interpretation, multi-step customer support queries, and any domain where information arrives in unpredictable order. A system whose accuracy swings by 30+ percentage points based on how information is sequenced is fundamentally unreliable for real-world deployment. The paper's R-GSM results make this concrete: GSM8K-style math word problems are a standard benchmark for deployed reasoning capabilities, and the paper shows that simply reordering sentences — without changing any numbers, relationships, or the underlying problem — causes models to fail on problems they previously solved. In production, a user who phrases their problem with sentences in a "wrong" order would receive incorrect answers, even though a human would recognize the problem as identical to one the user stated more conventionally.

Theoretical implications for what LLMs learn. The premise order effect is a diagnostic signal about the nature of LLM reasoning. If a model has genuinely learned to perform multi-step logical deduction, premise order should be largely irrelevant — the model should be able to scan the available premises, identify which ones are applicable at each step, and chain them together regardless of where they appear in the text. The fact that order matters so much suggests that LLMs are not doing this kind of flexible symbolic reasoning. Instead, their reasoning appears to be position-bound and sequential: they process premises left-to-right, applying rules as they encounter them in the text, and struggle when the next rule needed for the proof doesn't happen to be the next rule in the prompt. This connects to the paper's description of the phenomenon in Section 1:

"Intuitively, such a preference on the premise order aligns with human preference... because in the preferred order, each derivation step can be done on-the-fly while looking at premises one by one, without needing to look back and forth across all premises at each step."

The fact that LLMs require this alignment — rather than merely preferring it — suggests that their reasoning mechanism is fundamentally different from the flexible premise selection and reordering that symbolic reasoners or attentive human reasoners perform. This is a theoretically important constraint on what the auto-regressive training objective actually teaches models about reasoning.

Connection to broader LLM brittleness patterns. The premise order effect doesn't exist in isolation. It connects to and extends several documented LLM failure modes. The paper explicitly positions itself relative to the Reversal Curse (Berglund et al., 2023), where LLMs trained on "A is B" fail to infer "B is A." That work studies order effects within single factual statements (two entities, one relation), while this paper extends the investigation to multi-premise reasoning chains with arbitrary numbers of entities and relationships. The finding that random permutations are often worse than the purely backward order (Section 3.2, Figure 5) is a novel observation not predicted by the reversal curse alone. The paper also connects to distractibility (Shi et al., 2023), showing that irrelevant premises not only degrade performance overall but specifically amplify the ordering effect (Figures 4 and 6), creating a compounding brittleness when problems are both long and include extraneous information.

Benchmark validity. The finding has methodological implications for how reasoning benchmarks are constructed and evaluated. If ordering affects accuracy so dramatically, then the de facto standard of presenting premises in a particular order (as GSM8K and other benchmarks naturally do) may systematically overestimate LLM reasoning capabilities. The R-GSM benchmark demonstrates that even within the same problem, accuracy can vary substantially based on sentence ordering — meaning that single-order benchmark scores are not robust measures of a model's reasoning ability. This paper effectively argues that reasoning evaluations should control for presentation order, either by reporting performance across orderings or by randomizing order to measure order-invariant reasoning.

Prior Approaches and Why They Fall Short

The paper doesn't propose a new method for fixing the ordering problem — it is primarily a diagnostic and characterization study. As such, the "prior approaches" it positions against are not competing solutions but rather existing frameworks for understanding LLM reasoning failures, each of which captures part of the phenomenon but fails to fully account for the systematic ordering brittleness observed here.

Existing failure mode frameworks are individually insufficient. The paper cites multiple documented LLM weaknesses:

Distractibility (Shi et al., 2023): LLMs perform worse when irrelevant context is present. This is consistent with the paper's finding that adding 5 or 10 distracting rules degrades performance and amplifies ordering effects (Figures 4, 6). However, distractibility alone doesn't explain why relevant premises, when reordered, cause such severe degradation. The premises are all necessary for the proof — none are irrelevant — yet order still matters. Distractibility is a contributing factor but not the root cause.
The Reversal Curse (Berglund et al., 2023): LLMs fail to reverse learned associations. This explains why the backward order (τ = -1) might perform worse than the forward order, but it doesn't explain why shuffled orders (which mix forward and backward subsequences) often perform worse than the purely backward order (as observed for GPT-4-turbo and GPT-3.5-turbo in Figure 5, where τ = -1 accuracy exceeds τ = 0.5 or τ = 0 accuracy). The reversal curse predicts forward > backward, but it doesn't predict forward > backward > shuffled.
Position bias / lost-in-the-middle (Liu et al., 2024): LLMs perform worse when relevant information appears in the middle of long contexts rather than at the beginning or end. The paper explicitly tests this alternative explanation in Appendix D (Table 5), varying whether relevant rules appear at the beginning, middle, or end of the input while holding the premise order constant. The performance variation from position alone is "very small" compared to the variation from changing order, and the paper notes that their logical reasoning problems are at most ~300 tokens — far shorter than the context lengths where lost-in-the-middle typically manifests. The ordering effect is not reducible to position bias.
Limited logical reasoning capability (Saparov and He, 2022; Saparov et al., 2023; Xu et al., 2023; Yan et al., 2023; Wan et al., 2024): Prior work has established that LLMs struggle with multi-step deduction, especially as proof length increases. However, this body of work typically studies reasoning difficulty as a function of proof complexity without controlling for premise order. The current paper shows that proof complexity interacts with premise order: longer proofs (more rules) amplify the ordering effect (Figure 3), meaning that prior studies may have conflated "LLMs can't do long proofs" with "LLMs can't do long proofs when premises are presented in a non-ideal order." The forward-order performance ceiling (GPT-4-turbo at 96.5% with 12 rules) suggests that when premises are well-ordered, LLMs can handle substantially longer proofs than previously recognized.

No prior work systematically characterizes premise ordering as a first-class variable. The key gap is that, before this paper, the premise order effect had not been isolated and systematically studied as an independent phenomenon. Each prior framework captured a piece (distraction, reversal, position) but missed the unifying pattern: that LLM reasoning performance is governed by the alignment between premise presentation order and proof derivation order, with a strong preference for forward alignment and a systematic degradation as this alignment breaks. The paper fills this gap by:

Constructing a controlled benchmark where only premise order varies (same premises, same conclusion, 15 variants per problem spanning five Kendall tau distances to the forward order).
Measuring the effect across multiple model families (GPT, PaLM, Gemini) to establish generality.
Extending the investigation beyond logical reasoning to mathematical problem-solving (R-GSM) to test domain generality.
Characterizing failure modes (fact hallucination, rule hallucination, wrong refutation) and showing how they shift with ordering (Table 1).

How This Paper Positions Itself Relative to Existing Work

The paper positions itself primarily as a systematic empirical characterization of an understudied failure mode, not as a solution paper. The conclusion states this explicitly:

"We can attempt to ascribe the premise order effect to several candidate factors, such as the auto-regressive model design, training objectives, and training data mixture. However, we leave proposing theoretical explanations of this limitation and developing new techniques towards addressing the premise order effect as future work."

This is a deliberate scoping choice. The contribution is the demonstration and measurement of the phenomenon, with the goal of making the community aware of a brittleness that has implications for evaluation, deployment, and our understanding of what LLMs actually learn about reasoning.

The paper's relationship to human reasoning research (Dekeyser et al., 2000; Girotto et al., 1997) is particularly instructive for understanding its positioning. The authors cite evidence that humans have a preference for premises ordered to allow sequential, on-the-fly deduction — what Dekeyser et al. called "co-reference." However, the key distinction is one of degree: humans prefer forward order but can flexibly reason with other orders, especially on simple modus ponens problems. LLMs, by contrast, exhibit a much more brittle dependence:

"While humans also have a preference of premise orders for reasoning problems, LLMs are much more susceptible to such ordering effects." (Section 5)

Girotto et al. found that premise order significantly affects humans only on harder modus tollens problems (if P then Q; not Q; therefore not P), not on the simpler modus ponens problems (if P then Q; P; therefore Q) that this paper studies. The fact that LLMs show dramatic ordering sensitivity even on these elementary modus ponens deductions — problems a human child could solve regardless of premise order — highlights a fundamental gap between surface-level reasoning performance and robust logical competence. The paper doesn't claim LLMs are worse than humans at reasoning (in fact, forward-order performance is quite high); it claims they achieve this performance through a mechanism that is more fragile than human reasoning in ways that aren't obvious from standard benchmark evaluations.

Methodologically, the paper positions its contribution through benchmark construction. The logical reasoning benchmark is carefully designed to isolate ordering from confounding variables: it uses only modus ponens deductions (eliminating theorem complexity as a confound), generates 15 ordering variants per problem, and enforces strict correctness criteria (the entire proof must be valid, not just the final answer). The R-GSM benchmark extends the investigation to a domain — grade-school math — where ordering effects are less obvious a priori. By showing that sentence reordering degrades math problem-solving as well, the paper establishes that the ordering effect is not an artifact of logical deduction formalisms but a general characteristic of how LLMs process multi-step reasoning problems.

Finally, the paper connects to work on input preprocessing for reasoning, specifically Yan et al. (2023)'s Concise and Organized Perception approach, which explicitly reorders premises to improve LLM deductive reasoning performance. The fact that such preprocessing works — that reordering premises to align with the proof dramatically improves accuracy — is cited as convergent evidence for the importance of premise ordering. However, the paper distinguishes its contribution by noting that such task-specific preprocessing pipelines "do not generalize across domains" and advocating instead for the development of "generic end-to-end reasoning techniques" that would make LLMs robust to ordering intrinsically (Section 4). The current paper is positioned as the diagnostic foundation that motivates and informs such future technique development.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a controlled experimental framework — not a model or a training method — designed to isolate and measure how the surface ordering of premises (the facts and rules presented in a reasoning problem's text) affects LLM reasoning accuracy, independent of the underlying logical task. The "system" being studied is the LLM itself (GPT-4-turbo, GPT-3.5-turbo, PaLM 2-L, Gemini 1.0 Pro) under zero-shot prompting, and the investigation systematically varies how premises are sequenced in the prompt to reveal a specific brittleness: LLMs achieve high accuracy when premises are ordered to match the step-by-step derivation of the proof (the forward order, quantified by Kendall tau distance τ = 1 relative to that ideal sequence) but suffer accuracy collapses of 30+ percentage points when the same premises are reshuffled, even though the logical content and the correct conclusion remain unchanged.

3.2 Big-Picture Architecture (Diagram in Words)

The experimental framework has five major components, each serving a distinct role in the measurement pipeline:

Logical Reasoning Benchmark Generator — A synthetic problem generator that creates modus ponens deduction problems (facts + rules → conclusion) with controlled parameters: number of relevant rules (4–12), number of distracting rules (0, 5, 10), and premise ordering (5 distinct Kendall tau distances τ ∈ {1, 0.5, 0, −0.5, −1} relative to the forward proof order). For each of the 200 problems per rule count, the generator produces 15 variants (5 orders × 3 distraction levels), yielding 27,000 total test instances.
R-GSM Benchmark Constructor — A manual rewriting pipeline that takes GSM8K test problems (grade-school math word problems with at least 5 sentences) and produces reordered variants where the sentence sequence is permuted while preserving the mathematical content and ground-truth answer. The pipeline filters out problems where no semantically valid reordering exists (e.g., causal event chains), enumerates alternative orderings programmatically to find ones that cause LLM failures, and applies minor grammatical edits to maintain fluency. The result is 220 paired problems (original + reordered).
LLM Inference Engine — Four state-of-the-art models (GPT-4-turbo, GPT-3.5-turbo, PaLM 2-L, Gemini 1.0 Pro) evaluated under identical zero-shot prompting with greedy decoding (temperature = 0). For logical reasoning, the prompt includes an instruction to produce step-by-step derivations citing which premise is used at each step. For R-GSM, the prompt contains only the problem description with no additional instructions.
Correctness Evaluator — A strict grading module that, for logical reasoning, requires the entire generated proof to be valid (not just the final answer), flagging any hallucinated facts or rules as errors; for R-GSM, compares the final numerical answer against the ground truth using standard GSM8K answer extraction and matching.
Analysis Module — An error categorization system that classifies logical reasoning failures into three types: wrong refutation (model incorrectly claims the conclusion is unprovable), rule hallucination (generates rules not present in the problem), and fact hallucination (generates facts that are neither given nor proven). For R-GSM, errors are classified into temporal order violations, unknown variable introduction, and other categories.

Information flows unidirectionally: the benchmark generators produce problem instances → the LLM inference engine processes each instance with greedy decoding → the correctness evaluator judges the output → the analysis module categorizes failures and computes accuracy statistics disaggregated by τ, number of rules, number of distractions, number of reasoning steps, and problem length.

3.3 Roadmap for the Deep Dive

The technical breakdown proceeds in five stages, ordered to build from the measurement formalism to the evaluation mechanics:

First, the Kendall tau distance formalism (τ) — the mathematical tool used to quantify premise order and define the experimental conditions. Understanding τ is prerequisite because every result in the paper is sliced by this metric.
Second, the logical reasoning benchmark construction — how problems are generated, how the forward order (τ = 1) is defined, how the five τ conditions are instantiated, and what constraints ensure that only ordering (not logical content) varies across variants.
Third, the R-GSM benchmark construction — the manual rewriting pipeline, the filtering criteria, the enumeration-based search for adversarial reorderings, and the quality control (grammar edits, answer preservation verification).
Fourth, the LLM evaluation protocol — the zero-shot prompting format, the greedy decoding configuration, the proof grading criteria for logical reasoning, and the answer extraction for R-GSM.
Fifth, the error analysis methodology — the three-category (plus correct) classification for logical reasoning, the temporal/unknown/other taxonomy for R-GSM, and the ablation design for ruling out position bias (lost-in-the-middle) as an alternative explanation (Appendix D).

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a controlled measurement and empirical characterization paper whose core idea is that premise order in reasoning prompts is a first-class variable governing LLM accuracy, and that systematic variation of this variable reveals a fragility not captured by standard single-order benchmark evaluations.

The Kendall Tau Distance Formalism

The paper needs a quantitative way to describe how "ordered" a given premise sequence is relative to an ideal reference sequence — the forward order that follows the step-by-step ground-truth proof. They adopt the Kendall tau distance (τ), a measure of rank correlation between two sequences, and normalize it to the range [−1, 1].

Definition and Normalization

Let π_forward be the sequence of premises ordered to match the ground-truth proof (rule applied in derivation step 1 appears first, rule applied in step 2 appears second, etc.). Let π_actual be the premise sequence as presented to the LLM. The Kendall tau distance measures the number of pairwise disagreements between these two orderings: for every pair of premises (A, B), if A appears before B in π_forward but after B in π_actual, that counts as one discordant pair. The normalized τ is computed as:

$\tau = 1 - \frac{2 \times (\text{number of discordant pairs})}{\binom{n}{2}}$

where n is the total number of premises in the sequence, and \binom{n}{2} = n(n-1)/2 is the total number of distinct pairs.

What it computes: τ quantifies the agreement between the premise presentation order and the order required by the forward-chaining proof. τ = 1 means the orders are identical (perfect forward alignment — every pair of premises appears in the same relative order as in the proof). τ = 0 means the orders are uncorrelated (random permutation — roughly half the pairs agree, half disagree). τ = −1 means the premise order is exactly the reverse of the forward order (perfect backward alignment — every pair appears in the opposite relative order, which aligns with backward-chaining proof where you start from the conclusion and work backwards through the rules).

Why this form: The Kendall tau is chosen because it is a natural rank correlation measure that directly captures the sequential reasoning preference the paper hypothesizes — that LLMs prefer premises to appear in the order they will be used during forward-chaining deduction. The normalization to [−1, 1] makes the metric interpretable across problems with different numbers of premises: τ = 0.5 always means "moderately aligned with forward order," regardless of whether there are 4 or 12 premises. This would not be true of raw discordant pair counts. The symmetric treatment of forward and backward (τ = 1 vs. τ = −1) is theoretically motivated: backward chaining is a well-established alternative inference method (start with the goal, work backwards through rules to reach known facts), and the paper wants to test whether LLMs can exploit backward-compatible ordering as effectively as forward-compatible ordering. The five sampled τ values (1, 0.5, 0, −0.5, −1) provide a roughly uniform grid over the possible range of order alignment, enabling the paper to observe whether accuracy degrades smoothly as alignment decreases, or whether there are sharp thresholds.

Premise Order Categorization

In the paper's terminology, the specific orderings correspond to:

τ = 1: Forward order. Premises appear exactly in the sequence they are used in a forward-chaining proof. If the proof applies Rule 3 to facts A and B to derive C, then applies Rule 7 to C to derive D, then Rule 3 appears before Rule 7 in the prompt.
τ = −1: Backward order. The exact reverse of the forward order. This would align with backward-chaining: the last rule used in forward-chaining appears first (corresponding to the rule needed to derive the conclusion from intermediate results), and the first rule used appears last (corresponding to the rule that connects to the base facts).
τ = 0.5, 0, −0.5: Intermediate shuffed orders with progressively weaker alignment to the forward sequence. τ = 0 is a random permutation with no systematic relationship to the proof order.
Shuffled (aggregate): In several figures (e.g., Figure 3), the paper reports "Shuffled" accuracy, which aggregates results across τ = 0.5, τ = 0, and τ = −0.5 — the non-extreme orderings — to provide a summary statistic for "not-ideally-ordered" performance.

This categorization is not merely descriptive; it drives the entire experimental design. Every logical reasoning problem is instantiated in all five τ variants, enabling within-problem comparisons where the logical content is held perfectly constant and only the surface ordering changes.

Logical Reasoning Benchmark Construction

The logical reasoning benchmark is the paper's primary measurement instrument, designed to isolate the premise order effect from all confounding variables. It is adapted from SimpleLogic (Zhang et al., 2022) but modified in several critical ways to serve the paper's specific measurement goals.

Problem Structure

Each problem in the benchmark consists of exactly three components:

A set of base facts A1, A2, ..., Am that are asserted to be true. These are atomic propositions like "Fiona is red" or "Fiona is kind" (the paper uses color and trait predicates in its examples).
A set of rules of three possible forms:
- Unary: If X, then Y (single condition → single conclusion)
- Binary: If X and Y, then Z (two conditions → one conclusion)
- Ternary: If X and Y and Z, then W (three conditions → one conclusion)
All rules are definite clauses (Horn clauses with exactly one positive literal in the conclusion), meaning they represent monotonic, deterministic deductions. There are no negations, no disjunctions in conclusions, and no exceptions — this is the simplest possible deductive reasoning framework.
A conclusion of the form C is True to be proved, where C is entailed by the facts and rules under modus ponens (if the antecedents of a rule are all known to be true, the consequent is also true).

Why this restricted form: The paper deliberately constrains the logical language to eliminate sources of difficulty other than premise ordering and proof length. By using only modus ponens over definite clauses, the authors ensure that: (a) every problem has exactly one valid proof (eliminating ambiguity about which proof the model should generate); (b) the reasoning requires no knowledge of logical theorems beyond modus ponens (the simplest inference rule — if A is true and A → B is a rule, then B is true); (c) the challenge comes purely from multi-step chaining and premise selection, not from handling negation, disjunction, or quantifiers. This makes the benchmark a minimal testbed for the premise order hypothesis: if LLMs show ordering fragility even here, the effect is likely fundamental rather than an artifact of complex logical operators.

The Forward Order (τ = 1)

The forward order is defined constructively from the ground-truth proof. The paper generates the proof by forward chaining:

Start with the set of known facts (the base facts given in the problem).
Find a rule whose antecedents are all in the known facts set. Apply it, add its conclusion to the known facts set, and record this as the first proof step.
Repeat step 2 until the target conclusion is in the known facts set or no more rules can be applied.
The sequence of rules applied, in the order they were applied (step 1's rule, then step 2's rule, etc.), defines the forward order of rules for τ = 1.

Facts are presented before rules in the prompt, but within the fact set and rule set, their ordering follows the forward-chaining sequence. Facts are listed in the order they appear in the proof: base facts needed early appear first; facts derived later (if any were explicitly listed — typically all facts are base facts in this benchmark) appear later. Rules are listed in the order they are applied: the rule used in proof step 1 comes first, then the rule used in step 2, and so on.

Why this definition matters: The forward order is not an arbitrary "nice" ordering — it is the computationally minimal ordering for sequential left-to-right processing. A reasoner that reads premises one at a time and immediately applies any rule whose conditions are satisfied will produce the proof with zero backtracking when premises are in forward order. Each new premise is immediately usable given what has been read so far. This operationalizes the paper's hypothesis about why order matters: LLMs, being auto-regressive left-to-right processors, are effectively doing something like "read premise, check if applicable, if so apply it," and forward order makes this strategy succeed without requiring the model to remember and re-scan earlier premises.

Generating the Five τ Variants

For each problem, the paper generates one premise ordering for each of the five target τ values:

τ = 1: The forward order as defined above.
τ = −1: The exact reverse of the forward order. The last-applied rule appears first, the first-applied rule appears last.

For the intermediate τ values (0.5, 0, −0.5), the paper does not describe an explicit sampling algorithm in the main text, but the implied approach (consistent with standard practice) is: compute the Kendall tau distance for the forward order, identify the set of permutations that achieve the target τ value, and sample one representative permutation from that set. For τ = 0 particularly, this is a random permutation with no systematic relationship to the forward sequence. The examples in Appendix B (Figure 11) show that as |τ| decreases, the rules become "less ordered" — the sequence of rules in the prompt deviates increasingly from the proof sequence, requiring the model to jump around the premise list to find applicable rules at each step.

Key property of the benchmark design: Across the five τ variants of a single problem, the set of premises (facts and rules) is identical. Only their order in the prompt changes. The conclusion is identical. The ground-truth proof is identical (it must be, since the logical entailment doesn't depend on premise order). This means any difference in model accuracy across τ variants for the same problem cannot be attributed to differences in logical difficulty, required knowledge, or problem ambiguity — it must be attributed purely to the surface ordering of premises in the input.

Varying Number of Relevant Rules

The paper generates 200 distinct problems for each number of relevant rules from 4 to 12 (inclusive), producing 200 × 9 = 1,800 base problems. "Relevant rules" means rules that are required in the ground-truth proof. Problems with 4 rules require 4-step proofs; problems with 12 rules require 12-step proofs. This range is chosen empirically: at 4 rules, all models perform well (≥88.5% for the weakest model, GPT-3.5-turbo, in forward order; Table 6d), providing a high baseline; at 12 rules, even the strongest model (GPT-4-turbo) drops to 96.5% in forward order (Table 6a) and much lower for other orders, revealing the scaling of the ordering effect with proof length.

Adding Distracting Rules

For each problem, the benchmark includes variants with 0, 5, and 10 distracting rules — rules whose predicates involve entities and properties present in the problem but that are never used in any valid proof of the target conclusion. For example, if the proof requires deducing properties of "Fiona" using rules about colors and traits, a distracting rule might be "If Fiona is green, then Harry is blue" — a rule that mentions known entities but whose antecedent is never satisfied given the facts and relevant rules, meaning it cannot contribute to any valid deduction.

Distracting rules are interleaved with relevant rules in the prompt (as stated in Appendix D: "In our primary experiments, for all logical reasoning problems, we interleave distracting rules with relevant rules in the input context"). The paper does not specify the exact interleaving pattern, but the design ensures that distracting rules are not segregated in a block — they appear mixed with relevant rules, increasing the difficulty of premise selection (the model must determine which rules are usable at each step while irrelevant rules are interspersed).

Why add distracting rules: This varies the difficulty of premise selection independently of proof length. Without distractions, the model knows that every rule in the prompt is needed; the only challenge is finding the right order to apply them. With distractions, the model must also determine which rules are relevant — a meta-reasoning challenge. The paper's hypothesis (which the results in Figures 4 and 6 confirm) is that ordering effects are amplified under distraction because when the model encounters an inapplicable rule (due to non-forward ordering), it cannot easily distinguish "this rule comes later in the proof" from "this rule is a distractor and never applies," leading to more hallucinated applications.

Problem Volume and Variant Multiplication

The total benchmark size results from a multiplicative design:

9 rule-count conditions: 4, 5, 6, 7, 8, 9, 10, 11, 12 relevant rules
200 problems per rule count
5 premise orderings (τ values) per problem
3 distraction conditions: 0, 5, 10 distracting rules
Total: 9 × 200 × 5 × 3 = 27,000 problem instances

For each model, all 27,000 instances are evaluated (or a relevant subset — the paper evaluates GPT-4-turbo and PaLM 2-L on all conditions including distractions, but focuses GPT-3.5-turbo and Gemini 1.0 Pro on the 0-distraction condition since their baseline performance with distractions would be near floor). Each instance is a self-contained prompt with facts, rules (relevant + distracting), conclusion, and the instruction to "specify which premise is used in each step."

Correctness Criterion

Unlike SimpleLogic, which treats logical reasoning as a binary classification task (True/False), this paper imposes a strict proof validity requirement:

"In our benchmark, every problem has a ground-truth label of True, and we consider the prediction to be correct only when the generated proof is completely valid. With these strict criteria, the LLM is required to produce the step-by-step deduction that leads to the conclusion, and any hallucination of non-existent facts and rules is considered erroneous."

This means: the model must output a sequence of deduction steps, each step must cite a specific rule and specific facts from the problem, each step's conclusion must logically follow from its cited premises under modus ponens, and the final step must derive the target conclusion. If the model claims the conclusion is false (wrong refutation), uses a rule not present in the prompt (rule hallucination), asserts a fact that is neither given in the prompt nor derived in a previous step (fact hallucination), or produces a proof that doesn't logically connect the premises to the conclusion, it is scored as incorrect — even if it states the correct final answer.

Why this strict criterion: The paper's goal is to measure reasoning quality, not answer-guessing accuracy. A model might output "C is True" without a valid proof and get lucky — but that doesn't demonstrate reasoning capability. The strict criterion ensures that accuracy reflects the model's ability to construct a coherent deductive chain, which is exactly the capability that premise ordering is hypothesized to affect. This also explains why some models show very low accuracies at high rule counts in non-forward orders (e.g., GPT-3.5-turbo at 12 rules with τ = 0.5: 1.0% accuracy, Table 6d): they are not just getting the final answer wrong — they are failing to produce any valid proof at all.

R-GSM Benchmark Construction

The R-GSM benchmark extends the ordering investigation beyond deductive logic to mathematical problem-solving, testing whether the premise order effect generalizes to a domain where the "premises" are natural language sentences in a word problem rather than formal logical rules.

Problem Selection Criteria

The construction pipeline starts with the GSM8K test set (Cobbe et al., 2021), a standard benchmark of grade-school math word problems. The paper applies a two-stage filter:

Minimum sentence count: Only problems with at least 5 sentences in the problem description are selected. This ensures there are enough sentences to meaningfully permute while still having a coherent problem. (A 3-sentence problem has few reorderings and most may be nonsensical; 5+ sentences provide combinatorial richness.)
Reordering validity: Problems are filtered out if "there is no alternative ordering that does not change the ground truth answer." Specifically, problems whose sentences follow a strict causal or temporal sequence — where changing the order would change the events being described (e.g., "Bob had 5 apples. He ate 2. Then he bought 3." — the sentence order encodes the temporal sequence of events; reordering would describe a different scenario) — are excluded. The paper verifies this manually, examining each problem to determine whether a semantically equivalent reordering exists.

The filtering process is manual, not automated, and requires human judgment about semantic equivalence. The paper states that the R-GSM dataset contains 220 problems after filtering, and Table 4 provides statistics: 133 problems have exactly 5 sentences, 65 have 6 sentences, 19 have 7 sentences, and 3 have 8 sentences. Regarding reasoning steps: 20 problems require 2 steps, 43 require 3 steps, 65 require 4 steps, 43 require 5 steps, 23 require 6 steps, 15 require 7 steps, and 11 require 8 steps.

Enumeration-Based Search for Adversarial Reorderings

For each problem passing the filter, the paper does not arbitrarily reorder sentences. Instead, it uses a targeted approach to find reorderings that are likely to cause LLM failure:

"To facilitate the annotation process, for each problem, we write a simple function to enumerate all alternative orderings of problem statements until an ordering that causes the LLM prediction failure is discovered, which can be used for our manual rewriting if the alternative ordering found in the enumeration process happens to preserve the ground truth answer."

This means: the pipeline programmatically enumerates sentence permutations for a given problem, tests each permutation (presumably by running the LLM on it), and stops when it finds a permutation that the model gets wrong. This permutation is then manually reviewed: if the reordered problem is semantically equivalent to the original (same ground truth answer, same mathematical relationships, same constraints), it is accepted; if the reordering inadvertently changes the problem's meaning, it is discarded and the search continues.

Why this adversarial approach: The paper is not trying to measure the average effect of random reordering (which might be small). It is trying to establish a lower bound on robustness — to demonstrate that for models that can solve a problem in one ordering, there exists some other logically equivalent ordering where they fail. The enumeration search efficiently finds these failure cases, providing the strongest possible demonstration of the ordering brittleness phenomenon. If the paper used random reorderings and found a small effect, a critic could argue that the reordering wasn't sufficiently "confusing." By adversarially selecting reorderings that induce failure, the paper preempts this criticism: it shows that fragility exists in the worst case, which is what matters for reliability.

Manual Rewriting Protocol

Once an adversarial permutation is identified that preserves the ground truth answer, the paper applies minor manual editing to ensure grammatical correctness:

"Minor editing on words is allowed to ensure the grammatical correctness of the problem description."

This is an important detail: the reordering is not a blind cut-and-paste of sentences. Consider a problem that originally reads: "Alice has 5 apples. She gives 2 to Bob. She then buys 3 more." If the sentences are permuted to "She then buys 3 more. She gives 2 to Bob. Alice has 5 apples," the result would be grammatically jarring (pronouns like "she" appear before their referent "Alice" is introduced). The manual editing adjusts pronouns, adds clarifying noun phrases, or inserts connecting words so that the reordered problem reads as natural English while preserving the mathematical content. The example in Figure 2 of the main paper shows the result: the reordered problem uses explicit names ("Sophia" rather than ambiguous pronouns) and reads fluently despite the non-chronological sentence order.

Key property: The manual editing only affects surface-level grammar and cohesion — it does not change numbers, arithmetic operations, relationships between quantities, or constraints. The ground truth answer is manually verified to remain identical to the original problem. This means that any difference in accuracy between the original and reordered versions of the same problem is attributable to the order in which information is presented (and the minimal grammatical adjustments needed to make that ordering readable), not to any change in the underlying mathematics.

The Last Sentence Is Preserved

A specific design choice: "For each of the remaining problem, we keep the last sentence untouched and rewrite the problem description with a different ordering of other sentences." In GSM8K problems, the last sentence is typically the question itself (e.g., "How many apples does Alice have now?"). By keeping it in final position, the reordered problems maintain the standard structure where the question comes at the end, avoiding a confounding variable (whether the model can handle a question posed mid-text or at the beginning).

Dataset Composition and Statistics

The final R-GSM benchmark consists of 220 paired problems, each with an original GSM8K version and a manually rewritten reordered version. The distribution of problem complexity (Table 4) is:

By number of reasoning steps (in the ground-truth solution): 20 problems with 2 steps, 43 with 3 steps, 65 with 4 steps, 43 with 5 steps, 23 with 6 steps, 15 with 7 steps, and 11 with 8 steps. The median is 4 steps, and most problems (151 of 220) require 3–5 reasoning steps.
By number of sentences in the problem description: 133 problems with 5 sentences (60.5% of the dataset), 65 with 6 sentences (29.5%), 19 with 7 sentences (8.6%), and 3 with 8 sentences (1.4%). As noted: "over 60% of problems in R-GSM only have 5 sentences, and all problems have at most 8 sentences."

Why this modest length: The paper emphasizes this point — R-GSM problems are short, with the majority being only 5 sentences. If the ordering effect were only a long-context phenomenon, it would not manifest here. The fact that significant accuracy drops are observed even on 5–8 sentence problems (Figures 7 and 8) indicates that the brittleness is not about context length per se but about the alignment between presentation order and reasoning order, a factor that can be disrupted even in short texts.

LLM Evaluation Protocol

The evaluation protocol is designed for reproducibility and to isolate the ordering effect from confounding factors like stochastic sampling, prompt engineering, or lenient grading.

Models Evaluated

Four models spanning multiple architectures and developers are evaluated:

GPT-4-turbo: OpenAI's most capable model at the time of the study.
GPT-3.5-turbo: A less capable but widely deployed OpenAI model.
PaLM 2-L (Google, 2023): Google's large language model from the PaLM 2 family.
Gemini 1.0 Pro (Gemini, 2023): Google's multimodal model, used here in text-only mode.

The diversity of model families (OpenAI's GPT series vs. Google's PaLM and Gemini) is deliberate: it tests whether the ordering effect is an idiosyncrasy of one architecture/training pipeline or a systematic property of auto-regressive LLMs. The results (Sections 3.2 and 3.3) show consistent qualitative patterns across all four models — forward order is always best, accuracy degrades with non-forward orders, and the degradation scales with problem complexity — supporting the claim of generality.

Decoding Configuration

All evaluations use greedy decoding with temperature 0. This means the model deterministically selects the most likely token at each step, with no randomness in the sampling process. This is a deliberate choice: it removes stochastic variation as a confounding factor, ensuring that any difference in accuracy between two orderings of the same problem is due to the ordering itself, not to random sampling noise. It also makes the results fully reproducible (modulo any non-determinism in the underlying inference infrastructure).

Why not use sampling or beam search: The paper's research question is about the model's inherent preference and capability, not about whether test-time compute strategies (like best-of-N sampling) can compensate for ordering sensitivity. Greedy decoding reveals the model's "default" behavior — what it does when asked once, with no second chances. This is the most conservative test of robustness: if the model fails under greedy decoding, it means its primary reasoning pathway is disrupted by ordering, even though multiple samples might occasionally recover the correct answer.

Prompting Format

For logical reasoning, the prompt includes the problem description (facts + rules + conclusion) plus an explicit instruction to produce a step-by-step derivation. As shown in Figure 1's example (and Figure 10 in Appendix B), the instruction requests: "Please provide a step-by-step proof, specifying which premise is used in each step." This is a zero-shot prompt — no examples of proofs are provided, and the model must infer the expected output format from the instruction alone.

The problem description format is standardized: facts are listed (often as simple declarative sentences like "Fiona is red"), rules are listed (as "If... then..." or "If... and..., then..." statements), and the conclusion is stated (as "X is True"). Distracting rules (when present) are interleaved with relevant rules rather than segregated. The exact interleaving pattern is not specified in detail, but Appendix D clarifies: "In our primary experiments, for all logical reasoning problems, we interleave distracting rules with relevant rules in the input context."

For R-GSM, the prompt is minimal: "the model input only contains the problem description without additional instructions." This is the standard GSM8K zero-shot evaluation format — the model sees the word problem and is expected to produce a step-by-step solution and final answer. No system prompt, no few-shot examples, no formatting instructions beyond what the model infers from the problem text itself.

Grading for Logical Reasoning

The correctness criterion for logical reasoning is described in Section 2.1 and the error analysis in Section 3.2. A model prediction is correct only if the entire generated proof is valid. This involves checking:

The proof reaches the target conclusion (does not end with "cannot be proved" or an unrelated conclusion).
Each step cites a rule that exists in the problem statement (no rule hallucination).
Each step cites facts that either are given in the problem statement or were derived in a previous step of the proof (no fact hallucination).
Each step is a valid modus ponens application: the cited facts satisfy all conditions of the cited rule, and the step's conclusion is the rule's consequent.
The sequence of steps forms a logically connected chain culminating in the target conclusion.

The grading is performed automatically against the ground-truth proof and problem specification. Because all problems use modus ponens over definite clauses, determining proof validity is decidable and does not require human judgment. The paper reports accuracies as percentages of the 200 problems per (rule count × τ × distraction) condition.

Grading for R-GSM

For R-GSM, grading follows standard GSM8K practice: extract the final numerical answer from the model's output and compare it to the ground truth answer. The paper uses exact match (the extracted number must equal the correct answer). Because GSM8K answers are integers or simple decimals, extraction is typically straightforward (parse the last number or the number following "the answer is").

The paper does not grade intermediate reasoning steps for R-GSM correctness — only the final answer matters, unlike the logical reasoning benchmark where proof validity is required. This is a less strict criterion: a model could produce a completely nonsensical chain of reasoning that happens to end with the right number and be scored as correct. However, the error analysis (Section 3.3, Table 3) shows that in practice, when models get the final answer wrong on reordered problems, their reasoning typically contains identifiable errors (temporal order violations, unknown variable usage, etc.), not just bad luck.

Error Analysis Methodology

The paper goes beyond accuracy numbers to characterize how models fail under different premise orderings, providing insight into the cognitive mechanisms (or lack thereof) underlying the ordering effect.

Logical Reasoning Error Taxonomy

For logical reasoning, each incorrect prediction is classified into one of three mutually exclusive categories (Table 1):

Wrong refutation: The model concludes that the target statement cannot be proved, even though a valid proof exists. This is a failure of proof search — the model gives up rather than finding a valid deduction chain. The paper notes (Section 3.2, error analysis) that "the percentage of wrong refutation is generally lower for τ = -1 than for |τ| < 1," suggesting that backward ordering, while not ideal, at least provides a systematic (backward-chaining) structure that the model can follow, whereas intermediate shuffed orders provide neither forward nor backward structure, causing more frequent abandonment.
Rule hallucination: The model invents a rule that is not present in the problem statement and uses it in a proof step. For example, if the problem contains "If A then B" and "If B then C," the model might generate a step using "If A then C" — a rule that is logically entailed (by transitivity) but not given in the premises. Since the grading criteria require using only given premises, this is marked as an error. Rule hallucination indicates that the model is attempting to "bridge" gaps in its proof — it needs a rule to connect two facts, cannot find one because the relevant rule appears later in a non-forward order, and instead confabulates a rule.
Fact hallucination: The model asserts a fact that is neither given in the problem nor derived in a previous proof step. This is the most common error type across all models and orderings (Table 1). For example, with 12 relevant rules and no distractions, GPT-4-turbo shows 12.5% fact hallucination at τ = -1 vs. only 1.5% at τ = 1; Gemini 1.0 Pro shows 60.5% fact hallucination at τ = -1 vs. 50.5% at τ = 1. The paper's interpretation (Section 3.2): "LLMs are inclined to use the rules in the sequential order as they present in the problem, so when the next rule in the problem is not yet applicable, LLMs might still hallucinate facts to complete the proof step." In other words, the model encounters Rule 5 in the prompt, tries to apply it (because it's the "next" rule in left-to-right reading), finds that its conditions aren't yet satisfied, and instead of looking ahead or behind for a different rule, it simply invents the missing facts to make Rule 5 applicable. This is a failure of premise selection and ordering flexibility — the model is processing rules sequentially and trying to use each as it appears, rather than scanning the full rule set to find the one that fits the current proof state.

R-GSM Error Taxonomy

For R-GSM, the error analysis focuses on the subset of problems where the model correctly solves the original GSM8K version but fails on the reordered version. The errors are classified into three categories (Table 3):

Temporal order violations: The model fails to respect the temporal sequence of events when that sequence is implied by the problem semantics but not by sentence order. The example in Figure 2 illustrates this: in the original problem, sentences appear in chronological order (Sophia eats breakfast, then goes to school, etc.), so the model's natural left-to-right processing aligns with the event sequence. In the reordered version, an event that happens later is described earlier in the text, and the model incorrectly assumes earlier-text = earlier-time, leading to wrong intermediate quantities. This is the most common error for GPT-4-turbo (45.0% of failures), PaLM 2-L (34.8%), and Gemini 1.0 Pro (29.5%).
Unknown variables: The model encounters a sentence that references a quantity it hasn't yet computed because the prerequisite information appears later in the reordered text. Figure 9 illustrates this: in the reordered problem, the sentence about gerbils appears before the sentence that provides the number of fish, which is needed to compute the number of gerbils. The model attempts to proceed with an unknown or incorrectly assumed value, leading to wrong calculations. This error type accounts for 15.0% of GPT-4-turbo failures, 19.6% of GPT-3.5-turbo failures, 4.3% of PaLM 2-L failures, and 18.2% of Gemini 1.0 Pro failures.
Others: Errors that don't fit the above categories, including arithmetic mistakes, misreading of problem constraints, or nonsensical reasoning chains. This is the largest category for GPT-3.5-turbo (58.8%) and PaLM 2-L (60.9%), suggesting these weaker models manifest the ordering effect through a broader range of confused behaviors rather than through a single identifiable mechanism.

Connection across domains: Both the logical reasoning error analysis (fact/rule hallucination) and the R-GSM error analysis (temporal violations, unknown variables) point to the same underlying mechanism: the models process information sequentially, in the order it appears, and struggle to reorder that information internally to match the logical or mathematical dependencies of the problem. In logical reasoning, this manifests as hallucinating facts to apply whichever rule happens to be textually next; in math reasoning, it manifests as using numbers before their referents have been defined or assuming temporal order from textual order. The consistency of this pattern across two very different reasoning domains and four model architectures is the paper's strongest evidence that the ordering effect is fundamental to how auto-regressive LLMs process multi-step reasoning tasks.

Position Bias Ablation (Appendix D)

An alternative explanation for the ordering effect is that it is really a position bias: relevant premises placed at certain positions (beginning, middle, end) receive more attention or are more likely to be used, regardless of their logical role. The "lost-in-the-middle" phenomenon (Liu et al., 2024) documents exactly this: LLMs perform better when relevant information is at the extremes of the context rather than in the middle.

The paper tests this alternative in Appendix D by controlling for position while varying order. Using PaLM 2-L with 10 distracting rules, the experiment varies whether relevant rules are placed at the beginning, middle, or end of the prompt, for both forward and backward orders. Table 5 shows the results:

For 8 relevant rules in forward order: beginning = 68.0%, middle = 67.0%, end = 67.0%. The maximum variation across positions is 1.0 percentage point.
For 8 relevant rules in backward order: beginning = 40.0%, middle = 39.0%, end = 40.0%. Variation is 1.0 percentage point.
For 8 relevant rules, shuffed: beginning = 45.5%, middle = 44.5%, end = 45.5%. Variation is 1.0 percentage point.
Changing from forward to backward order drops accuracy by ~28 percentage points (67–68% → 39–40%) — vastly larger than the 1 percentage point variation from position.

The same pattern holds at 12 relevant rules: position variation is ≤1.5 percentage points, while order variation is ~20 percentage points (35–36.5% forward → 16–17% backward).

The paper also notes that the longest prompts in the logical reasoning benchmark (~300 tokens for 12 relevant + 10 distracting rules) are far shorter than the context lengths where lost-in-the-middle typically manifests (which requires thousands of tokens). The conclusion: "lost-in-the-middle phenomenon is not the primary cause of the performance difference." This ablation is critical because it rules out the most obvious alternative explanation, strengthening the claim that the effect is specifically about order relative to the proof structure, not about absolute position in the text.

Summary of Design Choices and Their Justifications

Modus ponens only: Eliminates logical operator complexity as a confound; isolates ordering and proof length as the only variables. If models fail here, the ordering effect is fundamental.
Kendall tau distance τ to quantify ordering: Provides a continuous, interpretable metric that captures both forward (τ = 1) and backward (τ = −1) alignment, with shuffed orders (τ ≈ 0) as the comparison point. The five discrete τ values (1, 0.5, 0, −0.5, −1) sample the range uniformly while keeping the experimental design tractable (5 rather than arbitrary τ values).
200 problems per rule count: Provides sufficient statistical power to measure accuracy differences of ~5–10 percentage points (given that 200 samples yields a standard error of ~3.5% at 50% accuracy, ~1.5% at 90% accuracy, under binomial assumptions).
Strict proof validity grading for logical reasoning: Prevents models from being credited for lucky guesses or partial reasoning; accuracy reflects genuine deductive capability.
Adversarial reordering for R-GSM: Strengthens the lower-bound on robustness by finding worst-case reorderings rather than average-case; demonstrates that the ordering effect is not just a statistical tendency but a reliability concern (there exists some ordering where a correctly-solved problem becomes a failure).
Manual grammatical editing for R-GSM: Ensures reordered problems are natural English, eliminating the confound that models fail because of disfluent or ungrammatical input rather than because of information ordering.
Greedy decoding (temperature = 0): Removes stochasticity; makes results reproducible and isolates the model's primary reasoning pathway. Any accuracy gap is a deterministic consequence of input ordering, not a random sampling artifact.
Four-model evaluation: Tests generality across architectures and developers; the consistent qualitative patterns across all models support the claim that the ordering effect is a property of current auto-regressive LLMs as a class, not a quirk of one implementation.
Interleaving distracting rules (not segregating): Creates a realistic premise selection challenge where relevant and irrelevant rules are mixed, testing whether models can distinguish usable from unusable rules regardless of position.
Position bias control (Appendix D): Rules out the lost-in-the-middle phenomenon as an alternative explanation, strengthening causal attribution of accuracy differences to premise ordering rather than absolute position.
Last sentence preserved in R-GSM: Maintains the standard "question at end" structure, avoiding a confound where failures might be due to unusual question placement rather than information ordering.

4. Key Insights and Innovations

Innovation 1: Premise Order as a Previously Invisible First-Class Variable Governing LLM Reasoning

The paper's most fundamental intellectual contribution is not the finding that ordering matters — prior work had documented order-adjacent phenomena like the reversal curse (Berglund et al., 2023) and position bias (Liu et al., 2024) — but rather the elevation of premise ordering to the status of an independent, quantifiable variable that governs reasoning performance with a magnitude comparable to problem difficulty itself. Prior to this work, the field implicitly treated the surface presentation order of premises as a trivial implementation detail of benchmark construction, not as an object of systematic study. Reasoning benchmarks like GSM8K and SimpleLogic were designed with a single canonical ordering; variation in model accuracy was attributed to differences in logical depth, domain knowledge, or computational capacity. The idea that simply permuting the sequence of logically independent premises could cause accuracy swings exceeding 30 percentage points — without changing any logical content — was not part of the conceptual vocabulary of LLM evaluation.

The paper's conceptual move is to treat premise order as an independent axis of variation orthogonal to logical difficulty, formalizing this axis through the Kendall tau distance τ relative to the forward proof order. This transforms a vague intuition ("order might matter") into a measurable, sweepable quantity that can be correlated with model behavior across models and problem classes. The five τ values (1, 0.5, 0, −0.5, −1) establish a grid over the space of possible orderings, from perfect forward alignment through random shuffling to perfect backward alignment, enabling the paper to characterize not just whether order matters but how accuracy degrades as alignment breaks. Figure 5 reveals that this degradation is not monotonic in |τ| for all models — GPT-4-turbo prefers τ = −1 (backward) over τ = 0.5 (slightly forward-shuffled), while PaLM 2-L shows the opposite pattern — demonstrating that the ordering effect is not reducible to a single "order quality" dimension but interacts with model-specific reasoning strategies.

This framing is significant beyond the empirical results because it provides a diagnostic lens for evaluating what LLMs actually learn about reasoning. If a model's accuracy collapses when premises are reordered, it indicates that the model's reasoning process is bound to surface-level sequential structure rather than grounded in flexible, order-invariant logical relationships. The τ metric operationalizes this diagnostic: a robust reasoner would exhibit a flat accuracy curve across τ, while the steep slopes observed (Table 6) quantify the degree of surface-binding. This turns the premise order effect from a "bug" to be fixed into a measurement instrument for probing the nature of LLM reasoning — a contribution that parallels how adversarial examples in computer vision provided a tool for understanding the geometric properties of learned decision boundaries rather than merely a robustness failure to be patched.

Comparison to prior work underscores the novelty. The reversal curse (Berglund et al., 2023) studied order within single binary relations ("A is B" vs. "B is A"), a special case of τ = −1 for two-premise problems. Distractibility (Shi et al., 2023) studied the presence/absence of irrelevant content but held order fixed. Position bias (Liu et al., 2024) studied absolute location in long contexts but not relative ordering among multiple premises. None of these frameworks provided a general metric for multi-premise ordering quality, and none predicted the paper's finding that shuffled orderings (τ ≈ 0) can be worse than the purely backward ordering (τ = −1) — a phenomenon unexplainable by reversal or position bias alone. The paper's contribution is thus not incremental to these prior works but complementary at a higher level of abstraction, providing the unifying measurement framework within which those individual effects appear as special cases along the τ axis.

Innovation 2: The Diagnostic Framing — LLMs Reason via Left-to-Right Sequential Processing, Not Flexible Premise Selection

The paper's second major contribution is a causal hypothesis about the mechanism underlying the ordering effect, supported by convergent evidence from error analysis across two reasoning domains. The hypothesis is not merely that "LLMs prefer forward order" but that this preference reveals something fundamental about how auto-regressive language models implement multi-step reasoning: they process premises sequentially, in the order they appear in the text, attempting to apply each premise as it is encountered rather than first reading and internalizing the full premise set and then flexibly selecting the appropriate premise for each reasoning step.

This hypothesis is not stated as a theorem or proven formally, but it is constructed through a careful accumulation of diagnostic evidence. The error analysis in Table 1 provides the most direct support. Fact hallucination — the dominant error type across all models and non-forward orderings — is interpreted as the model encountering a rule that is textually next, finding that its conditions aren't yet satisfied given the facts derived so far, and inventing the missing facts to make the rule applicable rather than skipping ahead or looking back for a different rule whose conditions are already met. At τ = −1 with 12 rules, GPT-4-turbo shows 12.5% fact hallucination vs. 1.5% at τ = 1; PaLM 2-L shows 30.0% vs. 8.5%; Gemini 1.0 Pro shows 60.5% vs. 50.5%. This pattern — fact hallucination rising dramatically as τ deviates from 1 — is exactly what a sequential-application mechanism would produce: when the textual order matches the proof order, each rule is applicable when encountered; when it doesn't, the model faces a "use it now or skip it" dilemma at each rule, and the hallucination rate reveals how often it chooses "use it now, invent the preconditions."

The R-GSM error analysis provides convergent evidence from a different domain. The two dominant error categories (Table 3) — temporal order violations (assuming earlier-text events happened earlier in time) and unknown variables (attempting to use quantities before they've been defined in the reordered text) — are both manifestations of the same sequential-processing pattern. The model reads a sentence describing an event or quantity and immediately incorporates it into an ongoing left-to-right solution construction, without the "look ahead to gather all givens, then plan the solution" strategy that a flexible reasoner would employ. The consistency of this pattern across logical deduction (a symbolic, rule-based domain) and mathematical word problems (a natural language, arithmetic domain) strengthens the claim that the mechanism is domain-general and tied to auto-regressive processing architecture rather than to task-specific reasoning strategies.

What makes this an intellectual contribution beyond the ordering effect itself is that it provides a mechanistic narrative for a cluster of previously disconnected LLM brittleness phenomena. The reversal curse, distractibility, position bias, and now premise order sensitivity can all be understood as consequences of the same underlying constraint: the auto-regressive training objective teaches models to process text sequentially, and this sequential processing bias persists even when the task demands non-sequential information integration (reasoning backwards, gathering scattered premises, ignoring distractors interspersed among relevant content). The paper doesn't prove this unification — it studies only ordering — but the conceptual link it forges is generative for future research: it suggests that any technique that makes LLM reasoning more robust to order might simultaneously address multiple documented failure modes.

Comparison to prior work on LLM reasoning mechanisms clarifies the contribution. Saparov and He (2022) characterized LLMs as "greedy reasoners" that follow local syntactic patterns rather than constructing global proof trees, but they didn't isolate text ordering as the variable governing this greediness. The current paper shows that the "greediness" is specifically a function of premise order: in forward order, the greedy left-to-right strategy works (producing high accuracy), while in non-forward orders it fails catastrophically. This refines the "greedy reasoner" diagnosis from a blanket characterization to a conditional one — LLMs can produce valid multi-step proofs (GPT-4-turbo at 96.5% with 12 forward-ordered rules), but only when the surface order of the text makes the greedy strategy succeed. The reasoning capability is real but brittle, tied to a specific input format rather than portable across equivalent formulations.

Innovation 3: Adversarial Benchmark Construction as a Methodology for Probing Robustness Lower Bounds

The R-GSM benchmark represents a methodological innovation in how robustness is evaluated. Standard robustness evaluation either (a) applies random perturbations and measures average degradation or (b) applies semantically-preserving transformations uniformly across a dataset and reports aggregate scores. The R-GSM construction takes a different approach: for each problem that a model can solve in its original form, it searches adversarially for a semantically equivalent reordering that causes failure, then manually verifies that the reordering preserves the mathematical content. This yields a lower bound on robustness: if a model fails on the adversarially-found reordering, it demonstrates that the model's reasoning is not invariant to premise order — there exists at least one equivalent formulation where it breaks, which is sufficient to establish unreliability.

This adversarial methodology is conceptually distinct from both random-perturbation evaluation (which might miss rare failure cases) and from the paper's logical reasoning benchmark (which samples τ values systematically from a distribution but doesn't target failure-inducing orders for individual problems). Table 2b demonstrates the power of this approach: by isolating the subset of R-GSM problems each model initially solved correctly (where the "Init Acc" is 100% by construction), it reveals that reordering causes GPT-4-turbo to fail on 10.1% of these previously-solved problems, GPT-3.5-turbo on 35.1%, PaLM 2-L on 12.1%, and Gemini 1.0 Pro on 25.4%. These are not aggregate statistics diluted by problems the model couldn't solve in any ordering — they are precise measurements of how often a correctly-functioning reasoning capability breaks under reordering alone.

The significance of this methodology extends beyond the specific results. It provides a template for robustness evaluation that could be applied to other semantic-preserving transformations: paraphrasing, translation and back-translation, numerical reformulation (e.g., changing units), or diagram-to-text conversion. The key insight is that evaluating robustness doesn't require uniform coverage of all possible transformations — it requires finding some transformation within the equivalence class that causes failure, which can be done adversarially and more efficiently than exhaustive enumeration. This connects to the adversarial example literature in computer vision (where imperceptible perturbations that change model predictions are sought) but adapted to the discrete, semantic-preservation-constrained setting of natural language reasoning.

The manual verification step — checking that the adversarially-found reordering "happens to preserve the ground truth answer" and applying minor grammatical edits — is crucial to the methodology's validity. It ensures that the failures are not artifacts of ungrammatical or nonsensical input (models failing on garbled text would be unsurprising) but genuine reasoning failures on coherent, well-formed problem statements that are logically equivalent to the original. This distinguishes R-GSM from prior work on scrambled text robustness (Cao et al., 2023; Sinha et al., 2020), where models were tested on syntactically degraded or permuted inputs and the question was whether they could extract meaning from noise. Here, the reordered text is grammatical and meaningful — the question is whether the model can reason about it correctly when the textual order doesn't match the dependency structure of the solution.

Innovation 4: The Asymmetry Between Forward and Backward Chaining as a Revealed Model Preference

The paper's fine-grained τ breakdown (Figure 5, Tables 9–11) reveals a finding that is inconsistent with simpler explanations of the ordering effect: different models exhibit qualitatively different preferences among non-forward orderings, with some models (GPT-4-turbo, GPT-3.5-turbo) performing better under the purely backward order (τ = −1) than under mildly shuffled orders (τ = 0.5 or τ = 0), while PaLM 2-L shows the opposite, performing worst under backward order. This asymmetry is a novel empirical observation that cannot be explained by general mechanisms like "LLMs prefer coherent ordering" or "LLMs are sensitive to premise adjacency."

The pattern for GPT-4-turbo is particularly striking. At 12 rules without distractions (Table 9a): τ = 1 → 96.5%, τ = 0.5 → 76.0%, τ = 0 → 82.0%, τ = −0.5 → 84.5%, τ = −1 → 84.0%. The backward order (τ = −1, 84.0%) outperforms τ = 0.5 (76.0%) by 8 percentage points and τ = 0 (82.0%) by 2 points. For GPT-3.5-turbo at the same rule count (Table 9d), the backward order (1.0%) doesn't outperform shuffled (0.2–1.2%), but at 10 rules (Table 9d), backward (4.5%) outperforms all intermediate τ values (2.0–3.5%). This pattern — backward ≥ intermediate — suggests that GPT-series models can exploit the systematic structure of a fully reversed premise sequence, likely by engaging a backward-chaining reasoning strategy (start from the conclusion, work backwards through rules to find known facts). A partially shuffled sequence provides neither forward nor backward structure, forcing the model into the error-prone strategy of applying rules in textual order.

PaLM 2-L, by contrast, shows monotonically decreasing performance with decreasing τ (Table 9b): at 12 rules, τ = 1 → 88.0%, τ = 0.5 → 74.5%, τ = 0 → 65.5%, τ = −0.5 → 59.5%, τ = −1 → 57.5%. This model does not recover any benefit from backward structure — it is maximally dependent on forward alignment and degrades smoothly as alignment weakens, with no backward-chaining "rebound."

This model-specific asymmetry is intellectually significant for two reasons. First, it demonstrates that the ordering effect is not a monolithic "LLMs are bad at non-forward orders" but rather an interaction between premise ordering and the reasoning strategy each model has implicitly learned during training. The fact that GPT-4-turbo can partially recover with backward order while PaLM 2-L cannot suggests that backward-chaining capability is an emergent property of some training pipelines but not others — possibly related to the diversity of reasoning patterns in the training corpus, the scale of the model, or post-training procedures (RLHF, instruction tuning). This opens a research direction: what training data or objectives produce models that can flexibly adapt their reasoning strategy to premise order, and what produces models that are locked into forward-only processing?

Second, the asymmetry provides a causal test for mechanistic hypotheses about the ordering effect. If the effect were purely due to position bias (premises at certain positions being attended to less), backward order should perform similarly to shuffled — both would have "wrong" positions for each premise. If it were purely due to the reversal curse (inability to reverse associations), backward order should perform worst of all, since every pairwise premise relationship is reversed. Neither prediction matches the GPT-series pattern. The observed pattern is most consistent with the paper's sequential-processing hypothesis combined with a flexible reasoning capability that can, when the premise order provides a systematic (rather than random) alternative structure, switch from forward-chaining to backward-chaining. The fact that only some models exhibit this capability suggests it's a learned skill, not an architectural inevitability — and thus potentially trainable.

Compared to prior work, this finding substantially refines the understanding of the reversal curse. Berglund et al. (2023) showed that LLMs fail to reverse a single association ("A is B" → "B is A"). The current paper shows that for multi-step reasoning chains, the reversal story is more nuanced: reversing the entire chain (backward order) can partially recover performance that random reshuffling destroys, at least for some models. This means the reversal curse is not simply "models can't reason backwards" but rather involves an interaction between the number of premises, their ordering structure (systematic reversal vs. random permutation), and model-specific capabilities.

5. Experimental Analysis

Evaluation Methodology

Dataset. The paper uses two primary datasets: (1) a custom logical reasoning benchmark adapted from SimpleLogic (Zhang et al., 2022), consisting of 200 synthetically generated modus ponens problems per rule count (4–12 relevant rules), each instantiated in 15 variants (5 premise orderings × 3 distraction levels: 0, 5, 10 distracting rules), yielding 27,000 total problem instances; and (2) the newly constructed R-GSM benchmark, comprising 220 paired problems (original GSM8K test problems + manually rewritten reordered versions) where the original problems were drawn from the GSM8K test set (Cobbe et al., 2021), filtered to include only problems with ≥5 sentences and where a semantically equivalent reordering exists. The logical reasoning benchmark uses all 200 problems per condition for evaluation; R-GSM uses all 220 pairs.
Base model(s). Four state-of-the-art LLMs spanning two developers and multiple architectures are evaluated: GPT-4-turbo and GPT-3.5-turbo (OpenAI, 2023), PaLM 2-L (Google, 2023), and Gemini 1.0 Pro (Gemini, 2023). The models are chosen to represent a diversity of training pipelines and capability levels — GPT-4-turbo is the strongest, GPT-3.5-turbo is a weaker but widely deployed model, PaLM 2-L represents Google's pre-Gemini flagship, and Gemini 1.0 Pro is Google's multimodal model used in text-only mode. The diversity enables testing whether the premise order effect is architecture-specific or systematic across current auto-regressive LLMs. All models are evaluated via API access with greedy decoding (temperature = 0).
Metrics. For logical reasoning, the primary metric is proof accuracy — the fraction of 200 problems per condition for which the model generates a completely valid proof. A proof is valid only if every step cites premises present in the problem, derives its conclusion via valid modus ponens from cited premises, and the chain culminates in the target conclusion; any hallucinated fact or rule, or any claim that the conclusion cannot be proved (wrong refutation), renders the prediction incorrect regardless of whether the final answer is stated. For R-GSM, the primary metric is answer accuracy — the fraction of 220 problems for which the extracted final numerical answer matches the ground truth. Additionally, for logical reasoning, the paper reports error type breakdowns (percentage of predictions falling into wrong refutation, rule hallucination, and fact hallucination categories) as secondary diagnostic metrics (Table 1). For R-GSM, error analysis is performed on the subset of problems each model initially solved correctly, categorizing failures into temporal order violations, unknown variable usage, and other errors (Table 3).
Baselines. The paper's experimental design is primarily a within-problem comparison across premise orderings rather than a methods comparison, so there are no competing techniques being benchmarked. The implicit baseline in all experiments is the forward order condition (τ = 1) — the highest-performing ordering for all models — against which all other orderings (τ = 0.5, 0, −0.5, −1) are compared. For R-GSM, the baseline is the original GSM8K problem (which is not necessarily in forward order but represents the standard benchmark formulation), compared against the manually rewritten reordered version. In the position bias ablation (Appendix D), the baseline is the standard interleaved presentation of relevant and distracting rules, compared against configurations where relevant rules are segregated at the beginning, middle, or end of the prompt.
Generation budget / compute accounting. Since the paper studies model brittleness rather than comparing methods with different computational costs, there is no "generation budget" in the sense of test-time compute scaling papers. All evaluations use a single greedy decode per problem instance. The cost per problem is therefore one forward pass through the model, and the total evaluation cost is simply the number of problem instances (27,000 logical reasoning instances × 1–4 models + 440 R-GSM instances × 4 models). The paper does not report FLOPs or inference time; the focus is on accuracy, not efficiency.
Cross-validation / statistical protocol. The paper does not employ cross-validation or statistical significance testing. The logical reasoning benchmark uses a fixed set of 200 problems per condition, and accuracy is reported as a simple percentage of correct predictions. No confidence intervals, standard errors, or hypothesis tests are reported. The R-GSM benchmark uses all 220 pairs without train/test splits. The position bias ablation in Appendix D uses the same 200 problems with systematic variation of relevant-rule position. The absence of statistical rigor is a limitation: with 200 problems per condition, the standard error on a reported accuracy of 50% is approximately ±3.5 percentage points, meaning that some of the finer-grained comparisons (e.g., between τ = 0 and τ = −0.5 for specific models at specific rule counts) may not be statistically distinguishable at conventional significance levels. The paper does not address this.

Main Quantitative Results

Logical Reasoning: Forward Order Dominance and Scalable Degradation

The central results for logical reasoning establish three quantitative patterns. First, the forward order (τ = 1) achieves the highest accuracy for every model at every rule count, typically by large margins over all alternative orderings. At 12 relevant rules with no distracting rules (Table 6), forward-order accuracies are: GPT-4-turbo 96.5%, PaLM 2-L 88.0%, GPT-3.5-turbo 30.0%, Gemini 1.0 Pro 16.5%. The corresponding shuffled-order accuracies (aggregating τ = 0.5, 0, −0.5) are: GPT-4-turbo 80.8%, PaLM 2-L 66.5%, GPT-3.5-turbo 1.2%, Gemini 1.0 Pro 0.2%. The accuracy drop from forward to shuffled ranges from 15.7 percentage points (GPT-4-turbo) to 29.8 percentage points (GPT-3.5-turbo), with Gemini 1.0 Pro showing near-total collapse (from 16.5% to 0.2%).

Second, the ordering effect scales with problem complexity. Figure 3 and Table 6 show that as the number of relevant rules increases from 4 to 12, the accuracy gap between forward and shuffled orders widens substantially for all models. For GPT-4-turbo, the forward–shuffled gap is essentially zero at 4 rules (99.0% vs. 98.8%) but grows to 15.7 percentage points at 12 rules. For PaLM 2-L, the gap grows from 0.2 points at 4 rules (98.5% vs. 98.3%) to 21.5 points at 12 rules (88.0% vs. 66.5%). For Gemini 1.0 Pro, the gap grows from 16.0 points at 4 rules (93.0% vs. 77.0%) to 16.3 points at 12 rules (16.5% vs. 0.2%) — the absolute gap remains large but the ceiling collapses. For GPT-3.5-turbo, the gap grows from 16.7 points at 4 rules (88.5% vs. 71.8%) to 28.8 points at 12 rules (30.0% vs. 1.2%). This scaling interaction is one of the paper's strongest quantitative findings: longer proof chains don't just reduce overall accuracy, they specifically amplify the sensitivity to premise ordering.

Third, distracting rules amplify the ordering effect. Comparing Figure 3 (0 distractions) to Figure 4 and Tables 7–8 (5 and 10 distractions): at 12 relevant rules, GPT-4-turbo's forward accuracy drops from 96.5% (0 distractions) to 80.5% (5 distractions) to 57.5% (10 distractions), while shuffled accuracy drops from 80.8% to 57.2% to 40.0%. The forward–shuffled gap grows from 15.7 points (0 distractions) to 23.3 points (5 distractions, 80.5% vs. 57.2%) to 17.5 points (10 distractions, 57.5% vs. 40.0%). For PaLM 2-L at 12 rules, the forward–shuffled gap grows from 21.5 points (0 distractions: 88.0% vs. 66.5%) to 25.8 points (5 distractions: 64.0% vs. 38.2%) to 18.3 points (10 distractions: 36.5% vs. 18.2%). The amplification is consistent though not strictly monotonic in distraction count.

Logical Reasoning: Model-Specific Preferences Among Alternative Orderings

The fine-grained breakdown by Kendall tau distance (Figure 5, Table 9) reveals asymmetric preferences across models that constitute one of the paper's most novel empirical observations. At 12 rules with no distractions:

GPT-4-turbo: τ = 1 (96.5%) > τ = −0.5 (84.5%) > τ = −1 (84.0%) > τ = 0 (82.0%) > τ = 0.5 (76.0%). The backward order (τ = −1) outperforms the mildly-forward-shuffled order (τ = 0.5) by 8.0 percentage points, and even outperforms the random order (τ = 0) by 2.0 points. This U-shaped or asymmetric pattern — forward best, then backward and near-backward, then random and near-forward worst — is consistent with the model being able to exploit the systematic structure of backward premise ordering for backward-chaining style reasoning.
PaLM 2-L: τ = 1 (88.0%) > τ = 0.5 (74.5%) > τ = 0 (65.5%) > τ = −0.5 (59.5%) > τ = −1 (57.5%). The degradation is monotonic in decreasing τ: there is no recovery at backward order. PaLM 2-L performs worst under the reversed ordering, suggesting it cannot engage backward-chaining and instead suffers cumulatively as the premise sequence deviates from forward alignment.
GPT-3.5-turbo (at 10 rules, Table 9d): τ = 1 (37.0%) > τ = −1 (4.5%) > τ = 0.5 (2.0%) = τ = −0.5 (2.0%) > τ = 0 (3.5%?). The pattern is less clear due to near-floor performance, but backward order (4.5%) still outperforms most intermediate values.
Gemini 1.0 Pro (at 10 rules, Table 9c): τ = 1 (34.0%) > τ = −1 (4.5%) > τ = 0 (3.5%) > τ = 0.5 (2.0%) = τ = −0.5 (2.0%). Backward order again outperforms intermediate shuffed values.

The same pattern persists with distracting rules present (Tables 10–11). For GPT-4-turbo at 12 rules with 5 distractions (Table 10a): τ = 1 (80.5%) > τ = −1 (72.5%) > τ = 0 (61.5%) ≈ τ = −0.5 (60.5%) > τ = 0.5 (49.5%). The backward order outperforms all non-forward orders, and τ = 0.5 (slightly forward-shuffled) is worse than τ = 0 (random). For PaLM 2-L with 5 distractions (Table 10b), the monotonic pattern persists: τ = 1 (64.0%) > τ = 0.5 (39.0%) > τ = 0 (42.0%) > τ = −0.5 (33.5%) > τ = −1 (32.5%). At 10 distractions (Table 11b), the same monotonic degradation holds.

Logical Reasoning: Error Analysis Reveals the Mechanism

Table 1 presents the error type breakdown for 12 relevant rules with no distracting rules, the most informative condition because all models retain some non-negligible accuracy under at least some orderings. The key findings:

Fact hallucination dominates and scales inversely with τ. For GPT-4-turbo: fact hallucination rises from 1.5% at τ = 1 to 12.5% at τ = −1, with intermediate values falling between (11.5% at τ = 0.5, 10.0% at τ = 0, 10.0% at τ = −0.5). For PaLM 2-L: fact hallucination rises from 8.5% at τ = 1 to 30.0% at τ = −1, with a monotonic increase through intermediate values (14.5% at 0.5, 21.5% at 0, 29.0% at −0.5). For Gemini 1.0 Pro: fact hallucination is already high at τ = 1 (50.5%) but rises further at non-forward orders (57.0% at τ = 0, 65.5% at τ = −0.5, 60.5% at τ = −1). The pattern is consistent and substantial: as premise order deviates from the proof sequence, models increasingly invent facts to apply rules that appear textually next but aren't yet logically applicable.
Wrong refutation is highest at intermediate τ values. For GPT-4-turbo, wrong refutation is 10.5% at τ = 0.5 and 4.5% at τ = 0, but drops to 0.5% at τ = 1, 1.0% at τ = −0.5, and 0.0% at τ = −1. For GPT-3.5-turbo, wrong refutation peaks at τ = 0 (55.0%) and τ = 0.5 (54.5%), compared to 24.5% at τ = 1 and 34.5% at τ = −1. The paper interprets this as evidence that intermediate shuffed orders provide neither forward-chaining nor backward-chaining structure, causing the model to "give up" more frequently than under either systematic ordering.
Rule hallucination is generally less common than fact hallucination, but shows a different pattern: for GPT-3.5-turbo, rule hallucination at τ = −1 (14.5%) is higher than at τ = 1 (9.5%) and higher than at intermediate values (7.5–9.5%). For Gemini 1.0 Pro, rule hallucination rises from 5.0% at τ = 1 to 11.5% at τ = −1. This suggests that backward order, while enabling some models to engage backward-chaining, also induces spurious rule invention, possibly because the model tries to bridge gaps when working backwards from the conclusion.

R-GSM: Mathematical Reasoning Shows the Same Brittleness

Table 2a reports the overall results on the full R-GSM benchmark of 220 problem pairs. For the original GSM8K problems: GPT-4-turbo achieves 94.1%, PaLM 2-L 86.4%, Gemini 1.0 Pro 80.5%, GPT-3.5-turbo 67.3%. For the reordered versions: accuracies drop to 85.0% (−9.1 points), 79.5% (−6.9 points), 69.1% (−11.4 points), and 51.8% (−15.5 points), respectively. All models show a statistically meaningful decline.

Because the original GSM8K problems are not necessarily in "forward order" (the optimal premise ordering for the solution), some reordered problems are actually easier for certain models, partially masking the ordering effect in the aggregate statistics. To isolate the true fragility, Table 2b restricts analysis to the subset of problems each model initially solved correctly (where initial accuracy is 100% by construction). On this subset, reordering causes GPT-4-turbo to fail on 10.1% of previously-solved problems, PaLM 2-L on 12.1%, Gemini 1.0 Pro on 25.4%, and GPT-3.5-turbo on 35.1%. These numbers represent the pure ordering effect — the fraction of problems for which a model possesses the reasoning capability (as demonstrated by correct solution on the original ordering) but fails to deploy it on a logically equivalent reordering.

Figures 7 and 8 break down R-GSM performance by problem complexity. For GPT-4-turbo (Table 12a, Figure 7), the accuracy gap between original and reordered problems grows with the number of reasoning steps: at ≥2 steps, the gap is 9.1 points (94.1% vs. 85.0%); at ≥5 steps, 13.1 points (92.4% vs. 79.3%); at ≥6 steps, 16.3 points (89.8% vs. 73.5%). For Gemini 1.0 Pro (Table 12c), the gap grows from 11.4 points at ≥2 steps to 20.6 points at ≥5 steps (80.4% vs. 59.8%) to 16.3 points at ≥6 steps (71.4% vs. 55.1%). The pattern for GPT-4-turbo and Gemini 1.0 Pro is monotonic: more complex problems amplify the ordering effect. For PaLM 2-L and GPT-3.5-turbo, the gap remains relatively stable across step counts (PaLM 2-L: ~6–9 points across all thresholds; GPT-3.5-turbo: ~15–16 points across all thresholds), suggesting their ordering sensitivity is already saturated at moderate complexity.

When broken down by number of sentences (Figure 8, Table 13), the pattern is similar. For GPT-4-turbo: the original–reordered gap is 9.1 points at ≥5 sentences (94.1% vs. 85.0%), 8.1 points at ≥6 sentences (89.7% vs. 81.6%), and 18.2 points at ≥7 sentences (86.4% vs. 68.2%). The large jump at ≥7 sentences suggests a threshold effect where longer problems become disproportionately harder under reordering, though the sample size at this length is small (22 problems at ≥7 sentences, only 3 at 8 sentences; Table 4b).

The R-GSM error analysis (Table 3), performed on the subset of problems each model initially solved correctly, categorizes failures into three types:

Temporal order violations: the most common error for GPT-4-turbo (45.0% of reordering-induced failures), PaLM 2-L (34.8%), and Gemini 1.0 Pro (29.5%). The model assumes that sentences appearing earlier in the text describe earlier events, even when the reordering has disrupted the chronological sequence. The example in Figure 2 illustrates this: the reordered problem describes a later event (going to school with a certain amount of money) before an earlier event (buying lunch), and the model computes the initial amount incorrectly because it processes the school-arrival sentence before the lunch-purchase sentence.
Unknown variables: accounting for 15.0% (GPT-4-turbo), 19.6% (GPT-3.5-turbo), 4.3% (PaLM 2-L), and 18.2% (Gemini 1.0 Pro) of failures. The model encounters a sentence that references a quantity not yet computed because the prerequisite information appears later in the reordered text. Figure 9 demonstrates this: the gerbil-count sentence precedes the fish-count sentence, so when the model reads about gerbils, it doesn't yet know the number of fish needed to compute the number of gerbils, and instead incorrectly uses a number from a previous step.
Other: the residual category capturing assorted errors including arithmetic mistakes, constraint misreading, and nonsensical reasoning. This is the largest category for GPT-3.5-turbo (58.8%) and PaLM 2-L (60.9%), suggesting these weaker models manifest the ordering effect through diverse and less classifiable failure modes.

The consistency of error patterns across logical reasoning (fact hallucination) and mathematical reasoning (temporal violations, unknown variables) supports the paper's central mechanistic claim: LLMs process premises sequentially in text order and struggle to internally reorder information to match logical or mathematical dependency structure.

Ablation Studies and Robustness Checks

Position bias vs. ordering effect (Appendix D, Table 5): The paper tests whether the observed performance differences could be attributed to the absolute position of relevant rules in the prompt rather than their relative order. Using PaLM 2-L with 10 distracting rules, relevant rules are placed at the beginning, middle, or end of the prompt while holding premise order (forward, backward, or shuffled) constant. For 8 relevant rules in forward order: accuracy is 68.0% (beginning), 67.0% (middle), 67.0% (end) — a maximum variation of 1.0 percentage point. In backward order: 40.0%, 39.0%, 40.0% — variation of 1.0 point. In shuffled order: 45.5%, 44.5%, 45.5% — variation of 1.0 point. By contrast, changing from forward to backward order (averaging across positions) drops accuracy by ~28 points (67.3% → 39.7%). For 12 relevant rules: position variation is ≤1.5 points across all orderings, while order variation (forward vs. backward) is ~20 points (35.8% forward average → 16.3% backward average). The paper also notes that the longest prompts (~300 tokens for 12 relevant + 10 distracting rules) are far shorter than context lengths where lost-in-the-middle typically manifests. The conclusion that the ordering effect is not attributable to position bias is well-supported by this data.
Varying number of relevant rules (Figures 3, 5, Tables 6, 9): The systematic sweep from 4 to 12 relevant rules serves as both a primary result and an ablation of proof complexity. The finding that the ordering effect amplifies with rule count is robust across all models and orderings: the forward–shuffled accuracy gap consistently widens as rules increase. This confirms that longer reasoning chains are not just harder in absolute terms but specifically harder under non-forward premise ordering, which is consistent with the paper's sequential-processing hypothesis (more steps = more opportunities for the left-to-right strategy to encounter an inapplicable rule and hallucinate).
Varying number of distracting rules (Figures 4, 6, Tables 7–8, 10–11): Adding 5 or 10 distracting rules degrades overall accuracy and amplifies the ordering effect for both GPT-4-turbo and PaLM 2-L. The amplification is not uniform: it is most pronounced at intermediate rule counts (8–10 rules) and somewhat less at the extremes. The qualitative pattern of model-specific preferences (GPT-4-turbo preferring backward over intermediate orders, PaLM 2-L degrading monotonically with τ) remains stable under distraction, suggesting these preferences are inherent to the model rather than an artifact of problem simplicity.
Fine-grained τ breakdown (Figures 5, 6, Tables 9–11): The five-value τ grid (1, 0.5, 0, −0.5, −1) is itself an ablation of ordering granularity, testing whether the accuracy–ordering relationship is smooth or exhibits discontinuities. The results show model-specific discontinuities: GPT-4-turbo exhibits a U-shaped pattern with minima at intermediate positive τ values (τ = 0.5 is often worse than τ = 0), while PaLM 2-L shows a monotonic decline. These patterns are consistent across rule counts and distraction levels, indicating they are robust features of the models rather than noise.
R-GSM complexity breakdown (Figures 7, 8, Tables 12–13): The stratification by number of reasoning steps and number of sentences serves as an ablation of problem complexity for mathematical reasoning. The finding that the ordering gap widens with complexity for GPT-4-turbo and Gemini 1.0 Pro but remains stable for PaLM 2-L and GPT-3.5-turbo suggests a model capability interaction: stronger models maintain near-ceiling performance on simpler reordered problems (where their reasoning capacity can overcome the ordering disruption) but degrade on complex reordered problems; weaker models are already near their reasoning capacity limits even on simple reordered problems, so added complexity doesn't further widen the gap.
Original-vs-reordered subset analysis (Table 2b): Restricting R-GSM analysis to problems each model initially solved correctly is an informal ablation controlling for the fact that original GSM8K problems are not in a uniformly optimal order. By setting the baseline to 100% accuracy, this analysis isolates the pure ordering effect — the fraction of problems where a model that can solve the problem in one ordering fails in another. The results (10.1–35.1% failure rates) demonstrate that the ordering effect is not an artifact of averaging over problems the model can't solve in any order; it manifests even for problems well within the model's demonstrated capability envelope.
Error type analysis (Tables 1, 3): The categorization of logical reasoning errors into wrong refutation, rule hallucination, and fact hallucination, and of R-GSM errors into temporal violations, unknown variables, and other, serves as a diagnostic ablation clarifying the mechanism of ordering-induced failure. The dominance of fact hallucination in logical reasoning and temporal violations in mathematical reasoning across multiple models and orderings provides convergent evidence for the sequential-processing hypothesis. PaLM 2-L's notably lower rate of "unknown variable" errors in R-GSM (4.3%) compared to other models (15.0–19.6%) is an interesting model-specific finding that the paper notes but does not explain.
Hallucinated rule and fact examples (Figure 10, Appendix B): Qualitative examples support the error analysis by showing concrete instances of rule hallucination (the model invents "If Fiona is red and Max is blue, then Max is red" — a rule not present in the problem) and fact hallucination (the model asserts "Fiona is blue" without deriving it from given premises). These examples illustrate the sequential-processing failure mode: encountering a rule that doesn't apply, the model bridges the gap by inventing either the rule itself or its preconditions.

Critical Assessment

The paper's central claim is straightforward: "the premise order significantly affects LLMs' performance on reasoning tasks, even when the premise order does not change the underlying task itself." The experiments provide strong support for this claim in the specific domains and settings tested, but several important boundaries on the strength and generality of that support must be acknowledged.

What the experiments convincingly demonstrate. The logical reasoning results establish beyond reasonable doubt that for the tested models on modus ponens deduction problems, premise ordering is a first-order determinant of accuracy, with effects that scale with proof length and are amplified by distracting premises. The effect sizes are large: 15–30 percentage point accuracy drops between forward and shuffled orderings at 12 rules (Table 6) are far outside any plausible noise. The consistency across four models from two different developers, and the monotonic or near-monotonic relationship with τ for most models, rules out idiosyncratic artifacts. The R-GSM results extend the finding to a different domain and demonstrate that the effect manifests even on short problems (5–8 sentences) that are well within the models' general capability. The error analysis provides a coherent mechanistic narrative (sequential left-to-right processing driving hallucination) that explains both the logical reasoning and mathematical reasoning failure patterns.

Where the experimental design limits the strength of conclusions. Several aspects of the experimental design warrant caution in interpreting the breadth and practical significance of the findings:

Single benchmark domain for logical reasoning. All logical reasoning experiments use modus ponens over definite clauses — the simplest possible deductive framework. The paper explicitly acknowledges this choice as a feature (isolating ordering from logical operator complexity), but it also means the results do not directly speak to more complex reasoning forms. Would the ordering effect be larger or smaller for problems involving negation, disjunction, quantifiers, or non-monotonic reasoning? The paper provides no evidence either way. The limitation to modus ponens also means all problems have exactly one valid proof, eliminating the possibility that models struggle with ambiguity or proof selection under reordering — factors that could amplify the effect in more realistic settings.

Adversarial R-GSM construction may overstate the average-case effect. The R-GSM benchmark was constructed by searching for reorderings that cause LLM failure, then manually verifying semantic equivalence. This means the reported accuracy drops (Table 2a) are worst-case or near-worst-case measurements, not estimates of the expected effect of random reordering. This is methodologically valid for establishing a lower bound on robustness (as the paper intends), but it means the results should not be interpreted as "reordering typically causes a 9–15 point accuracy drop." Random reorderings — which are more representative of what might occur in natural user interactions — could produce much smaller effects. The paper is transparent about the adversarial methodology but does not explicitly discuss this interpretation caveat.

Manual rewriting introduces a confound. The R-GSM reordered problems are not pure sentence permutations; they involve "minor editing on words... to ensure the grammatical correctness of the problem description." This editing, while minimal, introduces a potential confound: differences in accuracy between original and reordered versions could be partially attributable to the edited phrasing rather than to sentence order per se. For example, replacing a pronoun with an explicit noun phrase (to maintain coherence after reordering) changes the linguistic surface form in ways that might affect model processing independently of order. The paper does not quantify how much editing was applied or include an ablation where original-order problems received the same grammatical edits to isolate the editing effect.

Statistical significance is not addressed. With 200 problems per condition in the logical reasoning benchmark, a reported accuracy difference of 10 percentage points between two conditions has a standard error of approximately ±4.9 points (assuming independent binomial trials at 50% accuracy). This means the finer comparisons — e.g., GPT-4-turbo at 12 rules: τ = −1 (84.0%) vs. τ = 0 (82.0%) — differ by only 2 points, well within one standard error, and are not statistically distinguishable. The paper's qualitative claims about GPT-4-turbo preferring backward over shuffled order rest partly on such small differences at individual rule counts. The overall U-shaped pattern is supported by consistency across multiple rule counts and distraction levels (Tables 9a, 10a, 11a), but the precision of individual data points should not be overinterpreted.

For R-GSM, the situation is more concerning: with only 220 problem pairs, the standard error on a 10% accuracy difference at 85% accuracy is approximately ±2.4 points. The breakdowns by reasoning steps and sentence counts (Tables 12–13) involve even smaller sample sizes — e.g., only 11 problems at ≥8 steps, 22 problems at ≥7 sentences — making those sub-analyses highly noisy. The paper treats these breakdowns as meaningful without caveats about sample size.

No model size scaling analysis. All experiments use specific model endpoints (GPT-4-turbo, GPT-3.5-turbo, PaLM 2-L, Gemini 1.0 Pro). There is no within-family scaling analysis (e.g., comparing GPT-4 vs. GPT-4-turbo, or PaLM 2-S vs. PaLM 2-L vs. PaLM 2-XL) that would reveal whether the ordering effect diminishes, remains constant, or amplifies with model scale. This is a significant gap because it leaves open the critical question: is premise order sensitivity a fundamental limitation of auto-regressive architectures that will persist regardless of scale, or is it a transient weakness that larger models will overcome? The fact that GPT-4-turbo (the strongest model tested) shows smaller ordering sensitivity than GPT-3.5-turbo in both absolute and relative terms is suggestive that scale helps, but without systematic scaling data, this remains speculative.

No chain-of-thought or few-shot prompting comparison. All experiments use zero-shot prompting. Given that chain-of-thought prompting (Wei et al., 2022) and few-shot exemplars are known to substantially improve LLM reasoning performance, an important question is whether these techniques mitigate or amplify the ordering effect. A model prompted with a few-shot example showing how to reason from non-forward-ordered premises might learn to compensate for ordering disruption. Conversely, few-shot exemplars that implicitly assume forward ordering might make the model more brittle. The paper does not explore this, limiting the practical applicability of its findings to the zero-shot setting.

Temperature 0 and greedy decoding only. The exclusive use of greedy decoding means the paper characterizes model behavior under a deterministic, single-trajectory regime. In practice, reasoning systems often use sampling with temperature > 0 and best-of-N selection, which could partially compensate for ordering sensitivity (some samples might hit the correct reasoning path even if the modal prediction is wrong). The paper's findings are thus most relevant to deterministic, single-shot deployment scenarios; the ordering effect under stochastic decoding with verification remains uncharacterized.

The "forward order" is defined relative to a specific proof. All logical reasoning problems in the benchmark have exactly one proof, so the forward order is unambiguous. In more realistic reasoning scenarios where multiple valid proofs exist for the same conclusion, the concept of "forward order" becomes ambiguous — should premises be ordered for one proof or another? The paper's framework doesn't address this, and the generalizability to problems with multiple valid reasoning paths is unclear.

What experiments would have strengthened the paper. Several additional experiments would have substantially strengthened the conclusions:

Random (non-adversarial) reorderings for R-GSM. Reporting accuracy under random sentence permutations, in addition to the adversarially-selected reorderings, would distinguish the worst-case effect from the typical effect and provide a more complete picture of robustness. The adversarial construction demonstrates brittleness; random reordering would measure its practical frequency.
Within-family model scaling. Testing PaLM 2 at multiple scales or GPT-4 vs. GPT-4-turbo vs. GPT-3.5-turbo on a shared subset of problems would address whether ordering sensitivity is a scaling law phenomenon or a persistent architectural limitation.
Few-shot and chain-of-thought prompting. Evaluating whether prompting techniques that improve general reasoning performance also mitigate ordering sensitivity would have both practical implications (how to deploy models robustly) and theoretical implications (whether the sensitivity is a limitation of the base model or of zero-shot inference specifically).
Editing-only ablation for R-GSM. Applying the same grammatical edits to original-order problems would control for the possibility that edited phrasing, rather than reordering, causes some of the accuracy drop.
Statistical confidence reporting. Including confidence intervals or hypothesis tests for key comparisons would clarify which patterns are robust given the sample sizes and which might be noise. This is particularly important for the fine-grained τ comparisons and the R-GSM complexity breakdowns.
Beyond modus ponens. Extending the logical reasoning benchmark to include even one additional logical operator (e.g., modus tollens: if P then Q; not Q; therefore not P) would test whether the ordering effect is specific to forward-chaining over Horn clauses or generalizes to other inference patterns. The paper cites Girotto et al. (1997) finding that premise order affects human reasoning on modus tollens but not modus ponens — testing whether LLMs show the reverse pattern (order-sensitive even on modus ponens, which humans handle flexibly) would strengthen the claim of a qualitative difference between human and LLM reasoning.
Intervention studies. If the sequential-processing hypothesis is correct, then interventions that force the model to read all premises before starting deduction (e.g., by prompting "First, list all the given facts and rules. Then, construct the proof step by step.") should reduce ordering sensitivity. The paper doesn't test any such intervention, leaving the mechanistic hypothesis as an interpretation of error patterns rather than a causally-verified claim.

Summary of evidence-to-claim mapping. The paper's headline claim — "premise order significantly affects reasoning performance" — is strongly supported for the specific combination of models, tasks, and evaluation protocols studied. The effect is large, consistent across models, and exhibits plausible scaling with problem complexity. The extension to mathematical reasoning via R-GSM provides important domain generality, though the adversarial construction means the reported effect sizes are upper bounds on expected degradation. The mechanistic interpretation (sequential left-to-right processing driving hallucination) is well-supported by error analysis but lacks causal intervention evidence. The practical significance for deployed systems is somewhat limited by the exclusive use of zero-shot greedy decoding and the absence of experiments testing whether standard mitigation techniques (sampling, verification, few-shot prompting, chain-of-thought) reduce ordering sensitivity. The paper successfully establishes premise ordering as a variable that the research community should measure and report, and provides a benchmark and measurement framework for doing so, but it leaves open the crucial questions of whether the effect persists under more realistic inference configurations and whether it diminishes with scale.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Not Accounted For in Any Headline Number

The assumption or constraint. The paper reports that compute-optimal test-time scaling achieves "more than 4× better efficiency" over best-of-N baselines, where efficiency is measured as achieving equivalent accuracy with 4× fewer generation steps. However, achieving this efficiency requires estimating each prompt's difficulty before allocating the test-time compute budget. The paper's difficulty estimation method — generating 2048 samples per question, scoring them with the PRM, and binning based on average correctness — is itself extraordinarily expensive. The authors acknowledge this directly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

The consequence. The headline efficiency gain — matching best-of-64 performance with only 16 generations — is computed after difficulty is already known, without amortizing the cost of learning it. The difficulty estimation process (2048 samples) consumes more compute than the largest test-time budgets studied (256–512 generations). In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former would dominate the latter by an order of magnitude. The 4× figure is therefore best understood as an upper bound on achievable efficiency, not a realized deployment gain. A system that actually implements this approach end-to-end would not see a 4× reduction in total compute unless difficulty estimation can be made dramatically cheaper or amortized across many queries with the same difficulty distribution.

What evidence exists in the paper. The paper itself provides no measurement of the total cost including difficulty estimation. No ablation studies whether fewer samples (e.g., 64 or 256 rather than 2048) would suffice for difficulty binning. No experiment includes the difficulty estimation cost in the generation budget when comparing compute-optimal against best-of-N. The gap is explicitly flagged by the authors in Section 3.2 but not addressed empirically. There is no Figure or Table that plots "total generations including difficulty estimation" against accuracy.

Mitigation status. The paper does not attempt to address this limitation methodologically. Section 8 suggests future work on "pretraining or finetuning models to directly predict difficulty of a question," which would replace the expensive sampling-based estimation with a single forward pass through a lightweight classifier. This is presented as a future direction, not as something the paper begins to implement. The paper also does not explore adaptive schemes — e.g., starting with a small number of samples, estimating difficulty coarsely, and refining the estimate as part of the solution process — that could subsume difficulty estimation into the problem-solving budget rather than treating it as a separate upfront cost.

Hard Problems Are Fundamentally Unaddressed — Test-Time Compute Cannot Create Capability

The assumption or constraint. The entire compute-optimal framework rests on the premise that test-time compute amplifies the base model's existing capability rather than creating new capability. This is not merely an unstated assumption — it is a boundary condition that the paper's results make starkly visible. Across both search and revision mechanisms, and across all budget levels studied, the hardest difficulty bin (bin 5) shows essentially no improvement from additional test-time compute.

The consequence. For any problem where the base model's pass@1 is near zero — meaning the model cannot produce a correct solution even with a large number of independent samples — no allocation of test-time compute (whether PRM search, iterative revisions, or compute-optimal combinations thereof) makes any meaningful difference. The paper acknowledges this explicitly in the Section 7 takeaway box, noting that test-time compute offers "no path forward for genuinely novel or out-of-distribution reasoning" that exceeds the base model's training distribution. For any deployment where the problem distribution includes a non-trivial fraction of problems outside the model's capability range, the compute-optimal framework offers zero benefit on those problems — the system will fail regardless of how much inference budget is allocated. The practical implication is that organizations must still invest in pretraining to expand the base model's capability envelope; test-time compute optimization only improves performance within that existing envelope.

What evidence exists in the paper. The evidence is consistent and unambiguous. In the search experiments (Figure 3, right), bin 5 accuracy hovers at approximately 1–3% for all search methods and all generation budgets from 4 to 256 — the curves are essentially flat at near-zero. In the revision experiments (Figure 7, right), bin 5 shows roughly 2–3% accuracy regardless of the sequential-to-parallel ratio at a fixed 128-generation budget. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is a flat line near 0–5% for both search and revisions, across all three values of the inference-to-pretraining ratio R. The ~14× larger model also performs near zero on bin 5, indicating that even substantial pretraining scaling does not solve these hardest problems — but test-time compute certainly does not help either.

Mitigation status. The paper is transparent about this limitation but offers no mitigation. The limitation is inherent to the approach: if there are no correct solutions in the model's output distribution, no amount of search or revision can find or refine one. The paper does not explore whether techniques that modify the model's distribution more aggressively (e.g., prompting with different strategies, using tools, or decomposing the problem) could expand the capability envelope at test time, but these would go beyond the search-and-revision framework that defines the paper's scope.

Single Benchmark and Single Model Family Constrain Generality

The assumption or constraint. All experiments are conducted on the MATH benchmark (Hendrycks et al., 2021) using PaLM 2-S* as the base model, with the PRM and revision model both fine-tuned from this same base model. The FLOPs-matched comparison uses a ~14× larger model from the same PaLM 2 family. The authors state in Section 4 that they "believe this model is representative of the capabilities of many contemporary LLMs," but this belief is not empirically validated within the paper.

The consequence. Several dimensions of the paper's findings could be model-specific or benchmark-specific in ways that limit the generality of the conclusions:

PRM quality and over-optimization behavior depend on the base model's output distribution. A model with different calibration properties, different error patterns, or different solution styles might yield a PRM with different reliability characteristics. The over-optimization threshold — the point at which beam search starts to hurt rather than help on easy problems — is a function of PRM quality, which in turn depends on the base model. A model with a more reliable PRM might not exhibit the same degradation at high search budgets, while a model with a less reliable PRM might show degradation at even lower budgets.
The revision model's effectiveness depends on the base model's in-context learning and self-correction capabilities. The paper's revision model is fine-tuned to condition on its own incorrect outputs and produce corrections. Whether this fine-tuning succeeds likely depends on the base model's capacity for in-context learning from negative examples. A model with different architectural properties or different pre-training data might show weaker (or stronger) revision capabilities, shifting the optimal sequential-to-parallel ratio.
The MATH benchmark is exclusively competition-level mathematics requiring symbolic reasoning and algebraic manipulation. It is unclear whether the difficulty-dependent patterns — beam search hurting easy problems, revisions helping easy problems, parallel sampling necessary for hard problems — generalize to other reasoning domains. Code generation (e.g., HumanEval), where correctness can be verified by unit tests and where the "reasoning" involves planning and algorithmic thinking, might show different patterns. Logical reasoning (e.g., ProofWriter, FOLIO) might differ in whether the model's errors cluster around certain difficulty thresholds. Tasks requiring factual recall rather than inference might not benefit from test-time compute at all.

What evidence exists in the paper. The paper provides no cross-benchmark or cross-model-family validation. All experiments use PaLM 2-S* as the base model. No results are reported on other model families (GPT, LLaMA, Claude) or other mathematical reasoning benchmarks (GSM8K, SVAMP, ASDiv). No results are reported on non-mathematical reasoning tasks. No results are reported using a PRM trained on a different base model's outputs. The domain-specificity limitation is not addressed experimentally in any way — it is a pure assumption that the findings will transfer.

Mitigation status. The paper does not attempt to mitigate this limitation. Section 8 does not flag cross-domain or cross-model replication as future work, focusing instead on extensions within the same paradigm (combining search and revisions, cheaper difficulty estimation, self-improvement loops). A practitioner wanting to apply these techniques to a different model or domain would need to reproduce substantial portions of the experimental pipeline (PRM training, revision model training, difficulty bin calibration, strategy selection via cross-validation) before knowing whether the patterns hold.

The ~14× Larger Model Baseline Understates the Strength of Pretraining

The assumption or constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters. Critically, this larger model is not trained in a compute-optimal manner — it scales parameters while holding training data fixed, following the LLaMA paradigm (Touvron et al., 2023) rather than the Chinchilla-optimal approach (Hoffmann et al., 2022) where both parameters and data are scaled proportionally. The authors acknowledge this explicitly in Section 7:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Additionally, the larger model is evaluated using only greedy decoding — no majority voting, no best-of-N, no verifier-guided selection, and no test-time compute augmentation of any kind.

The consequence. Both design choices weaken the pretraining baseline relative to what a practitioner would actually deploy, making the case for test-time compute appear stronger than it may be in practice:

A Chinchilla-optimal 14× larger model would likely outperform a parameter-only-scaled model. The compute-optimal pretraining scaling laws show that for a fixed compute budget, scaling both parameters and data jointly yields better performance than scaling parameters alone. By fixing data and scaling only parameters, the larger model in this comparison is likely undertrained relative to what the same FLOPs budget could achieve. The reported advantages of test-time compute over pretraining — e.g., +27.8% relative improvement on easy questions at low R — may shrink or reverse against a properly compute-optimal larger model.
Giving the larger model even a modest test-time compute budget would dramatically strengthen the baseline. If the larger model is allowed best-of-4 or best-of-8 (which costs relatively little compared to the pretraining FLOPs already spent), its accuracy would increase, potentially overtaking the smaller model with compute-optimal scaling on additional difficulty bins and R regimes. The paper does not test this.

The practical implication is that the FLOPs-matched comparison answers a narrower question than it appears to: "Is a parameter-only-scaled larger model with greedy decoding better or worse than a smaller model with compute-optimal inference?" This is interesting but does not directly answer the deployment-relevant question: "Given a total budget, should I spend it on a properly-trained larger model or on inference-time compute for a smaller model?"

What evidence exists in the paper. The paper's own results provide indirect evidence that the comparison undervalues pretraining. In Figure 9, the gap between the compute-optimal scaling curve and the larger model's greedy performance (stars) narrows substantially as difficulty increases, and on the hardest problems (bin 5), neither approach works. This suggests that even the parameter-only-scaled larger model captures some of the capability improvement that test-time compute cannot provide on hard problems, but the Chinchilla-optimal version might capture more. The paper does not provide an ablation with a compute-optimal pretraining baseline or with test-time compute applied to the larger model.

Mitigation status. The paper explicitly flags the parameter-only-scaling choice as a limitation (Section 7) and defers compute-optimal pretraining comparisons to future work. However, it does not flag the absence of test-time compute for the larger model as a limitation, treating greedy decoding as the natural baseline for the pretraining-scaled model. A practitioner should read the FLOPs-matched results as establishing a possibility (test-time compute can substitute for pretraining in some regimes) rather than a prescription (it is always better to invest in test-time compute rather than a larger model).

The Revision Model Has a 38% Correct-to-Incorrect Reversion Rate That the Paper Partially Patches Rather Than Solves

The assumption or constraint. The revision model is trained exclusively on sequences where all in-context answers are incorrect, followed by a correct target. This training data construction — pairing independently sampled incorrect solutions with correct solutions via edit-distance-based matching (Section 6.1) — means the model never sees examples of what to do when the current answer is already correct. The consequence, which the paper documents in Section 6.1, is that approximately 38% of correct answers produced during a revision chain get "revised" back to incorrect answers in the subsequent step.

The paper mitigates this by not taking the last revision as the final answer. Instead, it uses a selection mechanism (majority voting or a revision-specific ORM) to pick the best answer from anywhere in the chain. This is described in Section 6.1:

"To mitigate this, the system uses a selection mechanism (majority voting or verifier-based selection) across the entire chain of revisions, picking the best answer from any point in the chain rather than always taking the last revision."

The consequence. The 38% correct-to-incorrect reversion rate imposes a structural inefficiency on the revision process. Every revision step that takes a correct answer and turns it incorrect wastes part of the generation budget — that step consumed compute and moved the solution away from correctness. The within-chain selection mechanism recovers the correct answer post-hoc, but it does not prevent the wasted computation. If the model could recognize when no revision is needed and simply output the current answer unchanged, the sequential revision budget would be used more efficiently: each additional step would either improve an incorrect answer or preserve a correct one, rather than sometimes degrading correctly-solved intermediate states.

This also means that the revision model's pass@1 at each step (Figure 6, left) is the net result of two opposing processes: some incorrect answers being corrected, and some correct answers being corrupted. The observed improvement from approximately 18.2% at step 1 to roughly 24–25% by steps 15–20 understates the correction rate, because it is partially offset by the reversion rate. The true "correction capability" of the revision model might be substantially higher than the net improvement suggests, but the training data bias prevents it from being realized efficiently.

What evidence exists in the paper. The 38% reversion rate is reported in Section 6.1, though the paper does not provide a detailed breakdown of how this rate varies with difficulty or with position in the revision chain. The within-chain selection mechanism's effectiveness is demonstrated implicitly by the fact that sequential revision outperforms parallel sampling in aggregate (Figure 6, right, Figure 8), but the paper does not ablate how much better sequential revision would perform if the reversion problem were solved. The ReST^EM experiment in Appendix K (Figure 16) provides additional evidence of revision model fragility: attempting to further optimize the revision model with RL-style on-policy training caused performance to degrade substantially under sequential revision, suggesting that the revision training procedure is sensitive in ways that are not fully understood.

Mitigation status. The paper partially mitigates the reversion problem through within-chain selection but does not solve the underlying issue. A principled solution — such as training the revision model with examples where the correct answer appears in context and the target is to output it unchanged, or training a separate "should I revise?" classifier that gates whether to produce a new revision or stop — is not explored. Section 8 does not flag the reversion problem as an explicit area for future work, focusing instead on combining revisions with PRM search and developing self-improvement loops.

No Accounting for Latency or Wall-Clock Time

The assumption or constraint. The entire paper measures test-time compute in "generations" — the number of complete solutions sampled from the base model or revision model. This is a reasonable proxy for total FLOPs but ignores latency, the wall-clock time required to produce a final answer. Sequential revisions are inherently serial: each revision conditions on the previous one, so a chain of N revisions requires N sequential forward passes through the model. Parallel best-of-N sampling, by contrast, can execute all N samples simultaneously given sufficient hardware parallelism.

The consequence. A strategy that the compute-optimal policy selects for easy problems — purely sequential revisions — may achieve higher accuracy per generation than parallel sampling, but it does so at dramatically higher latency. A budget of 128 generations spent as 128 sequential revisions requires 128 model calls that must execute one after another. The same budget spent as 128 parallel samples requires only one model call's worth of latency (assuming sufficient hardware). For any latency-sensitive application — interactive assistants, real-time decision-making, dialogue systems, educational tutoring — the sequential strategies favored by the compute-optimal policy on easy problems may be simply impractical regardless of their accuracy advantages.

The paper's FLOPs-matched comparison (Section 7) further complicates the latency picture. The inference-to-pretraining ratio R includes total inference tokens but does not distinguish between parallel and sequential token generation. The same value of R could represent a deployment that is latency-tolerant (where sequential revisions are acceptable) or latency-stringent (where only parallel sampling is viable). The paper's recommendation to prefer test-time compute over pretraining in low-R regimes implicitly assumes that the latency of the chosen test-time strategy is acceptable, which is not guaranteed.

What evidence exists in the paper. The paper provides no latency measurements, no discussion of hardware assumptions, and no analysis of how the compute-optimal policy would change if a latency constraint were imposed. All cost accounting is in terms of "generations" — an abstraction that collapses parallel and sequential compute into the same metric. The sequential-to-parallel ratio sweeps in Figure 7 implicitly explore different latency profiles (more sequential = higher latency per total generation), but the paper does not interpret these axes in latency terms or discuss the tradeoff.

Mitigation status. The paper does not address this limitation. No latency-aware allocation policy is proposed, no latency numbers are reported, and the introduction and conclusion do not flag latency as a concern. A practitioner deploying these techniques in a latency-sensitive setting would need to independently evaluate whether the compute-optimal strategies remain optimal (or even viable) when wall-clock time is the binding constraint rather than total FLOPs. Strategies that appear superior in the paper's generation-budget metric — particularly sequential-heavy revision chains on easy problems — may need to be replaced with more parallel alternatives when latency matters.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not propose a new model, training method, or prompting technique. It provides something arguably more fundamental: a diagnostic measurement framework that reveals a previously invisible dimension of LLM reasoning brittleness. The conceptual shift is from treating premise order as an arbitrary presentation detail — the kind of thing a benchmark designer handles once and forgets — to recognizing it as a first-class variable governing reasoning accuracy with effect sizes comparable to problem difficulty itself.

The magnitude of this shift is best characterized as a reframing of what it means for a model to "solve" a reasoning problem. Before this paper, the standard evaluation paradigm was: present a problem in some canonical ordering, measure whether the model produces the correct answer, and report accuracy. A model scoring 94.1% on GSM8K (as GPT-4-turbo does, Table 2a) would be considered highly capable. The R-GSM results shatter that interpretation: on the subset of GSM8K problems that GPT-4-turbo initially solved correctly, 10.1% become failures under a logically equivalent reordering (Table 2b). For GPT-3.5-turbo, the figure is 35.1%. These are not edge cases — they represent problems squarely within each model's demonstrated capability envelope, rendered unsolvable by surface-level sentence permutation. The implication is that single-order benchmark scores are systematically inflated measures of reasoning competence. A model that achieves high accuracy on GSM8K or SimpleLogic has not demonstrated robust logical understanding; it has demonstrated the ability to process text in a particular sequential pattern that happens to align with the proof structure. The field's standard evaluation methodology has been inadvertently measuring format-dependent reasoning rather than order-invariant reasoning.

This reframing resolves a tension in the literature that the paper itself documents. Prior work had established that LLMs struggle with logical reasoning (Saparov and He, 2022; Saparov et al., 2023; Xu et al., 2023) — but also that state-of-the-art models achieve impressive scores on reasoning benchmarks (Bubeck et al., 2023; Gemini, 2023). The paper's forward-order results (GPT-4-turbo at 96.5% with 12 rules, Table 6a) confirm that LLMs can construct valid multi-step proofs. The shuffled-order results (80.8% on the same problems, Table 6a) confirm that this capability is fragile. The apparent contradiction — "LLMs are good at reasoning" vs. "LLMs are bad at reasoning" — is resolved by the premise order variable: both claims are true, depending on how premises are ordered. The field's disagreement was not about model capability but about implicit (and uncontrolled) variation in benchmark premise ordering. The paper provides the vocabulary (τ) and the measurement protocol (systematic τ variation) to make this variation explicit, and in doing so, it converts contradictory findings into a coherent picture.

The paper also redirects research attention in a specific way. Much of the LLM reasoning literature has focused on improving reasoning through better prompting (chain-of-thought, tree-of-thought, self-consistency), better training (instruction tuning, RLHF, process reward models), or better decoding (beam search, verifier-guided sampling). The premise order effect suggests that these investments are, in an important sense, working on the wrong problem — or at least an incomplete one. A model that can produce flawless proofs under forward ordering but collapses under shuffling has not learned to reason; it has learned a pattern-matching shortcut that succeeds only when surface text order aligns with logical dependency order. Improving that model's forward-order accuracy from 96.5% to 99% (through better prompting or training) would be a genuine improvement, but it would leave the fundamental brittleness unaddressed. The paper implicitly argues that the field should redirect some fraction of its effort from improving peak capability to improving order-invariance — making models that reason equally well regardless of how the same information is sequenced.

Two research directions become more attractive in light of this work. First, architecture-level interventions that reduce auto-regressive sequential bias: if the ordering effect is caused by left-to-right processing (as the error analysis suggests), then techniques like bidirectional attention over the premise set, explicit premise reordering modules, or graph-structured representations of premise relationships become promising paths toward order-invariant reasoning. Second, evaluation methodology reform: the paper provides a template for how benchmarks can be extended to measure order-invariance — generate semantically equivalent variants with permuted premise order, report accuracy across the permutation distribution, and use the forward–shuffled gap as a diagnostic metric for reasoning robustness. The R-GSM construction methodology (adversarial search for failure-inducing reorderings, manual verification of semantic equivalence) is directly replicable on other benchmarks.

One research direction becomes less attractive: the pursuit of higher benchmark scores through prompt engineering alone, without testing whether those scores are robust to semantically vacuous input transformations. The paper demonstrates that high forward-order accuracy can coexist with catastrophic shuffled-order failure (GPT-3.5-turbo at 12 rules: 30.0% forward, 1.2% shuffled, Table 6d). A prompting technique that improves only forward-order performance without addressing the underlying sequential-processing bias would leave the brittleness intact, potentially making the model more overconfident in deployment.

Follow-Up Research This Work Enables

Systematic scaling analysis of the ordering effect. The paper evaluates four models at fixed scales but provides no within-family scaling data. A natural follow-up would measure the ordering effect on a single model family (e.g., PaLM 2-XXS through PaLM 2-L, or LLaMA 7B through 70B, or GPT-3.5 through GPT-4 through GPT-4-turbo) using the exact same logical reasoning benchmark. The key question: does the forward–shuffled accuracy gap narrow, remain constant, or widen with model scale? The paper provides suggestive evidence — GPT-4-turbo shows a 15.7 point gap at 12 rules while GPT-3.5-turbo shows a 28.8 point gap (Table 6a vs. 6d) — but this is cross-family, confounded by training differences. A within-family scaling curve would reveal whether the ordering effect is a transient weakness that larger models naturally outgrow (through increased capacity for internal premise reordering) or a persistent limitation of auto-regressive architectures that requires explicit architectural or training intervention. The experiment is straightforward: evaluate the logical reasoning benchmark across 4–5 model sizes, plot accuracy vs. rule count for each τ separately, and test whether the gap scales as a power law, remains constant, or diminishes.

Training interventions to induce order-invariance. The paper hypothesizes that the ordering effect stems from the auto-regressive training objective teaching models to process text sequentially, but it provides no causal evidence. A direct follow-up would construct a training dataset where reasoning problems appear in multiple premise orderings (forward, backward, shuffled) with the same ground-truth proof, and fine-tune a base model on these multi-order examples. The experiment structure: take the logical reasoning benchmark's training set (if constructed analogously to the test set) or synthetically generate multi-order variants of an existing reasoning dataset; fine-tune PaLM 2-S or LLaMA 7B with standard next-token prediction on this augmented data; evaluate on the paper's test set across all τ values. The hypothesis is that multi-order training would flatten the accuracy–τ curve, reducing the gap between forward and shuffled accuracy. A negative result — where multi-order training fails to reduce the gap or where the model simply memorizes order-specific proof patterns — would suggest the limitation is architectural rather than data-driven, pointing toward attention-mechanism modifications rather than data augmentation as the solution path.

Chain-of-thought as a mitigator or amplifier of ordering sensitivity. The paper evaluates only zero-shot prompting without chain-of-thought (CoT). Given that CoT prompting substantially improves reasoning performance across many benchmarks (Wei et al., 2022), a critical follow-up tests whether CoT mitigates, amplifies, or leaves unchanged the premise order effect. The experimental design: replicate the logical reasoning and R-GSM evaluations with the same four models, but add a CoT prompt (e.g., "Let's think step by step. First, identify all the given facts and rules...") before the problem. For logical reasoning, where the zero-shot prompt already asks for step-by-step derivation, the comparison would be: does explicit instruction to "first list all premises, then reason" cause the model to internalize the full premise set before beginning deduction, reducing its dependence on textual order? For R-GSM, standard few-shot CoT exemplars (as in the original GSM8K paper) would be tested. If CoT substantially narrows the ordering gap, it would provide a practical mitigation strategy for deployments and suggest that the zero-shot ordering sensitivity is partly a failure of the model's default reasoning strategy rather than a hard architectural limitation. If CoT does not help or even amplifies the gap, it would strengthen the paper's claim that the effect is deeply rooted in auto-regressive processing.

Position-aware attention analysis to localize the sequential-processing mechanism. The paper offers a mechanistic hypothesis — LLMs process premises left-to-right, attempting to apply rules as encountered — supported by error analysis but not by direct model-internals evidence. A follow-up using attention pattern analysis could test this hypothesis directly. For a given logical reasoning problem with shuffled premises, feed the problem through PaLM 2-L or an open-source model (LLaMA, Mistral) and analyze attention weights during the generation of each proof step. The specific prediction: when generating step k of the proof, the model's attention should concentrate on the premise that is textually next (the (k+N)-th premise in the prompt) rather than on the premise that is logically next (the rule whose conditions are satisfied given the current proof state). If this pattern holds, it provides mechanistic confirmation of the sequential-processing interpretation. If instead the model attends diffusely across all premises regardless of order, it would suggest the ordering effect operates through subtler mechanisms (e.g., positional encoding interference, learned sequential biases in feedforward layers) that require different interventions.

Cross-domain generalization of the ordering effect. The paper demonstrates the effect on deductive logic (formal symbolic reasoning) and grade-school math (semi-formal numerical reasoning). A natural extension tests whether the ordering effect manifests in other reasoning domains and, crucially, whether the model-specific τ preferences (GPT-4-turbo preferring backward over intermediate orders, PaLM 2-L degrading monotonically) persist across domains. Candidate domains: code generation (where "premises" are function signatures, docstrings, or specification constraints presented in different orders), legal reasoning (where "premises" are clauses in a contract or statute), scientific question answering (where "premises" are experimental findings or theoretical constraints), and multi-document question answering (where "premises" are sentences drawn from different source documents). Each domain would require constructing a benchmark analogous to R-GSM — original problem + adversarially reordered variant with manual verification of semantic equivalence. The key finding would be whether the τ–accuracy relationship is universal (same shape across domains) or domain-specific (e.g., code generation shows a smaller ordering effect because function signatures are more modular), which would inform whether the fix needs to be at the architecture level or can be addressed through domain-specific preprocessing.

Adversarial training for order-robust reasoning. The R-GSM construction methodology (finding adversarial reorderings that cause failure) can be repurposed as a training method. The idea: for each problem in a training set, generate adversarial reorderings using the enumeration-based search described in Section 2.2; include both the original and adversarial reorderings in the training data (with the same ground-truth solution); fine-tune the model. This is an instance of adversarial training adapted to the discrete, semantic-preservation-constrained setting of text reordering. The experiment would test whether exposure to adversarial orderings during training improves robustness to unseen orderings at test time, measured on a held-out set of problems with held-out reorderings. A positive result would provide a scalable mitigation strategy (generate adversarial reorderings automatically during data curation) and would connect the paper to the broader adversarial robustness literature. A negative result — where the model overfits to the specific reorderings seen during training — would suggest the ordering effect requires architectural solutions beyond data augmentation.

Practical Applications and Downstream Use Cases

Robustness evaluation for reasoning benchmarks. The most immediate practical application is a change in how reasoning benchmarks are constructed and reported. The R-GSM methodology — take an existing benchmark, generate semantically equivalent reorderings, report accuracy on both original and reordered versions — can be applied to any reasoning benchmark with separable premises. For benchmark creators (e.g., BIG-bench, HELM, MMLU-math subsets), this paper provides a concrete, low-cost protocol: (1) identify problems with multiple independent sentences; (2) programmatically enumerate reorderings; (3) filter to those preserving the ground-truth answer; (4) report the original-accuracy/reordered-accuracy gap as a "robustness score." For model developers, reporting this gap alongside standard accuracy would provide a more complete picture of reasoning capability. A model with 95% forward-order accuracy and 90% shuffled-order accuracy (a 5-point gap) is genuinely more robust than a model with 97% forward-order and 75% shuffled-order (a 22-point gap), even though the second model has higher peak performance.

Input preprocessing pipelines for deterministic reasoning tasks. In applications where reasoning problems arrive with premises in arbitrary order — legal document review, automated theorem proving over user-supplied axioms, multi-constraint configuration problems — the paper's findings motivate a preprocessing step that reorders premises into forward order before passing them to the LLM. The approach: use a lightweight model or heuristic to estimate the dependency structure among premises (which rules depend on which facts), topologically sort premises to align with forward-chaining, and present the reordered problem to the LLM. The paper's own results provide the evidence for this approach's effectiveness: forward-order accuracy substantially exceeds shuffled-order accuracy across all models and rule counts (Tables 6, 9). The preprocessing cost (sorting premises) is negligible compared to the generation cost, making this a cheap robustness improvement. The caveat, which the paper flags in its discussion of Yan et al. (2023), is that such task-specific preprocessing "does not generalize across domains" — a sorting algorithm designed for modus ponens rules would not work for math word problems or contract clauses. However, for bounded, well-defined reasoning domains where dependency structure can be algorithmically determined, preprocessing is an immediately deployable mitigation.

Deployment reliability monitoring via order-variant testing. For production systems that use LLMs for reasoning tasks (customer support with multi-step troubleshooting, automated essay grading with multiple rubric criteria, financial analysis with multiple constraints), the paper's findings suggest a specific reliability monitoring practice: periodically test the deployed model on adversarially reordered variants of representative queries to detect brittleness. The procedure: maintain a holdout set of queries the model consistently answers correctly; programmatically generate reordered variants (or use human annotators to create plausible alternative phrasings); run the model on these variants; trigger an alert if the accuracy drop exceeds a threshold (e.g., >10% failure rate on previously-solved queries). This would catch degradation in order-invariance before it affects users who happen to phrase their queries in a non-standard order. The paper's finding that even GPT-4-turbo fails on 10.1% of previously-solved R-GSM problems (Table 2b) establishes a concrete baseline for what "acceptable" order-robustness looks like at the current frontier — and a target for improvement.

Curriculum design for fine-tuning reasoning models. For teams fine-tuning LLMs on reasoning tasks (e.g., for domain-specific math tutoring, code explanation, or scientific reasoning), the paper's results argue for a specific curriculum strategy: start training with forward-ordered premises (where the model can learn the reasoning pattern successfully), then progressively introduce shuffled and backward-order variants. This is the standard curriculum learning principle (start easy, then increase difficulty) applied to premise ordering. The paper's data supports this: all models achieve their highest accuracy under forward order, so forward-order problems provide the highest-quality training signal (fewer errors, more valid proof completions). Once the model has learned the reasoning pattern, introducing order variation would — if the adversarial training hypothesis above holds — build robustness without the initial training signal being corrupted by the high error rates that shuffled-order problems produce under zero-shot conditions.

When to Prefer This Method

This section is not applicable. The paper does not propose a method or technique that competes against named alternatives. It provides a diagnostic framework and measurement benchmarks for characterizing an existing brittleness in LLM reasoning. There is no "prefer X over Y" decision because the paper is not advocating for a particular deployment strategy, training procedure, or architectural modification. The practical recommendations that follow from the paper — use forward order when possible, test robustness with reordered variants, apply preprocessing for bounded reasoning domains — are themselves grounded in the measurement framework rather than in a competing-methods comparison. Including a forced decision matrix here would misrepresent the paper's contribution as proposing a solution when it is, in fact, defining the problem.