General365: Benchmarking General Reasoning in Large Language Models Across Diverse and Challenging Tasks

🎯 Pitch

Top LLMs scoring gold medals on math olympiads can't pass a 60% threshold when stripped of specialist knowledge—Gemini-3-Pro tops the GENERAL365 benchmark at just 62.8%. The benchmark peels reasoning away from retrieval, exposing a stark brittleness where models collapse under complex logical constraints, semantic traps, and multi-step strategy problems drawn entirely from K-12 concepts.

1. Executive Summary

This paper introduces GENERAL365, a benchmark specifically designed to assess general reasoning in large language models by decoupling reasoning ability from domain-specific expertise, restricting required knowledge to a K-12 level. Evaluating 26 leading LLMs on 365 seed problems and 1,095 variants across eight challenging categories—including Complex Constraints (interconnected logical predicates requiring global consistency), Semantic Interference (cognitive traps that defy commonsense intuition), and Optimal Strategy (utility-based optimization under constraints)—the paper finds that even the top-performing model, Gemini-3-Pro, achieves only 62.8% accuracy, while the majority of models fail to reach a passing threshold, establishing that contemporary LLMs' reasoning abilities remain heavily domain-dependent and that general reasoning proficiency degrades sharply when problems require suppressing pretrained biases or executing long-term strategic planning rather than retrieving specialized knowledge.

2. Context and Motivation

The Core Gap: LLM Reasoning Evaluations Confound Knowledge Recall with Logical Inference

The motivating problem this paper addresses is a subtle but fundamental measurement error in how the field evaluates LLM reasoning. Current benchmarks overwhelmingly test reasoning within domains that demand deep specialized knowledge: competition mathematics (AIME, IMO), advanced physics (IPhO), graduate-level science (GPQA, Super-GPQA), and professional software engineering (SWE-bench). On these benchmarks, top-tier models have recently achieved expert-level or even gold-medal performance — for instance, scoring at IMO gold medalist levels and achieving near-perfect scores on physics olympiads (Hubert et al., 2025; Huang, 2026).

The problem, as the paper argues in Section 1, is that these results conflate two fundamentally distinct capabilities: knowledge retrieval (accessing specialized facts, theorems, and domain-specific patterns memorized during pretraining) and logical reasoning (manipulating constraints, performing systematic search, suppressing misleading priors, and executing multi-step inference chains). A model that scores 95% on AIME may have learned to recognize problem templates and apply memorized solution strategies from its training corpus rather than demonstrating genuine inferential ability that would transfer to unfamiliar reasoning contexts.

Making this distinction matters because it has direct implications for what we can expect LLMs to do in practice. Real-world reasoning — interpreting a legal contract, diagnosing an unusual equipment failure, resolving a scheduling conflict with dozens of constraints — rarely maps cleanly onto textbook problem types. It demands applying logic to novel situations where the facts are mundane but the constraint interactions are complex. If current LLMs' impressive benchmark scores reflect predominantly domain-specific pattern matching rather than transferable reasoning skill, then deploying them in open-ended real-world reasoning scenarios constitutes a significant over-extrapolation from existing evidence.

The paper formalizes this concern by explicitly contrasting the type of reasoning GENERAL365 targets with the type tested by existing benchmarks:

"These results do not necessarily translate to proficiency in real-world daily reasoning, which often relies on general logic rather than deep domain-specific knowledge."

This is not merely a claim about benchmark construction philosophy — it is an empirical hypothesis: that if you strip away the requirement for specialized knowledge and test reasoning in its "pure" form, current LLM performance will be substantially lower than domain-specific benchmarks suggest.

Why This Matters: The Practical and Diagnostic Significance

Practical significance. The gap between domain-specific and general reasoning performance has immediate consequences for LLM deployment. Organizations adopting LLMs for reasoning-intensive workflows (legal analysis, business strategy, medical diagnosis support, competitive intelligence) are implicitly betting that benchmark scores on math and coding tasks predict performance on their actual problems. If reasoning is heavily domain-dependent — if the model's apparent logical prowess is largely borrowed from memorized domain patterns — that bet may be unsound. GENERAL365 provides evidence that it likely is unsound: across 26 models spanning proprietary and open-source, reasoning-optimized and standard architectures, the best model achieves barely above 60%, and the majority fail to reach even a basic passing threshold (Figure 1). This is a dramatically different picture than what domain benchmarks paint.

Diagnostic significance. The paper's fine-grained taxonomy of eight challenge categories enables something prior benchmarks could not: pinpointing which specific reasoning capabilities break down. Finding that "Semantic Interference" and "Optimal Strategy" are primary bottlenecks (Table 1, Figure 5) tells model developers where to focus improvement efforts. This is analogous to how medical diagnostics break "health" into specific organ function tests — you learn not just that the patient is sick, but what is failing.

Theoretical significance. The paper contributes to an ongoing debate about the nature of LLM reasoning capabilities. One perspective, supported by work on chain-of-thought prompting and scaling laws, holds that reasoning emerges as a general capability from sufficient pretraining scale. An alternative, more skeptical perspective argues that apparent reasoning is largely pattern recognition on training data distributions. GENERAL365 provides evidence relevant to this debate: the sharp performance drop when knowledge requirements are constrained to K-12 level suggests that something beyond domain knowledge is being tested, and that something is not yet solved by scale alone. This constrains theories of how reasoning emerges in LLMs.

Prior Approaches and Their Specific Shortcomings

The paper identifies several existing benchmarks that have attempted to evaluate general reasoning, and its critique of each is specific enough to warrant careful attention.

BBH (BIG-Bench Hard; Suzgun et al., 2023)

BBH selected 23 tasks from the original BIG-Bench collection (Ghazal et al., 2013) where early LLMs failed to surpass human performance, intentionally targeting reasoning dimensions that were challenging at the time. The paper acknowledges BBH's contribution to identifying reasoning gaps but identifies two problems:

Insufficient difficulty due to model advancement. The paper shows in Figure 8 that models like GPT-5-Thinking achieve 92.0% accuracy on BBH, and even GPT-4.1 — a substantially weaker model — reaches 89.4%. This is a saturation problem: when scores cluster in the 90%+ range, the benchmark ceases to distinguish between moderately capable and highly capable models. This is not BBH's fault per se — it was designed for an earlier generation of models — but it means BBH can no longer serve as the primary general reasoning stress test.

Limited diversity from task-level structure. While BBH has 23 distinct tasks, the paper's visualization analysis in Figure 6b reveals that its instances exhibit significant "local collapse" — forming isolated, high-density clusters in embedding space. The quantitative logical similarity analysis (Figure 7) is even more striking: 77.8% of BBH instance pairs receive the maximum similarity score of 5, with a mean similarity of $\mu = 4.71$ on a 0–5 scale. This indicates that within each BBH task, the instances are highly homogeneous — they share essentially identical "reasoning skeletons" differing only in surface features. A model that solves one instance of a BBH task may succeed on all instances of that task via template matching rather than demonstrating general reasoning flexibility.

BBEH (BIG-Bench Extra Hard; Kazemi et al., 2025)

BBEH was designed specifically to address the difficulty saturation of BBH by introducing higher structural complexity. The paper acknowledges BBEH as more challenging than BBH (Figure 8 shows Gemini-3-Pro at 77.0% on BBEH vs. 94.4% on BBH), but identifies that BBEH inherits and even amplifies BBH's diversity limitation:

Extreme template homogeneity. The paper's analysis in Figure 7 shows BBEH with a mean similarity of $\mu = 4.80$ , with 83.5% of instances receiving the maximum similarity score of 5. This is higher than BBH, meaning BBEH's greater difficulty comes from increasing the complexity of a narrow set of reasoning templates rather than from introducing genuinely diverse reasoning challenges. The paper's characterization is precise:

"While existing benchmarks often expand via template-based perturbations—resulting in high structural homogeneity, GENERAL365 preserves a diverse spectrum of independent reasoning patterns."

The concern here is not academic: template-homogeneous benchmarks risk performance inflation through what the paper calls "reasoning shortcuts or hacks" — models may learn to recognize the template (e.g., "this is a state-tracking puzzle with N entities and K constraints") and execute a memorized solution strategy without engaging in the open-ended constraint propagation that characterizes genuine reasoning on novel problems.

ARC-AGI (Chollet et al., 2024)

ARC-AGI takes a fundamentally different approach: it presents visual reasoning tasks requiring models to infer abstract transformation rules from few input-output examples, explicitly targeting generalization to novel patterns. The paper cites ARC-AGI favorably for its emphasis on reasoning over knowledge, but notes that its visual modality and few-shot inference format test a different reasoning capability than the text-based, multi-step constraint reasoning that GENERAL365 targets. ARC-AGI's average difficulty is also noted as having become insufficient with recent model advances (though concrete evidence for this claim is not presented in the paper).

KOR-Bench (Ma et al., 2024)

KOR-Bench evaluates "knowledge-orthogonal reasoning" — logical operations on information where domain knowledge is deliberately made irrelevant. This aligns closely with GENERAL365's philosophy, and the paper acknowledges KOR-Bench as presenting substantial challenges. However, the paper identifies a limitation similar to BBEH:

"They are often constrained by narrow task coverage and template-based instance construction. Such structural rigidity risks performance inflation, where models may leverage spurious reasoning shortcuts."

The critique is consistent across BBH, BBEH, and KOR-Bench: existing general reasoning benchmarks have one or both of two flaws — (1) difficulty saturation from model advancement, or (2) narrow diversity from template-based scalable construction methods that prioritize instance count over reasoning variety.

Domain-Specific Benchmarks (Mathematics, Physics, Programming)

The paper's treatment of domain-specific benchmarks is notably even-handed. It does not claim they are flawed — rather, it argues they answer a different question than what GENERAL365 asks. AIME, IMO, IPhO, GPQA, SWE-bench, and HLE test whether models can solve problems requiring both domain expertise and reasoning. GENERAL365 tests whether models can solve problems requiring reasoning without domain expertise beyond K-12 fundamentals. Both questions matter. The problem is that the field has heavily invested in answering the first question while largely neglecting the second, creating a lopsided picture of LLM reasoning capabilities.

This framing is important because it positions GENERAL365 as complementing rather than replacing existing benchmarks. A model that scores 95% on AIME but 60% on GENERAL365 is not necessarily "bad at reasoning" — it is good at mathematical reasoning and weaker at general logical reasoning, and we need both measurements to understand its capabilities.

How GENERAL365 Positions Itself

The paper's positioning can be understood through four deliberate design choices that directly address the identified gaps:

1. Decoupling reasoning from knowledge by constraining to K-12 scope. This is the paper's central methodological innovation. By restricting required background knowledge to what a typical high school student would know, GENERAL365 ensures that failure on its problems cannot be attributed to lack of specialized expertise. If a model fails a GENERAL365 problem, it failed because it could not execute the required reasoning, not because it lacked access to an advanced theorem or domain-specific fact pattern. This creates a clean signal: performance on GENERAL365 measures reasoning ability per se, uncontaminated by knowledge retrieval.

The paper underscores this through contrast in Section 1:

"The performance on [domain-specific] benchmarks often stems from the models' ability to memorize intricate patterns within specialized training corpora rather than a mastery of fundamental logical rules. This heavy reliance on domain-specific knowledge obscures the assessment of a model's true inferential ability."

This is a strong claim. The paper is arguing that existing benchmark evaluations suffer from a fundamental confound, and that GENERAL365's K-12 constraint provides the appropriate control condition to isolate reasoning from knowledge.

2. Human-curated seed problems guard against template homogeneity. The paper's construction pipeline (Section 2.2) emphasizes that all 365 seed problems are manually crafted by human experts, not LLM-generated from templates. This is expensive — far more expensive than the template-based expansion used by BBH and BBEH — but it is the mechanism that prevents the "local collapse" observed in those benchmarks. The embedding visualizations (Figure 6) and logical similarity distributions (Figure 7) are not merely diagnostic; they are validating evidence that the human-curated approach achieved its intended diversity. The mean similarity of $\mu = 2.16$ on GENERAL365 vs. $\mu = 4.71$ – $4.80$ on BBH/BBEH is the paper's key quantitative evidence that its construction methodology produces qualitatively different — and more diverse — reasoning challenges.

This connects to a deeper point about benchmark construction methodology. Template-based expansion is seductive because it produces large datasets cheaply. But the paper's analysis demonstrates that this efficiency comes at a steep cost in reasoning diversity — an entire benchmark of thousands of instances may test only a handful of distinct logical patterns. GENERAL365's argument is that reasoning diversity per dollar of construction cost is a metric that matters more than raw instance count.

3. High difficulty floor is a feature, not a bug. The paper deliberately designs GENERAL365 to be difficult — it filters out problems easily solvable by top models (Section 2.2) and ensures that even state-of-the-art reasoning models cannot saturate the benchmark. Figure 8 demonstrates this starkly: while Gemini-3-Pro scores 94.4% on BBH and 77.0% on BBEH, it scores only 62.8% on GENERAL365. The gap between BBH and GENERAL365 (~32 percentage points) is larger than the gap between GPT-4.1 and Gemini-3-Pro on GENERAL365 (~36 points), meaning the benchmark difficulty effect dwarfs the model capability effect.

This matters because a benchmark that models can saturate loses all ability to track progress. If BBH scores for top models have compressed into the 90–95% range, improvements in reasoning capability over the next generation of models will be invisible on BBH. GENERAL365's 62.8% ceiling leaves substantial headroom for measuring future advances, making it forward-compatible with model improvements in a way that existing general reasoning benchmarks are not.

4. The eight-category taxonomy enables diagnostic, not just evaluative, use. Rather than reporting a single aggregate score, GENERAL365 provides per-category breakdowns (Table 1). This transforms the benchmark from a pass/fail test into a diagnostic instrument. The finding that "Semantic Interference" and "Optimal Strategy" scores are consistently ~10 percentage points below models' overall averages (Section 3.2, Figure 5) tells a specific story: current LLMs are surprisingly brittle when rules conflict with commonsense priors, and their ability to plan optimal sequences of actions under constraints is a particular weakness. This is actionable — model developers know where to focus improvement efforts, and users know which types of reasoning tasks are likely to cause failures.

The radar chart in Figure 5 visualizes this diagnostic use: different model families show distinct "fingerprints" of strengths and weaknesses even when their aggregate scores are similar. This level of diagnostic granularity is absent from benchmarks that report only overall accuracy.

Summary of the Positioning Logic

The paper's argument for why GENERAL365 is needed can be compressed into a syllogism:

Existing general reasoning benchmarks are either too easy (performance-saturated) or too homogeneous (template-dominated), or both. BBH is easy and homogeneous. BBEH is harder but more homogeneous. KOR-Bench is hard but narrow. ARC-AGI tests a different modality.
Domain-specific benchmarks (AIME, GPQA, SWE-bench) conflate reasoning with knowledge, obscuring the measurement of reasoning per se. High scores on these benchmarks could reflect reasoning, knowledge retrieval, or (most likely) some combination, and we cannot disentangle them from the aggregate score.
Therefore, we need a benchmark that is (a) difficult enough to resist saturation, (b) diverse enough in its reasoning patterns to resist template-matching shortcuts, and (c) knowledge-constrained enough to isolate reasoning from expertise. GENERAL365 is designed to satisfy all three criteria.

The paper positions itself not as an incremental improvement over BBH or BBEH but as a fundamentally different type of benchmark — one that prioritizes reasoning diversity and knowledge-orthogonality over instance count, and that is designed to remain challenging through multiple generations of model advancement. The 62.8% ceiling, far from being a limitation, is presented as evidence that GENERAL365 successfully targets a capability frontier that remains unsolved, providing a meaningful north star for general reasoning research.

3. Technical Approach

3.1 Reader Orientation

This paper builds a benchmark dataset and evaluation framework, not a machine learning model. The "system" is a carefully constructed collection of 365 human-crafted reasoning problems (expanded to 1,095 variants) paired with an eight-category diagnostic taxonomy, a hybrid rule-based and model-based grading pipeline, and a set of diversity validation procedures — all designed to measure how well LLMs can perform logical reasoning when specialized domain knowledge is deliberately excluded. The core problem it solves is measurement validity: existing benchmarks either saturate at high performance levels (ceiling effects that prevent tracking progress), conflate reasoning with domain expertise (making it impossible to tell whether a model is thinking or remembering), or rely on template-based construction that produces homogeneous instances susceptible to pattern-matching shortcuts rather than genuine generalization. The "shape" of the solution is a human-curated seed corpus that enforces reasoning diversity at the source, a K-12 knowledge constraint that decouples inference from expertise, difficulty filtering that resists saturation, and a multi-pronged validation methodology that quantitatively verifies that the benchmark actually achieves the diversity and difficulty it claims.

3.2 Big-Picture Architecture (Diagram in Words)

The GENERAL365 system has five major components, organized as a pipeline from problem conception through evaluation:

Challenge Categories Taxonomy — an eight-category conceptual framework (Complex Constraints, Branching & Enumeration, Spatial & Temporal Reasoning, Recursive & Backtracking, Semantic Interference, Implicit Information Reasoning, Optimal Strategy, Probability & Uncertainty) that defines the types of reasoning the benchmark will test. This taxonomy is the design blueprint: every seed problem must align with at least one category, and the category labels enable fine-grained diagnostic analysis after evaluation.
Seed Problem Construction + Difficulty Filtering — a human-driven, multi-stage process that crowdsources problems from diverse real-world domains, manually verifies each against the taxonomy, filters out problems easily solvable by top models or semantically similar to existing benchmarks, and augments under-represented categories. This component produces the 365 seed problems that form the benchmark's reasoning backbone.
Post-Processing + Expansion Pipeline — takes the seed problems and (a) standardizes output formats to enable reliable answer extraction, (b) uses LLMs to generate 10 candidate variants per seed problem (altering surface semantics while preserving core reasoning), and (c) applies human quality review to select the three best variants per seed, producing the final 1,095 evaluation instances. This component multiplies the dataset size without sacrificing reasoning diversity.
Hybrid Grading System — a dynamic two-branch evaluator: for numerical answers (238/365 seeds), a rule-based parser using math-verify extracts \boxed{} answers and checks equivalence against ground truth with manually specified precision thresholds; for selection and text answers (46/365 and 81/365 seeds respectively), GPT-4.1 acts as a model-based grader comparing model outputs to reference answers using structured prompts. This component achieves 99.6% grading accuracy as verified by manual review of 1,460 instances.
Diversity and Difficulty Validation Suite — a set of analytical procedures that empirically verify the benchmark's claimed properties: t-SNE visualization of embedding space coverage to check semantic dispersion, LLM-based pairwise logical similarity scoring (0–5 scale) to quantify reasoning redundancy, and cross-benchmark difficulty comparisons to confirm that GENERAL365 resists saturation. This component is not part of the evaluation infrastructure per se but provides the evidence that the benchmark measures what it claims to measure.

Information flows sequentially through components 1–4 for benchmark construction, then components 4–5 operate on model outputs during evaluation. The challenge taxonomy (component 1) guides construction (component 2), the seeds flow through post-processing and expansion (component 3), the complete dataset is administered to LLMs whose outputs feed into the grading system (component 4), and the resulting scores are analyzed through the category taxonomy for diagnostic breakdowns and through the validation suite (component 5) to confirm the benchmark's properties.

3.3 Roadmap for the Deep Dive

First, the challenge categories taxonomy (Section 2.1), because it is the conceptual architecture that defines what "general reasoning" means in this paper and constrains all subsequent design decisions. Without understanding the eight categories, the construction pipeline's filtering criteria and the diagnostic analysis in Table 1 are opaque.
Second, the seed data construction pipeline (Section 2.2), because it is where the benchmark's claimed diversity originates — the human curation, crowdsourcing, difficulty filtering, and multi-label annotation processes that produce the 365 seed problems. This section explains why the benchmark avoids template homogeneity.
Third, the post-processing and expansion pipeline (Section 2.2, continued), because it explains how 365 seeds become 1,095 evaluation instances without sacrificing reasoning diversity — the format standardization, LLM-based variant generation, and human quality review that scale the dataset while preserving its core properties.
Fourth, the hybrid grading methodology (Section 2.4), because evaluation accuracy is the foundation of benchmark credibility — without understanding how answers are parsed and scored, the reported accuracy numbers (and the 99.6% grading accuracy claim) lack grounding.
Fifth, the diversity validation methodology (Section 3.3.1), because it constitutes the paper's strongest empirical argument that GENERAL365 achieves its claimed differentiation from BBH and BBEH. The embedding visualization and pairwise similarity scoring are the quantitative evidence that the human-curated approach worked.
Sixth, the difficulty validation (Section 3.3.2), because it closes the loop on the paper's core claim: that GENERAL365 provides a meaningfully harder reasoning challenge than existing benchmarks, evidenced by both accuracy gaps and output length analysis.

3.4 Detailed, Sentence-Based Technical Breakdown

This is a benchmark construction and evaluation paper whose core idea is that general reasoning in LLMs should be measured in a setting where (a) required knowledge is constrained to K-12 level to decouple reasoning from expertise, (b) problem diversity is enforced through human curation rather than template expansion to prevent pattern-matching shortcuts, and (c) difficulty is deliberately maintained at a level where no current model saturates performance, providing headroom for tracking future progress.

Challenge Categories Taxonomy: The Conceptual Architecture

Before constructing any problems, the paper establishes an eight-category taxonomy that defines the dimensions of "general reasoning" it intends to measure. This taxonomy serves three functions: it provides the design specification that seed problems must satisfy (each problem is manually verified to align with at least one category), it enables fine-grained diagnostic analysis after evaluation (Table 1 breaks down accuracy by category), and it ensures structural diversity at the category level by covering qualitatively different cognitive demands.

The eight categories are defined as follows (Section 2.1):

Complex Constraints. Problems in this category involve "a web of interconnected logical predicates that must be satisfied simultaneously." The defining cognitive demand is global consistency maintenance: models must track state across multi-step reasoning chains, ensuring that inferences made at step 3 do not contradict constraints established at step 7. The representative example (Example 2 in Appendix A) illustrates this: six people with different clothing colors, shoe colors, and regions of origin seating around a table with 14 constraints that interact non-trivially (e.g., age relations, adjacency relations, color-region dependencies). Solving the problem requires propagating constraints across multiple dimensions simultaneously — a person's age constrains their position, which constrains their neighbors, which interact with clothing and origin constraints. The answer is a single integer (420, the product of possible ages for person C), requiring complete determination of the logical space before producing the final value.

Branching & Enumeration. These problems demand "systematic exploration of complex solution spaces" with "branching case analysis and exhaustive reasoning to ensure that no marginal scenarios or latent boundary conditions are overlooked." The cognitive demand is completeness: the model must consider all possible configurations (e.g., distribution of milk cartons across boxes in Example 3) and count how many satisfy a specific condition, without missing edge cases or double-counting. Example 3 requires determining, among all milk distribution schemes satisfying the given constraints (5 of 8 boxes contain milk with specific sum constraints on subsets of boxes), how many have Box A containing milk. The answer is 16 — a count that can only be obtained by systematic case analysis, not by pattern recognition or constraint propagation alone.

Spatial & Temporal Reasoning. These tasks evaluate "proficiency in spatial manipulation or temporal progression, such as geometric arrangements and causal sequencing." The cognitive demand is dynamic state modeling: the model must track how spatial positions or temporal states evolve under specific sequences of operations. Example 4 requires navigating a 5×5 grid maze following a sequence of moves and turns (e.g., "move forward one step, then move one step to the upper right, and turn left 45 degrees"), where the model must maintain orientation and position across multiple transformations. The final answer is "the starting point" — a result that requires correctly accumulating all spatial transformations.

Recursive & Backtracking. This category is "designed to evaluate non-linear problem-solving abilities" where "the core is self-correction." Models must "formulate hypotheses and engage in an iterative process of trial, verification, and backtracking." The cognitive demand is search with failure recovery: the model cannot solve the problem in a single forward pass but must explore possibilities, recognize dead ends, and backtrack to try alternative paths. Example 5 illustrates this with a pursuit problem: finding a wounded soldier who moves between adjacent tunnels each time period, where the model must determine the minimum number of checks (answer: 14) that guarantees capture regardless of the soldier's initial position and movement choices. This is a classic search problem that requires reasoning about worst-case adversarial behavior — the soldier actively tries to avoid capture, so the model must prove that no evasion strategy exists for any starting position given the checking sequence.

Semantic Interference. These problems "introduce cognitive traps by setting rules that defy common-sense intuition or by embedding misleading information." The cognitive demand is suppression of prior knowledge: the model must adhere strictly to a set of novel axioms that contradict its pretrained understanding of how the world works, ignoring the strong pull of commonsense defaults. Example 6 is particularly instructive: it describes a parallel universe where leap years and common years are reversed, a native tribe's calendar originally had the opposite number of days per month compared to the Gregorian calendar (except February), and then a policy changed all odd-numbered months to have one extra day and all even-numbered months to have one fewer day — applied retroactively to all time. The model must compute $10a + b$ where $a$ is the number of days in March 2000 and $b$ is the number of days in February 2105 under this deliberately contorted system. Success requires methodically applying the stated (counterfactual) rules rather than defaulting to the real-world calendar, which is what semantic interference tests.

Implicit Information Reasoning. Rather than providing all premises explicitly, these tasks "require clue discovery." Models must "bridge disjointed observations (e.g., symbolic patterns or latent mathematical regularities) to infer the underlying logical framework." The cognitive demand is pattern induction from incomplete data: the model cannot simply apply given constraints to reach a conclusion; it must first discover what the constraints are by identifying hidden relationships in the provided information. Example 7 presents a puzzle where country names are used in arithmetic expressions (Australia ˆ Brazil = 7776; India - Canada = 5; China + United States = 7) and the model must determine the hidden encoding rule and apply it to compute Argentina * Kazakhstan (answer: 72). The operation is not mathematical in the standard sense — the countries stand for something else (likely population rankings or alphabetical indices), and discovering that mapping is the core challenge.

Optimal Strategy. This category moves "beyond simple correctness" to "decision-making under constraints," where models must "evaluate multiple viable action sequences to determine the most efficient path, testing their capacity for utility-based optimization and strategic planning." The cognitive demand is constrained optimization: finding not just any solution but the best solution according to a specified objective. Example 8 asks for the shortest possible time for a fire truck to travel 10 km given speed limits (60 km/h limit, 80 km/h theoretical maximum), three traffic lights with specified cycles (58s red, 58s green, 4s yellow), a guarantee of encountering at least one full red light, and the ability to choose speed arbitrarily. The answer (7.5 minutes) requires modeling the tradeoff between speed and timing — going faster may cause arrival at a light during a red phase, creating waiting time that a slower approach might avoid.

Probability & Uncertainty. These tasks evaluate "the ability of LLMs to reason within stochastic environments characterized by incomplete information," requiring models to "estimate the likelihood of complex propositions or assess their logical possibility." The cognitive demand is probabilistic inference under ambiguity: the answer is not a deterministic logical consequence but a likelihood assessment that requires weighing multiple uncertain factors. Example 9 is a detective puzzle where several characters make statements about a murder, and the model must identify the most likely suspect based on inconsistencies in their testimony. The answer (D, the detective) emerges not from a single deductive chain but from noticing that the detective's claim of knowing the murderer contradicts information available at the time, making him suspicious — a form of probabilistic reasoning about deception and information states.

Multi-label annotation. Figure 3b reveals a crucial design choice: nearly 70% of seed problems are annotated with two or more challenge tags. This means GENERAL365 is not testing isolated reasoning primitives in separate problems but rather composite challenges where multiple cognitive demands interact within a single problem. A problem might require constraint propagation (Complex Constraints) while also demanding that the model avoid commonsense traps (Semantic Interference) and explore branching possibilities (Branching & Enumeration). This multi-label structure reflects real-world reasoning more accurately than single-dimension tasks but also makes the benchmark genuinely harder — models cannot succeed by mastering one reasoning skill at a time.

The category distribution (Figure 3a) shows Complex Constraints as the most prevalent (the bar extends furthest), with Probability & Uncertainty as the least represented (still containing over 20 seed problems, which the paper notes is "at least comparable to established benchmarks such as BBH and BBEH" in terms of per-category instance count). The paper explicitly justifies this minimum threshold: even the smallest category maintains sufficient intra-category diversity to support meaningful analysis.

Seed Data Construction Pipeline: Where Diversity Originates

The seed construction pipeline (Figure 2, first two phases) is the mechanism that distinguishes GENERAL365 from template-based benchmarks. The process has four sub-stages, each with explicit quality gates:

Crowdsourcing with domain breadth. Problems are crowdsourced "from diverse, real-world domains to ensure the inherent diversity of the seed data" (Section 2.2). The paper does not specify the number of contributors, their qualifications, or the exact domains covered, but the emphasis on "diverse, real-world domains" signals an intention to sample broadly from the space of possible reasoning scenarios — logic puzzles, scheduling problems, navigation tasks, detective scenarios, calendar manipulation, resource allocation — rather than constraining to a single problem type like math word problems or grid puzzles. This broad-domain sampling is the upstream cause of the embedding-space dispersion visible in Figure 6a.

Multi-criteria quality verification. Each crowdsourced problem must satisfy four explicit criteria before acceptance:

Taxonomy alignment: manually verified to align with at least one of the eight defined challenge categories.
Originality guarantee: must be "original and non-replicable via standard web searches" — this prevents contamination where models might have memorized solutions from training data. This is a critical validity constraint: if a problem appears verbatim in pretraining corpora, solving it tests memorization, not reasoning.
Complete triplet structure: each seed instance is structured as a triplet comprising (a) the problem description, (b) a detailed reasoning trace showing the step-by-step solution, and (c) the ground-truth answer. The reasoning trace is essential not for evaluation (models don't see it) but for quality verification — human reviewers use it to cross-check that the problem is well-posed and the answer is correct.
Final consistency check: a last round of manual inspection cross-verifies consistency between the problem statement, the gold answer, and the reasoning path to catch errors introduced during editing or formatting.

Difficulty filtering. After collecting seed data, the authors "conducted a rigorous filtering process to eliminate easy or similar instances" (Section 2.2). The filtering has two axes:

Solvability by top models: problems that top-tier LLMs can solve with high reliability are removed. The paper does not specify which models were used for this filtering, what threshold defined "easily solvable," or how many problems were filtered out, but the intent is clear: GENERAL365 should contain only problems that challenge even the best available models, ensuring the benchmark resists saturation as models improve.
Semantic similarity to existing benchmarks: problems that "demonstrated high semantic similarity to existing benchmarks" are removed to prevent overlap with BBH, BBEH, or other general reasoning benchmarks. The similarity detection method is not specified.

Category augmentation. For categories with insufficient sample counts after filtering, "the dataset was augmented by manually expanding categories with limited samples" (Section 2.2). This ensures the per-category minimum of ~20 seed problems, maintaining intra-category diversity for diagnostic analysis. The augmentation is manual, not LLM-based — maintaining the human-curation guarantee that underpins the benchmark's diversity claims.

The output of this pipeline is 365 seed problems, each annotated with one or more challenge category labels, each verified for correctness and originality, and collectively spanning the eight reasoning dimensions with sufficient per-category representation.

Post-Processing and Full-Scale Expansion: From Seeds to Evaluation Instances

The post-processing and expansion phase (Section 2.2, third and fourth stages) transforms the 365 seed problems into a standardized evaluation dataset of 1,095 instances while preserving reasoning diversity.

Post-processing for evaluation reliability. Before expansion, three refinements are applied to the seed problems:

Ambiguity resolution: "By analyzing the interactions between LLMs and seed data, we identified and rectified potential linguistic ambiguities, ensuring that the task premises are clear and unmistakable." This is an empirical refinement step — the authors ran seed problems through LLMs, observed where models misinterpreted instructions, and clarified the problematic language. This is important because ambiguity creates measurement noise: a model might fail a problem not due to reasoning limitations but because it interpreted a phrase differently than intended.
Output format standardization: Each problem description is appended with explicit output format instructions, such as "Select one or more appropriate options as your final answer based on the question above" for selection problems or \boxed{<answer>} directives for numerical problems (as shown in Examples 11–13 in Appendix B). These instructions guide LLMs to produce parseable outputs that the grading system can reliably extract. Without this standardization, the grading accuracy of 99.6% would be impossible — model-based graders would face ambiguous formatting, and rule-based parsers would miss answers embedded in non-standard locations.
Answer canonicalization: "Wherever feasible, we restructured answers into numerical or canonical formats, facilitating robust, rule-based extraction and grading." This means converting answers that might be expressed in multiple equivalent ways (e.g., "seven", "7", "VII") into a single canonical representation so that the grading system can perform exact matching rather than semantic equivalence checking. For problems where canonicalization is not feasible (e.g., text answers with multiple valid phrasings), the system falls back to model-based grading.

LLM-based variant generation with human-in-the-loop. The expansion strategy deliberately separates surface diversity from logical diversity:

For each seed problem, an LLM (unspecified which model) generates 10 candidate expansions. These candidates alter "surface semantics or constraints while preserving core reasoning skills" (Section 1). For example, a problem about distributing milk cartons might become a problem about assigning office supplies or allocating parking spaces — the constraint structure remains identical, but the surface framing changes. This tests whether models can transfer reasoning across surface forms.
Human experts then perform "a stringent quality review, discarding incorrect or illogical entries." The paper does not report the rejection rate, but the process implies that LLM-generated variants frequently contain errors (incorrect answers, internally inconsistent constraints, or changes that alter the reasoning structure rather than just the surface).
For seed problems where fewer than three variants survive the quality review, "manual intervention was employed to supplement the data." This maintains the 3-variants-per-seed ratio while ensuring all variants meet quality standards.

Final selection via difficulty estimation and diversity audit. After generating and reviewing variants, the authors "performed a comprehensive difficulty estimation and diversity audit to select the optimal candidates" (Section 2.2). The paper does not detail the difficulty estimation method or diversity audit procedure, but the outcome is 1,095 instances (365 seeds × 3 variants each) that collectively form the GENERAL365 evaluation dataset.

A crucial property of this expansion approach: because variants are derived from seeds by surface alteration while preserving core reasoning, the variant generation does NOT increase the benchmark's logical diversity — it increases instance count while maintaining diversity. The logical diversity comes entirely from the 365 human-curated seeds. The variants test whether models can generalize across surface forms of the same reasoning challenge, which is a different (and valuable) property, but it is the seeds that provide the reasoning diversity quantified in Figures 6 and 7.

Hybrid Grading Methodology: Achieving 99.6% Accuracy

The grading system (Section 2.4) is a dynamic two-branch evaluator that selects the grading method based on answer type, achieving a manually verified accuracy of 99.6% on 1,460 grading instances across four representative models.

Answer type classification. GENERAL365 problems are divided into three answer types:

Numerical answers: 238 of 365 seed problems (65.2%) require a numerical result. These are graded via rule-based extraction and equivalence checking.
Select answers: 46 of 365 seed problems (12.6%) require selecting one or more options from a provided set. These are graded via model-based comparison.
Text answers: 81 of 365 seed problems (22.2%) require free-text responses. These are graded via model-based comparison.

Rule-based branch for numerical answers. For numerical problems, the paper mandates use of the LaTeX command \boxed{<answer>} to encapsulate the final answer in model outputs (enforced via the output format instructions appended during post-processing). The extraction and verification pipeline operates as follows:

The math-verify library (from HuggingFace, repository huggingface/Math-Verify) parses model outputs to extract the content within \boxed{} delimiters.
For problems involving numerical approximations, the paper "manually annotated the required precision threshold, specifying the minimum number of decimal places for a response to be considered correct." This is a critical detail: without specified precision, a model that outputs 3.14159 when the answer is 3.14 would be penalized (or vice versa). The manual annotation ensures that precision requirements are determined by the problem's mathematical structure, not by arbitrary string matching.
The extracted answer is compared to the ground truth for mathematical equivalence (not just string equality), handling LaTeX formatting, fraction representations, and root expressions.

The choice of math-verify over custom parsing is a robustness decision: mathematical expression equivalence is non-trivial (e.g., $\frac{1}{2}$ and $0.5$ and $2^{-1}$ are all the same answer), and leveraging an existing, tested library reduces parsing errors that would contaminate benchmark scores.

Model-based branch for select and text answers. For problems where rule-based matching is infeasible (multiple-choice selections and free-text responses), the paper uses GPT-4.1 as the grading model with a structured prompt (Example 14 in Appendix B). The prompt presents the grading model with:

The original problem statement
The reference (ground-truth) answer
The student's (LLM's) answer

And asks the model to determine equivalence, outputting either "Correct" or "Incorrect" at the end. The prompt includes a precision instruction for decimal answers: "If the answer involves a decimal approximation, it must be accurate to at least four decimal places."

The choice of GPT-4.1 as the grader raises a question the paper does not address: could grader biases (e.g., GPT-4.1 systematically accepting its own model family's outputs more often than competitors') skew results? The 99.6% manual verification rate on 1,460 instances provides partial reassurance — if grader bias were present, it would manifest as systematic errors that human reviewers would detect — but the paper does not break down grading accuracy by model being evaluated, so differential bias across models cannot be ruled out from the reported data.

Manual verification of grading accuracy. The paper conducts a "rigorous manual quality review prior to the full-scale evaluation" on 1,460 grading instances. These instances span "four representative frontier models" (o4-mini, Gemini-2.5-Pro, DeepSeek-V3.1-Thinking, and LongCat-Flash-Thinking) evaluated on the seed problems. The choice of four diverse models (spanning OpenAI, Google, DeepSeek, and Meituan) for the verification set is deliberate — it ensures the grading system is tested across different output styles, formatting conventions, and error patterns.

The reported 99.6% grading accuracy means that out of 1,460 grading decisions, approximately 1,454 were correct and 6 were erroneous. A 0.4% error rate on a benchmark with 1,095 instances implies approximately 4-5 misgraded answers in a full evaluation run, producing a maximum measurement error of roughly 0.4 percentage points in reported accuracy — negligible relative to the performance differences between models (which span 37.6% to 62.8%).

Why hybrid rather than purely rule-based or purely model-based. The paper explicitly frames this as an efficiency-accuracy tradeoff. Rule-based grading "offers substantial computational efficiency and accuracy, provided the model responses adhere to parseable structures." Model-based grading "provides superior flexibility, enabling the nuanced assessment of complex, open-ended responses." By routing 65.2% of problems through the efficient rule-based branch and only 34.8% through the more expensive model-based branch, the system achieves both high accuracy and reasonable computational cost. A purely model-based system would cost roughly 3× more to run (grading all 1,095 instances with GPT-4.1 API calls) while a purely rule-based system would fail on the 34.8% of problems with non-numerical answers.

Diversity Validation Methodology: Proving the Benchmark Is Not Template-Homogeneous

Section 3.3.1 presents the paper's most technically sophisticated contribution: a two-pronged methodology for empirically validating that GENERAL365 achieves its claimed reasoning diversity, with direct comparisons to BBH and BBEH.

Qualitative validation via embedding space visualization. The procedure operates as follows:

All problem queries from GENERAL365 (1,460 instances, which includes the public subset and variants — the paper counts 1,460 in Figure 6a, suggesting these are the full expansion instances plus possibly additional analysis), BBH (6,511 instances), and BBEH (4,520 instances) are encoded into 1,536-dimensional vectors using the text-embedding-ada-002 model (OpenAI's embedding API).
The high-dimensional embeddings are projected to 2D using t-SNE with perplexity parameter set to 30. t-SNE (t-distributed Stochastic Neighbor Embedding) is a non-linear dimensionality reduction technique that preserves local structure — points that are close in the original 1,536-dimensional space remain close in the 2D visualization. The perplexity parameter of 30 controls the balance between local and global structure; a value of 30 is a standard default that prioritizes local neighborhood preservation.
Coordinates are normalized to the range $[-1, 1]$ for visual clarity.
The resulting 2D scatter plots are compared qualitatively (Figure 6).

The interpretation of Figure 6 is central to the paper's diversity argument. GENERAL365 (Figure 6a) shows a "highly dispersed and uniform distribution" with points spread broadly across the 2D manifold. In contrast, BBH (Figure 6b) and BBEH (Figure 6c) exhibit "significant 'local collapse' with isolated, high-density clusters" — tight groupings of points that indicate semantically near-identical problem instances. The paper characterizes this as evidence that BBH and BBEH instances are "often confined to a narrow set of linguistic templates or logical skeletons."

The mechanism linking construction methodology to embedding dispersion is direct: human-curated seeds from diverse real-world domains naturally occupy different regions of semantic space, while template-based expansion from a small set of task templates concentrates instances around the template's semantic center. The t-SNE visualization makes this mechanism visually evident.

Quantitative validation via LLM-based pairwise logical similarity scoring. The embedding visualization demonstrates semantic diversity but does not directly measure logical diversity — two problems could use different vocabulary to describe the same reasoning structure. The pairwise similarity scoring method addresses this gap directly:

For each problem $T_A$ in a benchmark, the system identifies its nearest neighbor $T_B$ in the embedding space (using the same text-embedding-ada-002 encodings). This ensures the comparison is between the most semantically similar problem pair — if any pair is logically redundant, this pair should be.
Gemini-3-Pro is employed as an expert evaluator to compare the triplet (Problem, Chain-of-Thought reasoning, Final Answer) for each pair. The choice of Gemini-3-Pro as the evaluator is justified by the paper noting it "demonstrates exceptional performance across diverse reasoning tasks, exhibiting significant potential in evaluating the semantic diversity and logical redundancy of problems."
The evaluator scores similarity on a 0–5 integer scale using a detailed rubric (Example 10 in Appendix B):
- 0 — Irrelevant: "Zero overlap in logic/knowledge."
- 1 — Weakly Related: "Macro-level connection (e.g., both 'math') but CoT paths are entirely different."
- 2 — Superficially Similar: "Same topic but fundamentally different reasoning depths."
- 3 — Logically Overlapping: "Significant overlap in reasoning logic or operators, though domains differ."
- 4 — Highly Structurally Similar: "Identical 'reasoning skeleton' and challenge sources."
- 5 — Essentially Identical: "Variants of each other; logic paths and operators are seamlessly transferable."
The prompt instructs the evaluator to focus on "the logic evolution trajectory within the CoT, rather than the surface-level wording of the problem" and provides four evaluation dimensions: Core Knowledge Graph (overlap in theorems, formulas, facts), Reasoning Paradigm (chaining, branching, network logic), Source of Challenge (hidden conditions, reverse thinking, logic depth), and Operational Primitives (symbolic transformation, case analysis, recursion).
The score distribution is aggregated and compared across benchmarks (Figure 7).

The results are stark:

GENERAL365: mean similarity $\mu = 2.16$ , with 68.2% of instances in the 1–2 score range. Only 3.8% receive the maximum score of 5. The distribution is left-skewed (mass concentrated at lower similarity scores).
BBH: mean similarity $\mu = 4.71$ , with 77.8% of instances receiving the maximum score of 5. The distribution is extreme right-skewed.
BBEH: mean similarity $\mu = 4.80$ , with 83.5% of instances receiving the maximum score of 5. Even more right-skewed than BBH.

The interpretation of mean similarity $\mu = 2.16$ is that even the most semantically similar problem pairs in GENERAL365 share only "superficial similarity" on average — same broad topic but fundamentally different reasoning structures. In contrast, BBH and BBEH's $\mu \approx 4.7$ – $4.8$ means that their most similar pairs are near-identical in reasoning skeleton. This is quantitative confirmation that template-based expansion produces logical homogeneity while human-curated seed diversity produces logical independence.

Why this validation methodology matters beyond GENERAL365. The pairwise similarity scoring approach is a methodological contribution in its own right. Prior benchmark papers typically assert diversity based on construction methodology (e.g., "we used 23 different tasks") without empirically verifying that those tasks actually require different reasoning. The paper's approach of embedding-space nearest-neighbor retrieval followed by LLM-based deep comparison of reasoning traces provides a replicable protocol for auditing any reasoning benchmark for logical redundancy. The specific rubric (0–5 scale with explicit definitions) and prompt template are provided in full (Example 10), enabling other researchers to apply the same methodology.

Caveat: evaluator model dependence. The similarity scores are produced by Gemini-3-Pro, which means the diversity validation is relative to that model's assessment of logical similarity. If Gemini-3-Pro has systematic biases in how it evaluates reasoning similarity (e.g., over-estimating similarity for problems within its training distribution), the absolute similarity scores could be miscalibrated. However, the relative comparison between benchmarks is more robust — any evaluator bias would affect BBH, BBEH, and GENERAL365 equally if the evaluation protocol is identical, so the large gaps in mean similarity (2.16 vs. 4.71 vs. 4.80) remain strong evidence of differential diversity even if the absolute scores have unknown calibration.

Difficulty Validation Methodology: Proving the Benchmark Resists Saturation

Section 3.3.2 validates that GENERAL365 achieves its claimed higher difficulty than existing general reasoning benchmarks using two complementary analyses.

Cross-benchmark accuracy comparison (Figure 8). The paper evaluates four models spanning capability levels (GPT-4.1, o3-mini, GPT-5-Thinking, Gemini-3-Pro) on GENERAL365, BBH, and BBEH, reporting accuracy for each model-benchmark pair. The results:

Model	GENERAL365	BBEH	BBH
GPT-4.1	26.7%	36.1%	89.4%
o3-mini	44.4%	50.4%	89.6%
GPT-5-Thinking	58.6%	71.1%	92.0%
Gemini-3-Pro	62.8%	77.0%	94.4%

Two patterns confirm GENERAL365's higher difficulty:

Absolute difficulty: For every model, GENERAL365 accuracy is substantially lower than BBEH and dramatically lower than BBH. The gap between BBH and GENERAL365 for the strongest model (Gemini-3-Pro) is 31.6 percentage points (94.4% − 62.8%). This is not a small margin — it's a fundamentally different performance regime.
Discriminative power: The performance spread between the weakest (GPT-4.1) and strongest (Gemini-3-Pro) models is 36.1 points on GENERAL365 (26.7% → 62.8%) versus 40.9 points on BBEH (36.1% → 77.0%) versus only 5.0 points on BBH (89.4% → 94.4%). BBH has essentially stopped discriminating between model capabilities — the gap between a mid-tier model and the state-of-the-art is smaller than the measurement noise. GENERAL365 maintains a large dynamic range, meaning it can track progress through multiple future model generations before approaching saturation.

Output length as a proxy for cognitive load (Figure 9). The paper analyzes the relationship between accuracy and average output token length across benchmarks for three models (o3-mini, GPT-5-Thinking, Gemini-3-Pro). The logic is that longer outputs indicate more elaborate reasoning chains — models spend more tokens on problems that require deeper multi-step inference.

The scatter plots in Figure 9 position each benchmark as a point in (output length, accuracy) space. For all three models, GENERAL365 requires the longest outputs (rightmost position on the x-axis) while achieving the lowest accuracy (lowest position on the y-axis):

o3-mini: BBH ~2k tokens at 89.6%, BBEH ~8k at 50.4%, GENERAL365 ~18k at 44.4%.
GPT-5-Thinking: BBH ~2k at 92.0%, BBEH ~5k at 71.1%, GENERAL365 ~13k at 58.6%.
Gemini-3-Pro: BBH ~2k at 94.4%, BBEH ~5k at 77.0%, GENERAL365 ~14k at 62.8%.

The paper interprets this as evidence that GENERAL365 problems "necessitate more elaborate, multi-step logical derivations and finer-grained thinking paths" and that "the increased token usage in GENERAL365 is not merely a sign of verbosity but a direct consequence of the taxing cognitive load." This interpretation is supported by the inverse relationship: models produce longer outputs on the benchmark where they perform worse, which is consistent with harder problems requiring more attempted reasoning (even if that reasoning is ultimately incorrect).

An alternative interpretation — that GENERAL365 simply prompts models to be more verbose due to its problem framing — is partially ruled out by the format standardization applied during post-processing, which uses similar output directives across problems. However, the paper does not control for problem length as a confound: if GENERAL365 problems are simply longer in word count than BBH/BBEH problems, models would naturally produce longer responses (more text to process and respond to) regardless of reasoning depth. The paper does not report average problem lengths across benchmarks, so this confound cannot be excluded.

Experimental Setup for Model Evaluation

Section 3.1 specifies the inference configuration used for all model evaluations. These settings matter because they affect both absolute performance and cross-model comparability.

Model selection. The paper evaluates 26 LLMs spanning "both proprietary and open-source models" from eight providers: OpenAI (GPT-5-Thinking, GPT-5.1-Thinking, o4-mini, o3-mini), Gemini (Gemini-3-Pro, Gemini-3-Flash, Gemini-2.5-Pro, Gemini-2.5-Flash), DeepSeek (V3.2-Speciale, V3.2-Thinking, V3.1-Thinking, V3.2-Chat), Qwen (Qwen3.5-397B-A17B-Thinking, Qwen3-Max-Thinking, Qwen3-235B-Thinking-2507, Qwen3-Max-Instruct), GLM (GLM-5-Thinking, GLM-4.7-Thinking, GLM-4.6-Thinking), Moonshot/Kimi (Kimi-K2.5-Thinking, Kimi-K2-Thinking), Anthropic (Claude-Sonnet-4.5), xAI (Grok-4.1-Fast-Reasoning), and Meituan/LongCat (LongCat-Flash-Thinking-2601, LongCat-Flash-Thinking-2509, LongCat-Flash-2512). The selection intentionally includes both reasoning-optimized architectures (typically denoted "Thinking" in model names) and standard chat models (e.g., Qwen3-Max-Instruct, DeepSeek-V3.2-Chat, LongCat-Flash-2512) to compare reasoning-specialized versus general-purpose architectures.

Temperature settings. The paper uses different temperatures for reasoning and non-reasoning models:

Reasoning models: temperature $T = 1.0$
Non-reasoning models: temperature $T = 0.7$

The rationale is only partially stated. The paper says the settings are chosen "to comprehensively explore the reasoning trajectories when facing the diverse tasks of GENERAL365" (Section 3.1). The higher temperature for reasoning models ( $T = 1.0$ , which is the maximum-entropy setting typically used for sampling from the full model distribution) encourages exploration of diverse reasoning paths — important for problems that may require non-obvious approaches. The lower temperature for non-reasoning models ( $T = 0.7$ ) is a more conservative setting that reduces stochastic variance but still permits some sampling diversity. The paper acknowledges (Appendix D) that the high-temperature setting "inevitably introduces stochasticity that can lead to noticeable variability in the final evaluation outcomes" but shows through four independent evaluation runs that the maximum deviation ( $\Delta = \text{max} - \text{min}$ ) remains under 3% for representative models (Table 3), confirming "that our evaluation metrics are reliable and highly reproducible despite the high-temperature setting."

This temperature asymmetry between reasoning and non-reasoning models is a potential confound in cross-architecture comparisons: reasoning models benefit from more exploration (higher temperature) while non-reasoning models are evaluated more conservatively. The paper does not discuss whether this choice advantages reasoning models in the head-to-head comparison.

Other sampling parameters. Top-p is fixed at $p = 1.0$ (no nucleus sampling truncation), top-k is left unconstrained, and maximum output length is set to "each model's capacity" (i.e., the model's architectural maximum, not a uniform cap). This means different models have different maximum output budgets, which could affect performance on problems requiring very long reasoning chains — a model with a 16k token output limit might truncate its reasoning on a problem where another model with a 32k limit would complete it.

Test-time compute settings. For models that support adjustable reasoning effort or test-time compute scaling budgets, the paper "defaulted to the highest available setting to elicit their peak reasoning performance" (Section 3.1). This means models like o4-mini, which have configurable reasoning effort levels (e.g., "low", "medium", "high"), are evaluated at their maximum setting. This is a deliberate choice to measure each model's ceiling reasoning capability rather than its efficiency at lower compute budgets — consistent with the benchmark's goal of testing the frontier of what models can achieve.

Why these settings matter for benchmark interpretation. The evaluation configuration represents a deliberate tradeoff between internal validity (fair comparison across models) and ecological validity (measuring models as they would be used in practice). Using model-specific maximum output lengths and the highest test-time compute settings measures what each model can do when given every advantage, but it makes cross-model comparisons less "fair" — a model with a larger output budget or more sophisticated test-time scaling has advantages beyond its raw reasoning capability. The paper's analysis of reasoning efficiency (Figure 4), which plots accuracy against average output tokens, partially addresses this by showing that Gemini-3-Pro achieves top performance with fewer tokens than competitors, but the efficiency analysis itself is post-hoc rather than controlled.

Summary of Design Choices and Their Justifications

The paper makes several interconnected design choices that collectively define GENERAL365's approach to measuring general reasoning. Here they are consolidated with explicit justifications:

K-12 knowledge constraint (Section 1): Chosen to decouple reasoning measurement from knowledge retrieval, directly addressing the confound identified in domain-specific benchmarks where high performance may reflect memorization of expert patterns rather than logical inference capability. The constraint is enforced by design — all seed problems are verified to require only K-12 level background knowledge.
Human-curated seed problems over LLM-generated or template-based construction (Section 2.2): Chosen to achieve genuine reasoning diversity, as validated by the embedding dispersion (Figure 6a) and low pairwise similarity scores ( $\mu = 2.16$ , Figure 7). Template-based approaches (used by BBH, BBEH) produce high instance counts but concentrate in narrow regions of both semantic and logical space, risking performance inflation through pattern-matching shortcuts.
LLM-based variant generation for expansion, not diversity (Section 2.2): Variants are generated by altering surface semantics while preserving core reasoning, so the 1,095 instances maintain the logical diversity of the 365 seeds. This is a deliberate separation of concerns: seeds provide diversity, variants provide instance count for statistical reliability. Using LLMs for variant generation rather than manual authoring is a cost-efficiency tradeoff, with human quality review serving as the guardrail against introduced errors.
Explicit multi-label challenge category annotation (Section 2.1, Figure 3b): Nearly 70% of problems carry multiple category labels, reflecting the paper's view that real-world reasoning involves composite challenges rather than isolated primitives. This enables diagnostic analysis (Table 1) while also making the benchmark harder — models cannot succeed by mastering one reasoning dimension at a time.
Difficulty filtering of seed problems (Section 2.2): Deliberately removes problems easily solvable by top models to prevent saturation. This is a forward-looking design choice: GENERAL365 is meant to remain challenging through multiple model generations, and the 62.8% ceiling for the current state-of-the-art leaves substantial headroom.
Hybrid grading with type-based routing (Section 2.4): Balances computational efficiency (rule-based parsing for 65.2% of problems) with flexibility (model-based grading for 34.8% of problems). The 99.6% verified accuracy provides credibility for the reported scores, while the separation of answer types enables format standardization during post-processing.
High-temperature sampling for evaluation (Section 3.1, Appendix D): $T = 1.0$ for reasoning models and $T = 0.7$ for non-reasoning models, chosen to encourage exploration of diverse reasoning paths on the benchmark's challenging problems. The stability analysis (Table 3) confirms that the resulting stochastic variance is acceptably small ( $\Delta < 3\%$ across four runs).
Two-pronged diversity validation (Section 3.3.1): The combination of qualitative embedding visualization and quantitative pairwise similarity scoring provides converging evidence for GENERAL365's claimed diversity advantage over BBH and BBEH. The quantitative method in particular is a methodological contribution that other benchmark papers could adopt — it provides an empirical, replicable way to audit reasoning diversity rather than relying on construction-methodology assertions.
Public subset release (Appendix C): 180 seed problems with their variants (720 instances total) are released publicly to "facilitate community research and ensure reproducibility." Table 2 confirms that public subset scores closely track full benchmark scores, validating that the subset is representative. The remaining problems are presumably held out to prevent contamination of future model training data, a standard practice in benchmark stewardship that preserves the benchmark's ability to measure genuine reasoning rather than memorization.

4. Key Insights and Innovations

Innovation 1: Decoupling Reasoning from Knowledge as a Measurement Principle, Not Just a Design Constraint

The paper's most conceptually distinctive move is elevating "knowledge-orthogonal reasoning" from a benchmark design preference to a measurement validity principle. Prior work in the general reasoning space—BBH, BBEH, KOR-Bench—acknowledged the desirability of testing reasoning rather than knowledge, but treated it as a matter of task selection: choose puzzles and logic problems rather than physics exams, and you've solved the confound. GENERAL365 argues that this is insufficient. The confound between reasoning and knowledge retrieval is not eliminated merely by picking non-expert topics; it must be actively enforced through a K-12 knowledge constraint that is verified during construction, not assumed from task type.

Why this is a conceptual innovation rather than an incremental refinement: the paper reframes the evaluation problem from "find tasks that don't require expertise" to "prove that your benchmark isn't measuring expertise." The distinction matters because many "general reasoning" tasks that appear knowledge-light can still be solved through memorized pattern recognition if the underlying logical structures appear in pretraining corpora. A logic puzzle about knights and knaves may test reasoning when first encountered but tests memory once it's been seen in a hundred training-set variations. The K-12 constraint is a necessary but not sufficient condition for knowledge-orthogonal measurement; GENERAL365 pairs it with the originality requirement (problems "non-replicable via standard web searches," Section 2.2) and the diversity validation (Section 3.3.1) to create a multi-layered defense against the knowledge confound.

The empirical payoff of this principle is visible in Figure 8: the same models that score 89–94% on BBH (where knowledge-light tasks have become familiar through benchmark exposure and training data contamination) drop to 27–63% on GENERAL365. The 30+ percentage point gap cannot be explained by BBH being "easier" in any absolute sense—BBH was explicitly designed to be hard. It is explained by BBH's reasoning patterns having been absorbed into model training distributions over time, transforming reasoning problems into recognition problems. GENERAL365's K-12 constraint, combined with its human-curated originality, is specifically designed to resist this absorption, making it measure reasoning at evaluation time rather than memorization at training time.

The paper doesn't frame it this way, but what GENERAL365 is really testing is out-of-distribution logical reasoning: can a model apply general inference principles to problems whose specific logical structures are unlikely to have appeared in its training data? This connects to the broader OOD generalization literature in a way that existing reasoning benchmarks, with their template-homogeneous construction, do not.

Innovation 2: Quantitative Reasoning Diversity as a Benchmark Quality Metric

Before GENERAL365, benchmark diversity was typically asserted through construction methodology: "we have 23 tasks from different domains," "we used 8 categories of problems." The paper introduces something new: a quantitative, replicable methodology for measuring logical diversity and uses it to demonstrate that its benchmark achieves an order-of-magnitude improvement over predecessors.

The pairwise logical similarity scoring protocol (Section 3.3.1, Figure 7) is the mechanism, but the innovation is the concept that logical diversity is measurable and should be measured. The paper operationalizes this through LLM-based comparison of (Problem, CoT, Final Answer) triplets on a 0–5 scale, but the specific rubric matters less than the principle: a benchmark's instances can be audited for reasoning redundancy, and the results of that audit should be reported alongside accuracy scores.

The evidence is dramatic. BBH and BBEH, which claim diversity through multi-task design, show mean logical similarity scores of μ = 4.71 and μ = 4.80 respectively on a 0–5 scale, with 77.8% and 83.5% of instances receiving the maximum similarity score of 5 (Figure 7). This means the most semantically similar problem pairs in these benchmarks share essentially identical reasoning skeletons—they are, in the evaluator's judgment, "variants of each other" with "logic paths and operators seamlessly transferable." GENERAL365's μ = 2.16, with 68.2% of instances in the 1–2 range, means even its closest neighbors are only "superficially similar" in reasoning structure. This isn't a marginal improvement; it's a qualitative difference in what the benchmark measures.

Why this changes the conversation around benchmark construction. Template-based expansion—take one reasoning template, perturb surface features, generate 100 instances—has been the dominant paradigm because it produces large datasets cheaply. The paper's diversity analysis demonstrates that this paradigm achieves instance count without reasoning variety. A benchmark of 6,511 instances (BBH) that tests only a handful of distinct logical patterns provides less genuine reasoning coverage than a benchmark of 365 seeds that each represent independent reasoning structures. This implies that reasoning diversity per construction dollar, not raw instance count, is the metric that benchmark designers should optimize—and the paper provides the tooling to measure it.

The embedding visualization (Figure 6) provides converging qualitative evidence: GENERAL365's t-SNE projection shows broad, uniform dispersion while BBH and BBEH show "local collapse" into isolated clusters. The paper is careful not to claim the embedding analysis alone proves logical diversity—semantic dispersion and logical dispersion are related but distinct—but the combination of visual evidence from embeddings and quantitative evidence from similarity scoring makes a stronger case than either would alone.

One limitation worth noting: the similarity scoring uses Gemini-3-Pro as the evaluator, meaning "logical diversity" is operationalized as "dissimilarity as judged by a specific state-of-the-art reasoning model." If Gemini-3-Pro has systematic tendencies in how it perceives reasoning similarity—for instance, rating problems from its training distribution as more similar than they "objectively" are—the absolute scores could be miscalibrated. However, the relative comparison between benchmarks is robust to this concern: any evaluator bias affects BBH, BBEH, and GENERAL365 equally under identical evaluation protocols, so the 2.16 vs. 4.71 vs. 4.80 gap remains strong evidence of differential diversity.

Innovation 3: The Diagnostic Taxonomy as a Capability Boundary Map

Most benchmarks report a single aggregate score, possibly with per-task breakdowns if the benchmark comprises distinct tasks. GENERAL365's eight-category taxonomy (Section 2.1) does something different: it provides a capability decomposition that reveals not just that models struggle with general reasoning, but which specific reasoning capabilities fail and how failure patterns differ across model architectures.

The taxonomy was introduced in Section 2.1 and its construction was detailed in Section 3, but its conceptual significance as an innovation is that it transforms the benchmark from an evaluative instrument into a diagnostic instrument. Table 1 and Figure 5 are not just additional results; they demonstrate a use case that prior general reasoning benchmarks could not support.

The key finding—that "Semantic Interference" and "Optimal Strategy" are primary bottlenecks, with per-category scores consistently ~10 percentage points below models' overall averages—is specific and actionable. "Semantic Interference" tests whether models can suppress pretrained commonsense biases and follow novel axioms (Example 6: a deliberately contorted calendar system with reversed leap years and month-length transformations). The fact that even top reasoning models score only ~55% on this category while scoring ~65% on "Complex Constraints" tells a precise story: current LLMs are surprisingly brittle when rules actively conflict with their training distribution's priors. They can handle many simultaneous constraints (Complex Constraints) better than they can handle one constraint that contradicts everything they know.

"Optimal Strategy" tests utility-based optimization—finding the best sequence of actions under constraints, not just any valid sequence (Example 8: the fire truck speed-vs-timing problem). Scores in the 40–50% range for top models indicate that current architectures struggle with evaluating multiple viable paths to select the optimal one. This is distinct from finding a solution (what Complex Constraints or Branching & Enumeration test) and represents a qualitatively different cognitive demand.

The diagnostic value extends to cross-model comparison. The radar chart (Figure 5) shows that different model families have distinct "fingerprints": one model might be strong on Implicit Information but weak on Spatial & Temporal Reasoning, while another shows the reverse pattern, even when their aggregate scores are similar. This enables capability-specific model selection: a practitioner choosing between models for a deployment where most reasoning tasks involve strategic planning (Optimal Strategy) versus constraint satisfaction (Complex Constraints) can consult per-category scores rather than relying on a single number that averages across irrelevant dimensions.

The multi-label annotation design (Figure 3b: ~70% of problems carry 2+ category labels) adds nuance: models aren't just tested on isolated capabilities but on interactions between them. A problem might require constraint propagation while also demanding resistance to semantic interference, and the model's performance on that composite challenge may not be predictable from its scores on each category in isolation. This reflects the reality that real-world reasoning involves overlapping cognitive demands, but it complicates clean attribution of failures to specific capability gaps.

The paper's contribution here is taxonomy-as-infrastructure: by establishing a set of named, defined, and empirically validated reasoning dimensions, GENERAL365 provides a vocabulary for discussing LLM reasoning capabilities that is more granular than "reasoning ability" but more general than per-benchmark scores. Future work can build on, critique, or refine this taxonomy, but having it articulated and operationalized in a benchmark enables conversations about specific capability boundaries that were previously impossible.

Innovation 4: Reasoning Efficiency as a First-Class Evaluation Dimension

Accuracy-per-dollar (or per-FLOP, or per-token) is a standard concept in ML evaluation, but the paper's analysis of reasoning density—accuracy achieved per output token generated—represents a shift in how the field thinks about reasoning model quality. Figure 4 shows why this matters: Gemini-3-Pro achieves state-of-the-art accuracy (62.8%) with an average output of ~14k tokens, while competitors like GLM-5-Thinking and DeepSeek-V3.2-Speciale require ~25–30k tokens to achieve lower accuracy (59.9% and 57.5% respectively). This isn't merely "Gemini-3-Pro is more efficient"—it's evidence that output length and reasoning quality are partially decoupled, contrary to the implicit assumption that "more thinking tokens = better reasoning."

The paper's framing of this as "reasoning density" or "reasoning efficiency" is a conceptual contribution because it introduces a Pareto frontier for evaluating reasoning models: accuracy vs. output length. A model that achieves high accuracy with many tokens (the "brute force" strategy) occupies a different position on the frontier than one that achieves similar or better accuracy with fewer tokens (the "efficient reasoning" strategy). Prior benchmark papers typically report accuracy alone, implicitly treating all correct answers as equivalent regardless of how much computation produced them. GENERAL365's efficiency analysis (Figures 4 and 9) demonstrates that this collapses an important dimension of model capability.

Figure 9 provides converging evidence through cross-benchmark comparison: models produce much longer outputs on GENERAL365 than on BBH despite lower accuracy, suggesting that GENERAL365 problems genuinely require more elaborate reasoning chains. The paper interprets the inverse relationship between accuracy and output length across benchmarks as evidence that "the increased token usage in GENERAL365 is not merely a sign of verbosity but a direct consequence of the taxing cognitive load." This is a stronger claim than it might appear: it argues that when problems are genuinely harder, models try harder (produce more reasoning tokens) but still fail more often, suggesting the reasoning is necessary but insufficient.

The practical implication is significant for model deployment. If two models achieve similar accuracy but one requires half the output tokens, the more efficient model is cheaper to serve, has lower latency, and consumes less energy per correct answer—all of which matter in production. By making this dimension visible and measurable, GENERAL365 enables optimization that goes beyond "pick the most accurate model" to "pick the model on the accuracy-efficiency Pareto frontier that matches your deployment constraints."

A subtle connection to the test-time compute scaling literature: the paper's efficiency analysis implicitly engages with the hypothesis that scaling inference compute (through longer outputs, more reasoning tokens) is the primary path to improved reasoning. Gemini-3-Pro's position on Figure 4—top accuracy with modest output—provides evidence that architectural improvements can substitute for inference-time compute, achieving better results with fewer tokens. This is a single data point, not a systematic analysis, but it suggests that the relationship between output length and reasoning quality may be more elastic than the "just think longer" paradigm assumes.

5. Experimental Analysis

Evaluation Methodology

Dataset. GENERAL365 comprises 365 human-curated seed problems and 1,095 evaluation instances (365 seeds × ~3 variants each) drawn from diverse real-world domains, with required background knowledge restricted to the K-12 level. The benchmark is split into a full set (365 seeds, 1,095 variants) used for the primary leaderboard and a publicly released subset of 180 seeds (720 variants) for community research and reproducibility (Appendix C). The dataset is not described as having train/validation/test splits in the traditional ML sense — all instances are evaluation instances, and the public subset is a random sample of the full set.
Base model(s). The paper evaluates 26 leading LLMs from eight providers: OpenAI (GPT-5-Thinking, GPT-5.1-Thinking, o4-mini, o3-mini), Google DeepMind (Gemini-3-Pro, Gemini-3-Flash, Gemini-2.5-Pro, Gemini-2.5-Flash), Anthropic (Claude-Sonnet-4.5), xAI (Grok-4.1-Fast-Reasoning), DeepSeek (DeepSeek-V3.2-Speciale, DeepSeek-V3.2-Thinking, DeepSeek-V3.1-Thinking, DeepSeek-V3.2-Chat), Qwen (Qwen3.5-397B-A17B-Thinking, Qwen3-Max-Thinking, Qwen3-235B-A22B-Thinking-2507, Qwen3-Max-Instruct), GLM (GLM-5-Thinking, GLM-4.7-Thinking, GLM-4.6-Thinking), Kimi (Kimi-K2.5-Thinking, Kimi-K2-Thinking), and LongCat (LongCat-Flash-Thinking-2601, LongCat-Flash-Thinking-2509, LongCat-Flash-2512). The selection intentionally spans proprietary and open-source models, reasoning-specialized architectures (denoted "Thinking") and standard chat models, covering a capability range from 37.6% to 62.8% accuracy on the full benchmark (Figure 1). No single model family is claimed as "the" base model — the evaluation is a cross-sectional survey of the current LLM landscape.
Metrics. The primary metric is accuracy (%) — the fraction of evaluation instances for which the model's extracted or judged final answer matches the ground-truth answer. For numerical problems (238/365 seeds), this is determined by extracting the \boxed{} answer via the math-verify library and checking mathematical equivalence against the ground truth, with manually annotated precision thresholds for approximations (Section 2.4). For select and text answer problems (46/365 and 81/365 seeds respectively), accuracy is determined by GPT-4.1 model-based grading comparing model outputs to reference answers. The overall accuracy is the micro-average across all instances. Secondary metrics include per-category accuracy (Table 1, Figure 5) and average output tokens as a proxy for reasoning effort (Figures 4, 9). The paper also introduces reasoning efficiency as an implicit metric through the accuracy-vs-output-tokens scatter plots.
Baselines. There are no baseline methods in this paper in the traditional sense — GENERAL365 is a benchmark, not a method being compared against alternatives. However, the paper implicitly treats BBH (Suzgun et al., 2023) and BBEH (Kazemi et al., 2025) as benchmark baselines against which GENERAL365's difficulty and diversity are compared. For difficulty comparison (Figure 8), the paper evaluates four models (GPT-4.1, o3-mini, GPT-5-Thinking, Gemini-3-Pro) on all three benchmarks to demonstrate that GENERAL365 yields substantially lower accuracy. For diversity comparison (Figures 6, 7), GENERAL365's embedding-space dispersion and pairwise logical similarity distributions are contrasted with those of BBH and BBEH.
Generation budget / compute accounting. The paper does not measure or control for total FLOPs or inference cost in the primary evaluation — models are benchmarked under their standard inference configurations. However, three aspects of compute allocation are specified: (1) For models that support adjustable reasoning effort or test-time compute scaling budgets, the evaluation uses "the highest available setting to elicit their peak reasoning performance" (Section 3.1), meaning models like o4-mini with "high" reasoning effort settings are run at maximum compute. (2) Maximum output length is set to "each model's capacity" (their architectural limit, not a uniform cap). (3) The post-hoc efficiency analysis (Figure 4) uses average output tokens as a proxy for inference compute, plotting accuracy against token expenditure to identify models on the accuracy-efficiency Pareto frontier. This is not a controlled compute-matched comparison — it is a descriptive analysis of the empirical relationship between output length and accuracy across models.
Cross-validation / statistical protocol. The paper does not use cross-validation for model evaluation (there is no training involved). However, two statistical robustness procedures are employed: (1) Sampling variance analysis (Appendix D): Four representative models (DeepSeek-V3.2-Thinking, Kimi-K2-Thinking, GLM-4.7-Thinking, LongCat-Flash-Thinking-2601, LongCat-Flash-2512) undergo four independent evaluation runs each at the specified temperatures (T = 1.0 for reasoning models, T = 0.7 for non-reasoning models). The maximum deviation Δ = max − min between runs is reported in Table 3, with all Δ values under 3 percentage points (ranging from 0.9% for LongCat-Flash-Thinking-2601 to 2.6% for LongCat-Flash-2512), confirming acceptable reproducibility despite stochastic sampling. (2) Public subset validation (Appendix C, Table 2): The 26-model accuracy rankings on the 720-instance public subset are compared to the full 1,095-instance benchmark scores to verify that the public release is representative. Scores on the public subset closely track full-benchmark scores (e.g., Gemini-3-Pro: 62.8% full vs. 61.3% public; GPT-5-Thinking: 58.6% vs. 56.0%), with no model deviating by more than ~2.6 points. (3) Manual grading accuracy verification (Section 2.4): 1,460 grading instances spanning four frontier models (o4-mini, Gemini-2.5-Pro, DeepSeek-V3.1-Thinking, LongCat-Flash-Thinking) are manually reviewed, yielding a verified grading accuracy of 99.6%.

Main Quantitative Results

Aggregate Leaderboard Performance (Figure 1, Table 1)

The headline finding is that no evaluated model exceeds 62.8% accuracy on GENERAL365, and the majority of the 26 tested models fail to reach even a 60% passing threshold. Figure 1 presents the full leaderboard, with models categorized by proprietary vs. open-source status and reasoning vs. non-reasoning architecture:

Gemini-3-Pro (Google DeepMind, reasoning model, proprietary) achieves state-of-the-art performance at 62.8% accuracy.
Gemini-3-Flash follows at 60.8%.
GLM-5-Thinking (open-source) leads the open-source category at 59.9%, placing it within 2.9 percentage points of the top proprietary model — a gap the paper interprets as evidence that "recent advancements in the open-source community are rapidly approaching the frontiers of commercial reasoning capabilities" (Section 3.2).
GPT-5-Thinking (OpenAI) and GPT-5.1-Thinking achieve 58.6% and 58.2% respectively.
The bottom of the leaderboard is occupied by DeepSeek-V3.2-Chat (non-reasoning, open-source) at 37.6% and Gemini-2.5-Flash at 39.6%.

The performance spread between the strongest and weakest model is 25.2 percentage points (62.8% – 37.6%), indicating that GENERAL365 provides substantial dynamic range for discriminating model capabilities. Notably, some non-reasoning models outperform dedicated reasoning models: Qwen3-Max-Instruct (a chat model) achieves 48.5%, surpassing Gemini-2.5-Flash (39.6%) and o3-mini with high reasoning effort (44.4%).

The paper flags a specific observation about ranking inversions relative to domain benchmarks: while models like DeepSeek-V3.2-Speciale and GLM-5-Thinking compete closely on mathematics benchmarks, their GENERAL365 rankings (57.5% vs. 59.9%) show a meaningful gap, suggesting that strong mathematical reasoning does not uniformly translate to general logical reasoning.

Per-Category Diagnostic Analysis (Table 1, Figure 5)

Table 1 provides the fine-grained breakdown of model accuracy across the eight challenge categories for all 26 models. The primary findings:

Semantic Interference and Optimal Strategy are the hardest categories. Across all top-performing models, scores in these two categories are consistently 5–15 percentage points below the models' overall averages. For example:

Gemini-3-Pro: 62.8% overall vs. 55.7% on Semantic Interference and 50.8% on Optimal Strategy.
GPT-5-Thinking: 58.6% overall vs. 57.2% on Semantic Interference and 47.5% on Optimal Strategy.
GLM-5-Thinking: 59.9% overall vs. 49.2% on Semantic Interference and 51.7% on Optimal Strategy.

The Optimal Strategy gap is particularly severe: the highest score any model achieves is 51.7% (GLM-5-Thinking), and top models cluster in the 47–51% range for this category, compared to 60–69% ranges for categories like Complex Constraints and Recursive & Backtracking. This indicates that strategic planning under constraints is a specific and severe weakness of current architectures, distinct from their ability to handle constraint satisfaction (Complex Constraints: 60–65% for top models) or systematic search (Branching & Enumeration: 62–69%).

For Semantic Interference, the paper interprets the findings as evidence that current LLMs are "notably susceptible to distracting information or cognitive traps, which often divert the model from the core logical constraints" (Section 3.2). The performance pattern is consistent across architectures: even Gemini-3-Pro, which excels at Implicit Information Reasoning (68.8%) and Recursive & Backtracking (66.8%), drops to 55.7% when rules actively conflict with commonsense priors.

Reasoning models substantially outperform chat models on Implicit Information Reasoning. The widest gap between reasoning and chat architectures appears in this category. Gemini-3-Pro achieves 68.8% while DeepSeek-V3.2-Chat manages only 29.6% — a 39.2 percentage point gap. The paper interprets this as evidence that "reasoning models excel at decoding subtle, non-explicit signals into structured logical frameworks that chat models often fail to capture" (Section 3.2). Other categories showing large architecture gaps include Spatial & Temporal Reasoning (57.6% for Gemini-3-Pro vs. 26.1% for DeepSeek-V3.2-Chat) and Recursive & Backtracking (66.8% vs. 50.3%).

Category performance is relatively balanced overall. The paper notes that despite the Semantic Interference and Optimal Strategy bottlenecks, the difficulty distribution across categories is "relatively balanced," with no single dimension appearing "exceptionally trivial or insurmountable" (Section 3.2). The radar chart in Figure 5 visualizes this, showing that different model families exhibit distinct "fingerprints" of relative strengths and weaknesses across the eight categories even when their aggregate scores are similar. For instance, GPT-5-Thinking is relatively strong on Semantic Interference (57.2%) compared to its overall average (58.6%), while GLM-5-Thinking is relatively weak on that same category (49.2% vs. 59.9% overall), revealing architectural differences in susceptibility to cognitive traps.

Models that lead in specific categories tend to lead overall. The paper observes a positive manifold: "models that achieve SOTA performance in specific challenge categories also tend to rank high in overall accuracy, suggesting that general reasoning proficiency is a composite of these foundational abilities" (Section 3.2). Gemini-3-Pro achieves the highest or near-highest score in seven of eight categories (the exception is Branching & Enumeration, where GLM-5-Thinking leads at 69.1% vs. Gemini-3-Pro's 64.3%). This pattern is consistent with a general factor (g-factor) interpretation of reasoning capability, though the paper does not conduct factor analysis to formally test this.

Reasoning Efficiency Analysis (Figure 4, Figure 9)

Accuracy vs. output tokens across models (Figure 4). The paper plots all evaluated models in a scatter plot with average output tokens on the x-axis and accuracy on the y-axis, revealing several patterns:

Overall positive correlation: There is a visible positive trend — higher-accuracy models generally produce more output tokens. Top-tier frontier models "generally utilize a high volume of tokens to navigate the complex reasoning paths required by GENERAL365" (Section 3.2). Several leading reasoning models (e.g., DeepSeek-V3.2-Speciale, GLM-5-Thinking) require approximately 25,000 to 30,000 tokens on average to achieve their scores.
Gemini-3-Pro is a notable outlier: It achieves the highest accuracy (62.8%) while maintaining a "remarkably concise" average output of approximately 14,000 tokens — roughly half the token expenditure of competitors with similar accuracy. The paper frames this as evidence of "superior reasoning efficiency" and "superior reasoning density, demonstrating an ability to resolve complex tasks with high reasoning efficiency" (Section 3.2).
Within-model-family trajectories: The paper highlights improvement trajectories within model series. Gemini-3-Pro shows a "significant performance leap over Gemini-2.5-Pro while maintaining a comparable and efficient token expenditure" (Section 3.2), suggesting that architectural improvements can increase accuracy without proportional increases in output length. The specific numbers: Gemini-2.5-Pro achieves 48.7% at an unspecified token count (visible as a point on Figure 4 but not numerically reported in the text), while Gemini-3-Pro achieves 62.8% at ~14k tokens.
Non-reasoning models cluster at low token counts, low accuracy: Chat models like DeepSeek-V3.2-Chat (37.6%) and LongCat-Flash-2512 (41.7%) occupy the lower-left region of the plot, producing relatively short outputs with low accuracy, contrasting with reasoning models that invest more tokens in deliberation.

Cross-benchmark output length comparison (Figure 9). The paper compares accuracy and output length across BBH, BBEH, and GENERAL365 for three models (o3-mini, GPT-5-Thinking, Gemini-3-Pro). The scatter plots position each benchmark as a point in (output length, accuracy) space:

For o3-mini: BBH achieves 89.6% at ~2,000 tokens; BBEH achieves 50.4% at ~8,000 tokens; GENERAL365 achieves 44.4% at ~18,000 tokens. The model produces its longest outputs on the benchmark where it performs worst.
For GPT-5-Thinking: BBH: 92.0% at ~2,000 tokens; BBEH: 71.1% at ~5,000 tokens; GENERAL365: 58.6% at ~13,000 tokens.
For Gemini-3-Pro: BBH: 94.4% at ~2,000 tokens; BBEH: 77.0% at ~5,000 tokens; GENERAL365: 62.8% at ~14,000 tokens.

The consistent pattern — lowest accuracy on GENERAL365 despite highest output length — is interpreted as evidence that "GENERAL365 tasks necessitate more elaborate, multi-step logical derivations and finer-grained thinking paths" and that "the increased token usage in GENERAL365 is not merely a sign of verbosity but a direct consequence of the taxing cognitive load" (Section 3.2). The paper argues that the inverse relationship between accuracy and output length across benchmarks (rather than within a benchmark) distinguishes genuine cognitive difficulty from mere verbosity: if models were simply producing more words on GENERAL365, they might achieve higher accuracy through more thorough reasoning, but they achieve lower accuracy despite the extra tokens, suggesting the reasoning is necessary but largely unsuccessful.

Cross-Benchmark Difficulty Comparison (Figure 8)

Figure 8 presents a direct difficulty comparison between GENERAL365, BBH, and BBEH using four models spanning capability levels:

Model	GENERAL365	BBEH	BBH
GPT-4.1	26.7%	36.1%	89.4%
o3-mini (High)	44.4%	50.4%	89.6%
GPT-5-Thinking	58.6%	71.1%	92.0%
Gemini-3-Pro	62.8%	77.0%	94.4%

The key findings:

BBH is essentially saturated: For GPT-4.1 through Gemini-3-Pro, scores range only from 89.4% to 94.4% — a 5.0 percentage point spread. BBH can barely distinguish between a mid-tier model and the state-of-the-art, and improvements of even several percentage points would push models into the 95–100% ceiling range where the benchmark provides no further signal.
BBEH is harder but still shows compression: Scores range from 36.1% to 77.0% (40.9 point spread), providing reasonable dynamic range, but top models are approaching or exceeding 75%, suggesting BBEH may saturate within 1–2 model generations.
GENERAL365 provides the widest dynamic range and lowest ceiling: Scores span 26.7% to 62.8% (36.1 point spread), and the ceiling of 62.8% leaves substantial headroom for future progress. The gap between BBH and GENERAL365 for the strongest model is 31.6 percentage points.

The paper presents these results as validating GENERAL365's design goal of being a "high-difficulty benchmark for general reasoning" that "effectively distinguishes the reasoning boundaries of current LLMs in real-world general tasks where existing benchmarks have become almost saturated" (Section 3.3.2).

Diversity Validation Results (Figures 6, 7)

Qualitative: t-SNE embedding visualization (Figure 6). The three panels present 2D projections of query embeddings for GENERAL365 (1,460 instances), BBH (6,511 instances), and BBEH (4,520 instances) using text-embedding-ada-002 and t-SNE with perplexity 30:

GENERAL365 (Figure 6a): "Highly dispersed and uniform distribution" — points are spread broadly across the 2D manifold without visible high-density clusters. The paper interprets this as evidence of "semantic independence achieved through our seed problems" and "effective prevention of semantic redundancy" (Section 3.3.1).
BBH (Figure 6b): "Significant 'local collapse' with isolated, high-density clusters" — tight groupings indicate semantically near-identical problem instances within tasks. Despite having 6,511 instances (~4.5× more than GENERAL365), the benchmark occupies less of the semantic space because template-based expansion concentrates instances around task templates.
BBEH (Figure 6c): Similar to BBH, showing "isolated, high-density clusters" indicative of template homogeneity, despite having 4,520 instances.

Quantitative: Pairwise logical similarity scoring (Figure 7). Distribution of similarity scores on the 0–5 scale for each benchmark:

GENERAL365: Mean similarity μ = 2.16, with 68.2% of instances receiving scores of 1 or 2 (weakly related to superficially similar). Only 3.8% receive the maximum score of 5 (essentially identical). The distribution is left-skewed.
BBH: Mean similarity μ = 4.71, with 77.8% of instances receiving the maximum score of 5. The distribution is extreme right-skewed.
BBEH: Mean similarity μ = 4.80, with 83.5% receiving the maximum score of 5. Even more right-skewed than BBH.

The paper interprets these results as demonstrating that "while existing benchmarks often expand via template-based perturbations—resulting in high structural homogeneity, GENERAL365 preserves a diverse spectrum of independent reasoning patterns, effectively minimizing cognitive redundancy" (Section 3.3.1). The gap between GENERAL365's μ = 2.16 and BBEH's μ = 4.80 is not marginal — it represents a qualitatively different regime of reasoning diversity, where GENERAL365's even most similar instances share only "superficial similarity" (score 2) on average while BBH/BBEH's most similar instances are "essentially identical" (score 5) for the vast majority of pairs.

Ablation Studies and Robustness Checks

This paper is a benchmark contribution, not a method contribution, so formal ablation studies in the traditional ML sense (removing components of a proposed system) are not present. However, several validation analyses serve the analogous function of testing whether the benchmark's claimed properties hold under scrutiny.

Public subset representativeness (Appendix C, Table 2): The paper releases a publicly accessible subset of 180 seed problems (720 instances) and compares model performance on this subset against the full 1,095-instance benchmark. Across 26 models, the maximum deviation between public subset accuracy and full benchmark accuracy is approximately 2.6 percentage points (observed for GPT-5-Thinking: 58.6% full vs. 56.0% public). Most models show deviations under 1.5 percentage points. This confirms that the public subset is representative, enabling community research without compromising the held-out portion's ability to serve as an uncontaminated evaluation set.

Sampling variance under high temperature (Appendix D, Table 3): To verify that the high-temperature inference settings (T = 1.0 for reasoning models, T = 0.7 for non-reasoning models) do not introduce unacceptable stochastic noise, four independent evaluation runs are conducted for five representative models:

Model	Run 1	Run 2	Run 3	Run 4	Δ (max − min)
DeepSeek-V3.2-Thinking	53.4%	54.9%	55.1%	54.9%	1.7%
Kimi-K2-Thinking	51.5%	53.2%	53.0%	53.0%	1.7%
GLM-4.7-Thinking	57.0%	55.4%	57.4%	57.4%	2.0%
LongCat-Flash-Thinking-2601	50.7%	50.9%	51.6%	50.8%	0.9%
LongCat-Flash-2512	40.4%	43.0%	41.0%	41.7%	2.6%

All deviations are under 3%, with the largest (LongCat-Flash-2512 at 2.6%) occurring for the non-reasoning model at T = 0.7. The paper concludes that "our evaluation metrics are reliable and highly reproducible despite the high-temperature setting" (Appendix D). This is an important robustness check because the high temperature for reasoning models (T = 1.0, which is the maximum-entropy setting) could theoretically produce high-variance outputs that undermine the reliability of single-run accuracy estimates. The <3% maximum deviation suggests this is not the case for the evaluated models.

Grading accuracy verification (Section 2.4): The hybrid grading system's accuracy is validated through manual review of 1,460 grading instances spanning four frontier models (o4-mini, Gemini-2.5-Pro, DeepSeek-V3.1-Thinking, LongCat-Flash-Thinking) evaluated on seed problems. The verified grading accuracy of 99.6% means approximately 6 errors in 1,460 grading decisions. At this error rate, the expected number of misgraded instances in a full 1,095-instance evaluation run is approximately 4–5, producing a potential measurement error of roughly 0.4 percentage points — negligible relative to the performance spread across models (25+ percentage points). The paper does not break down grading accuracy by answer type (numerical vs. select vs. text) or by the model being evaluated, so differential grading error across these dimensions cannot be assessed.

Temperature asymmetry between reasoning and non-reasoning models (implicit): While not formally presented as an ablation or robustness check, the paper's evaluation protocol uses different temperatures for reasoning models (T = 1.0) and non-reasoning models (T = 0.7). The higher temperature for reasoning models encourages more diverse sampling, which could advantage them in exploration-heavy problems. Conversely, the lower temperature for non-reasoning models reduces stochastic variance but may limit their ability to explore alternative reasoning paths. The paper does not ablate temperature (e.g., testing reasoning models at T = 0.7 or non-reasoning models at T = 1.0), so the effect of this asymmetry on cross-architecture comparisons cannot be quantified from the reported data.

Evaluator model dependence in similarity scoring (implicit): The pairwise logical similarity scores (Figure 7) are produced by Gemini-3-Pro as the evaluator model. The paper does not report inter-evaluator agreement (e.g., using a second model to independently score a subset of pairs) or analyze whether Gemini-3-Pro exhibits systematic scoring biases. If the evaluator model tends to rate problems from its likely training distribution as more similar than they "objectively" are, the absolute similarity scores could be miscalibrated. However, the paper's primary claim is relative — that GENERAL365 exhibits substantially lower logical redundancy than BBH and BBEH — and relative comparisons are robust to evaluator bias that affects all benchmarks equally under identical scoring protocols.

Cross-validation for strategy selection: Not applicable. GENERAL365 is a benchmark for evaluation, not a method requiring hyperparameter tuning or strategy selection.

Critical Assessment

Claim 1: "Even the top-performing model achieves only 62.8% accuracy, establishing that contemporary LLMs' reasoning abilities remain heavily domain-dependent."

What the experiments actually demonstrate: The leaderboard (Figure 1, Table 1) unequivocally shows that 26 models spanning the current capability spectrum achieve accuracies between 37.6% and 62.8% on GENERAL365. This is the paper's strongest and most straightforward empirical claim, and the evidence supports it without qualification.

What the experiments do NOT demonstrate: The claim that this establishes domain-dependence of reasoning — that the gap between GENERAL365 scores and math/physics benchmark scores proves reasoning is domain-specific — requires a comparison that the paper does not fully execute. To demonstrate domain-dependence, one would need to show that the same models achieve near-perfect scores on domain-specific benchmarks and substantially lower scores on GENERAL365, and that this gap is larger than would be expected from difficulty alone. The paper provides cross-benchmark comparisons (Figure 8) but only for BBH and BBEH (other general reasoning benchmarks), not for AIME, GPQA, or SWE-bench. The claim about domain-dependence in Section 1 references math and physics benchmarks where "top-tier LLMs have achieved gold-medal-level scores," but the paper does not measure those same models on those benchmarks and show the differential. The evidence for domain-dependence is therefore largely anecdotal (citing prior work's results on AIME/IMO/IPhO) rather than experimentally established within this paper. A stronger design would have evaluated a subset of the 26 models on both GENERAL365 and a domain-specific benchmark (e.g., AIME 2025) to directly quantify the reasoning-knowledge confound.

Conditional assessment: The 62.8% ceiling is well-supported. The interpretation that this demonstrates domain-dependence is plausible but would be strengthened by within-paper cross-benchmark evidence against domain-specific tasks, not just against BBH and BBEH.

Claim 2: "GENERAL365 exhibits substantially higher diversity than existing general reasoning benchmarks, as evidenced by embedding dispersion and low pairwise logical similarity."

What the experiments actually demonstrate: The diversity validation methodology is the paper's most impressive technical contribution. The t-SNE visualization (Figure 6) provides qualitative evidence of semantic dispersion, and the pairwise similarity scoring (Figure 7) provides quantitative evidence of logical independence. The numbers are striking: μ = 2.16 for GENERAL365 vs. μ = 4.71 for BBH and μ = 4.80 for BBEH on a 0–5 scale. The difference between "superficially similar" (score 2) and "essentially identical" (score 5) is large and meaningful. The 77.8% and 83.5% of BBH/BBEH instances receiving maximum similarity scores versus 3.8% for GENERAL365 is a dramatic contrast that supports the diversity claim strongly.

Genuine weaknesses: (1) The similarity scoring uses a single evaluator model (Gemini-3-Pro) without reporting inter-evaluator agreement. If Gemini-3-Pro has systematic biases in similarity assessment, the absolute scores could be miscalibrated, though the relative comparison across benchmarks would be affected only if the bias is benchmark-dependent (e.g., Gemini-3-Pro might rate problems within its training distribution differently). (2) The evaluation compares GENERAL365 (1,460 instances) against BBH (6,511 instances) and BBEH (4,520 instances). The larger instance counts for BBH/BBEH might inherently inflate the chance of finding near-identical pairs (more instances → more opportunities for the nearest neighbor to be very close). The paper's use of nearest-neighbor retrieval partially addresses this (each instance is compared to its single closest neighbor regardless of dataset size), but the density of the embedding space could still affect similarity scores — in a denser space, nearest neighbors will tend to be closer, which could inflate similarity scores for larger benchmarks independent of their template diversity. (3) The embedding model (text-embedding-ada-002) is itself an LLM, and its representation of semantic similarity may be influenced by its own training distribution, potentially encoding different notions of "similarity" for problems from different sources. This is a subtle confound that the paper does not discuss.

Missing analysis: The paper does not report the relationship between semantic distance (embedding similarity) and logical similarity (0–5 score). If semantically close instances (low embedding distance) always receive high logical similarity scores, the embedding analysis alone would suffice. If the correlation is weak, it would highlight the value of the more expensive logical similarity scoring. Reporting this correlation would strengthen the methodological contribution.

Claim 3: "Semantic Interference and Optimal Strategy emerge as the primary performance bottlenecks for current LLMs."

What the experiments actually demonstrate: Table 1 provides per-category scores for all 26 models. Across top-performing models, Semantic Interference and Optimal Strategy scores are consistently ~5–15 percentage points below overall accuracy (e.g., Gemini-3-Pro: 62.8% overall, 55.7% Semantic Interference, 50.8% Optimal Strategy). This pattern is consistent across models and architectures. The radar chart (Figure 5) makes this visually evident — the "Semantic Interference" and "Optimal Strategy" axes show a visible inward dent relative to other categories for most model series.

Caveats on interpretation: (1) The categories have different numbers of seed problems (Figure 3a shows Complex Constraints as most prevalent, Probability & Uncertainty as least). Per-category scores are therefore computed over different sample sizes and different problem compositions, which could introduce variance independent of genuine difficulty differences. (2) The paper does not control for problem difficulty within categories — it's possible that Semantic Interference problems are simply harder independent of their category label, rather than being harder because they test semantic interference. The category label is a design intention, not a verified property of the problems independent of difficulty. (3) The multi-label problem structure (Figure 3b: ~70% of problems have 2+ labels) means per-category scores are not cleanly separable. A problem labeled as both "Complex Constraints" and "Semantic Interference" contributes to both categories' scores, so differential category performance may reflect interactions between challenge types rather than pure per-category difficulty.

Missing analysis: The paper does not report per-category difficulty calibration — e.g., having human raters (or multiple LLMs) solve a subset of problems without time pressure to establish per-category "solvability" baselines. This would help distinguish whether low scores on Optimal Strategy reflect the inherent difficulty of strategic planning problems or simply that the specific Optimal Strategy problems in GENERAL365 happen to be harder than the specific Complex Constraints problems.

Claim 4: "Gemini-3-Pro achieves state-of-the-art performance while maintaining a remarkably concise average output of approximately 14k tokens, demonstrating superior reasoning efficiency."

What the experiments actually demonstrate: Figure 4 provides clear evidence that Gemini-3-Pro (62.8%, ~14k tokens) achieves higher accuracy with fewer output tokens than competitors like GLM-5-Thinking (59.9%, ~25–30k tokens) and DeepSeek-V3.2-Speciale (57.5%, ~25–30k tokens). It is unambiguously on the accuracy-efficiency Pareto frontier.

What the experiments do NOT demonstrate: (1) The paper does not establish that output tokens are a valid proxy for computational cost. Different models may have different per-token FLOP costs (depending on architecture, parameter count, KV-cache implementation, etc.), so "fewer tokens" does not necessarily mean "less computation." (2) The efficiency analysis does not control for model size — Gemini-3-Pro may achieve its efficiency through larger parameter count (more FLOPs per token but fewer tokens) rather than genuinely superior reasoning density. Without parameter count or per-token FLOP estimates, the efficiency comparison is purely about output length, not total computational cost. (3) The paper does not compare models at matched accuracy (e.g., how many tokens does Gemini-3-Pro need to achieve 57.5%, matching DeepSeek-V3.2-Speciale's accuracy?) or matched token budget (e.g., if DeepSeek-V3.2-Speciale were constrained to 14k tokens, what accuracy would it achieve?). These controlled comparisons would provide stronger evidence for efficiency differences than the observed scatter plot.

Single-data-point concern: The claim about Gemini-3-Pro is based on one model. The paper does not establish that reasoning efficiency varies systematically across architectures in ways that are informative about reasoning capability rather than about training objectives (e.g., a model trained to be concise might achieve efficiency through brevity bias rather than reasoning quality).

Claim 5: "BBH has become essentially saturated as a general reasoning benchmark."

What the experiments actually demonstrate: Figure 8 shows BBH scores of 89.4% (GPT-4.1), 89.6% (o3-mini), 92.0% (GPT-5-Thinking), and 94.4% (Gemini-3-Pro). The 5.0 percentage point spread across models with very different GENERAL365 scores (26.7% to 62.8%) strongly supports the saturation claim — BBH has largely stopped discriminating between models of different capability levels.

Interpretation robust: This finding is among the paper's most solid. The contrast between BBH's compressed scores and GENERAL365's wide dynamic range is stark and does not depend on subtle methodological choices.

One nuance: The paper's claim that BBH's saturation is due to "model advancement" (i.e., models got better) is confounded with potential training data contamination — it's possible that BBH problems or their reasoning patterns appeared in pretraining corpora of more recent models, inflating scores independent of genuine reasoning improvement. The paper's originality requirement for GENERAL365 seeds (problems must be "non-replicable via standard web searches," Section 2.2) is an attempt to address this, but it doesn't establish that contamination is the mechanism for BBH's saturation. It could be that models genuinely became better at BBH's specific reasoning patterns through legitimate generalization, not memorization.

Cross-Cutting Strengths and Weaknesses of the Experimental Design

Strengths:

Scale of model coverage (26 models, 8 providers) is exceptional for a benchmark paper. Most benchmark introductions evaluate 5–10 models. The breadth across proprietary and open-source, reasoning and non-reasoning architectures provides a comprehensive snapshot of the current capability landscape and makes the findings about performance ceilings and category bottlenecks more generalizable.
The diversity validation methodology is rigorous and replicable. The combination of embedding visualization and pairwise logical similarity scoring provides converging qualitative and quantitative evidence for GENERAL365's claimed diversity advantage. The prompt template for similarity scoring (Example 10) is provided in full, enabling other researchers to apply the same methodology to other benchmarks — a genuine methodological contribution beyond the benchmark itself.
The per-category diagnostic analysis (Table 1) provides actionable insights. Identifying Semantic Interference and Optimal Strategy as specific bottlenecks with quantitative evidence across 26 models gives model developers concrete targets for improvement. This is more useful than a single aggregate score.
The openness about limitations is commendable. The paper reports sampling variance (Appendix D), releases a public subset and confirms its representativeness (Appendix C), and documents grading accuracy (Section 2.4). These are practices that many benchmark papers omit.

Weaknesses:

The claim about domain-dependence of reasoning is not experimentally established within this paper. The paper argues that domain-specific benchmarks conflate reasoning with knowledge and that GENERAL365 decouples them, but it does not evaluate the same models on both types of benchmarks to quantify the confound. This is a central interpretive claim that rests on cross-paper comparisons rather than within-paper experiments. A simple addition — evaluating 4–5 representative models on one domain benchmark (e.g., AIME 2025) alongside GENERAL365 — would have directly tested the domain-dependence hypothesis.
The evaluator dependence in similarity scoring is unexamined. The logical similarity scores that underpin the paper's strongest diversity claim are produced by Gemini-3-Pro without inter-evaluator reliability analysis. If the evaluator model has systematic biases (e.g., rating problems as more similar when they share surface features that the model is sensitive to), the absolute scores could be miscalibrated. The relative comparison across benchmarks mitigates but does not eliminate this concern.
Output length as a proxy for reasoning effort is unvalidated. The efficiency analysis (Figure 4) and the cross-benchmark difficulty analysis (Figure 9) both rely on the assumption that longer outputs reflect deeper reasoning rather than verbosity, architectural differences in generation behavior, or differences in problem length. The paper provides no validation of this assumption (e.g., by correlating output length with human-judged reasoning depth on a sample of problems). The claim that "the increased token usage in GENERAL365 is not merely a sign of verbosity but a direct consequence of the taxing cognitive load" is asserted, not demonstrated.
Temperature asymmetry between reasoning and non-reasoning models is not ablated. The evaluation protocol uses T = 1.0 for reasoning models and T = 0.7 for non-reasoning models. This choice may advantage reasoning models in exploration-heavy problems and disadvantage non-reasoning models. Without temperature ablation (or at minimum a justification for the asymmetry beyond "to comprehensively explore reasoning trajectories"), the cross-architecture comparisons are confounded.
No per-category difficulty calibration. The finding that Semantic Interference and Optimal Strategy are bottlenecks (Table 1) could reflect either genuine capability gaps or the specific difficulty of the GENERAL365 problems assigned to those categories. Without independent difficulty calibration (e.g., human expert solve rates, or per-category item response theory parameters), the category-level difficulty is confounded with problem selection.
Missing analysis of inter-category correlations. The paper reports per-category scores but does not analyze the correlation structure among categories. If category scores are highly correlated (which the paper hints at by noting "models that achieve SOTA performance in specific challenge categories also tend to rank high in overall accuracy"), then the diagnostic value of eight separate categories is reduced — they may be measuring a largely unidimensional construct with noise rather than eight distinct capabilities. A correlation matrix or factor analysis would clarify this.
No confidence intervals on leaderboard scores. Despite conducting four-run stability analysis for five models (Appendix D), the paper does not report confidence intervals or error bars on the main leaderboard (Figure 1) or per-category scores (Table 1). Given the stochastic variance (Δ up to 2.6%), some of the fine-grained ranking differences (e.g., GLM-4.7-Thinking at 57.4% vs. Qwen3-Max-Thinking at 57.2%) may fall within measurement noise. Without confidence intervals, readers cannot assess which ranking differences are statistically reliable.

Missing experiments that would have strengthened the paper:

Within-paper domain-specific comparison: Evaluating a subset of the 26 models on a domain benchmark (AIME, GPQA, or MATH) to directly quantify the reasoning-knowledge confound that motivates GENERAL365.
Inter-evaluator agreement for similarity scoring: Having a second evaluator model (e.g., GPT-5-Thinking) independently score a random subset of GENERAL365, BBH, and BBEH instance pairs to assess reliability of the similarity scores.
Temperature ablation: Running at least one reasoning model at T = 0.7 and one non-reasoning model at T = 1.0 to quantify the effect of temperature asymmetry on cross-architecture comparisons.
Per-category difficulty calibration: Having human experts (or a "slow" reasoning system with unlimited time) solve a sample of problems from each category to establish per-category solvability baselines independent of LLM capability.
Correlation or factor analysis of category scores: Analyzing whether the eight categories measure distinct capabilities or a largely unidimensional construct, which would inform whether per-category breakdowns provide genuine diagnostic value beyond the aggregate score.
Controlled efficiency comparison: Matching models on token budget (e.g., constraining all models to 14k output tokens and measuring accuracy) or accuracy (e.g., measuring tokens needed to reach 50% accuracy) rather than purely descriptive scatter plots.

6. Limitations and Trade-offs

1. The K-12 Knowledge Constraint Is Asserted Rather Than Empirically Validated

The assumption or constraint. The paper's central measurement validity claim is that GENERAL365 decouples reasoning from domain-specific expertise by restricting required background knowledge to a K-12 level (Section 1, Section 3). The paper asserts that "the knowledge required is strictly confined to the K-12 scope, ensuring the dataset measures a model's reasoning capabilities rather than knowledge retrieval." However, the paper provides no empirical validation that this constraint actually holds — no systematic audit of which K-12 concepts problems require, no verification by K-12 educators that the problems are solvable using only K-12 knowledge, and no analysis of whether certain GENERAL365 problems inadvertently draw on advanced concepts that a typical K-12 student would not possess.

The consequence. If some GENERAL365 problems require knowledge beyond K-12 level (e.g., advanced logical notation, mathematical concepts typically taught in university, or domain-specific reasoning patterns that appear more frequently in graduate-level training corpora), then the benchmark is measuring a mixture of reasoning and knowledge — precisely the confound it claims to eliminate. In that case, performance differences between models could reflect differential exposure to the relevant knowledge in pretraining rather than genuine reasoning capability. For example, the calendar manipulation problem (Example 6) involves reasoning about leap year rules across a deliberately contorted calendar system — while the individual facts are K-12 (months have 30 or 31 days, February is exceptional), the combinatorial manipulation of these rules may require cognitive strategies that correlate with advanced mathematical training, creating an implicit knowledge gradient.

More subtly, even if the declarative knowledge required is K-12, the procedural reasoning patterns tested by GENERAL365 may be more prevalent in certain training distributions. A problem about logical constraint propagation (Complex Constraints) may be solvable with K-12 facts, but models trained predominantly on code or mathematics may have encountered the reasoning pattern far more frequently than models trained on general web text. The K-12 constraint controls for declarative knowledge but not for procedural familiarity, which means the benchmark may still inadvertently favor models with exposure to formal reasoning domains — a subtler form of the knowledge confound.

What evidence exists in the paper. None. The paper states the K-12 constraint as a design intention but provides no verification mechanism, no audit results, and no discussion of edge cases where the constraint might fail. The construction pipeline describes problem verification against the eight challenge categories and for originality (Section 2.2), but not against the K-12 knowledge constraint specifically. The lack of verification is particularly concerning given the diversity of problem sources — the paper states problems were crowdsourced "from diverse, real-world domains," and different contributors may have different conceptions of what constitutes K-12-level knowledge.

Mitigation status. The paper does not acknowledge this as a limitation, does not attempt to verify the constraint empirically, and does not propose future work to address it. The K-12 constraint is treated as satisfied by design rather than as a hypothesis requiring validation. This is the paper's most significant unaddressed validity threat because it concerns the foundational measurement claim on which the benchmark's value proposition rests.

2. Difficulty Estimation Cost Is Not Accounted for and Practical Deployment Feasibility Is Unclear

The assumption or constraint. The benchmark's construction methodology assumes that high-quality seed problems can be produced through manual curation, difficulty-filtered by testing against top models, and validated through human review — a process that requires substantial expert labor, access to the best available LLMs at construction time, and iterative refinement. The paper reports that 365 seed problems were produced through crowdsourcing, LLM-based variant generation (10 candidates per seed with human quality review), manual inspection for correctness and consistency, and filtering to remove problems solvable by top-tier models (Section 2.2). However, the paper provides no accounting of the resources required: number of contributors, person-hours, cost of expert review, number of iterations between model testing and problem refinement, or the rejection rate during difficulty filtering.

The consequence. This is not merely an omission about construction methodology — it has direct consequences for the benchmark's long-term viability and for the replicability of its approach. GENERAL365's core value proposition is that it resists saturation by filtering out problems that current models can solve. As models improve, this filtering must be repeated: the problems that were challenging at construction time may become easy for the next generation of models, and new, harder problems must be sourced to maintain the benchmark's discriminative power. If the construction process is expensive and labor-intensive (requiring expert crowdsourcing, manual verification, and iterative model testing), then maintaining GENERAL365 as a living benchmark that tracks the reasoning frontier will require sustained investment. The paper provides no analysis of this maintenance cost or a plan for ongoing benchmark updates.

Furthermore, the difficulty filtering pipeline creates a subtle circularity: problems are selected because current top models struggle with them. This means GENERAL365 is explicitly designed to be hard for the generation of models that existed at construction time. But whether this difficulty transfers to future models with fundamentally different architectures, training paradigms, or reasoning strategies is unknown. The benchmark's difficulty may be an artifact of specific weaknesses in current architectures (e.g., susceptibility to semantic interference, poor strategic planning) rather than a stable measure of "general reasoning" that will challenge all future systems equally. A future model that excels at semantic interference but struggles with, say, causal attribution might find GENERAL365 easy while being genuinely poor at other dimensions of reasoning — a failure mode the benchmark's construction methodology cannot detect in advance.

What evidence exists in the paper. None directly. The paper does not report any metrics on construction cost, expert labor, rejection rates during filtering, or the number of problems discarded for being "easily solvable." The diversity validation (Section 3.3.1) and difficulty validation (Section 3.3.2) confirm that the output of the construction process has the claimed properties, but the cost of achieving those properties is opaque. The paper's closest acknowledgment is in the description of the quality review process: "human-experts performed a stringent quality review, discarding incorrect or illogical entries" and "for seed problems with fewer than three valid expansions, manual intervention was employed to supplement the data" (Section 2.2) — phrasing that implies substantial human effort without quantifying it.

Mitigation status. The paper acknowledges this limitation implicitly by releasing a public subset (180 of 365 seeds) to enable community research without exposing the full benchmark to contamination (Appendix C), which suggests awareness that the full benchmark is a valuable asset whose construction cost justifies keeping part of it private. However, the paper does not discuss the scalability of its construction methodology, the cost of future updates, or alternative approaches to maintaining benchmark difficulty (e.g., adversarial problem generation, dynamic difficulty calibration). The maintenance challenge is acknowledged only to the extent that the public/private split implies the benchmark is a finite resource that must be protected from overexposure — not that the construction process itself may be too expensive to repeat as models advance.

3. The Benchmark Validates Diversity and Difficulty but Does Not Validate That It Measures What It Claims to Measure (Construct Validity)

The assumption or constraint. GENERAL365 claims to measure "general reasoning" — a construct the paper operationalizes through the eight-category taxonomy (Section 2.1) and the K-12 knowledge constraint. The paper provides extensive evidence for two properties of its benchmark: diversity (Figures 6, 7) and difficulty (Figures 8, 9). But it provides no evidence for construct validity — the claim that the eight categories collectively capture the most important dimensions of general reasoning, that performance on GENERAL365 correlates with real-world reasoning outcomes, or that the benchmark's assessment of model capabilities aligns with human judgments of reasoning quality.

The consequence. A benchmark can be diverse and difficult while still measuring the wrong thing. For example, if GENERAL365 predominantly tests a narrow form of constraint-satisfaction reasoning under artificial puzzle conditions, models that excel at these puzzles may not transfer their skills to the messy, open-ended reasoning tasks that practitioners actually care about (legal analysis, medical diagnosis, business strategy, scientific hypothesis generation). The paper's claim that GENERAL365 targets "real-world daily reasoning" (Section 1) is asserted, not demonstrated — none of the validation analyses connect benchmark performance to real-world reasoning outcomes, expert judgments of reasoning quality, or established cognitive assessments of general reasoning ability in humans.

This absence of construct validation has specific downstream consequences for how the benchmark should be used. The paper claims that model developers should focus improvement efforts on Semantic Interference and Optimal Strategy based on per-category scores (Section 3.2). But if GENERAL365's categories do not correspond to separable cognitive dimensions that generalize beyond the benchmark, then optimizing for per-category scores may produce models that are better at GENERAL365 without being better at general reasoning — a classic case of "teaching to the test" enabled by a benchmark that claims construct validity without establishing it.

The cross-benchmark comparison with BBH and BBEH (Figure 8) demonstrates that GENERAL365 measures something different from those benchmarks, but difference is not the same as validity. It is possible that BBH measures genuine reasoning (but has become saturated) while GENERAL365 measures a different construct — puzzle-solving ability under artificial constraints — that is less correlated with real-world reasoning. Without external validation, users of the benchmark cannot distinguish between "GENERAL365 measures general reasoning better than BBH" and "GENERAL365 measures a different, less relevant construct than BBH at a harder difficulty level."

What evidence exists in the paper. None for construct validity. The paper's validation efforts focus entirely on internal properties of the benchmark (diversity, difficulty, grading accuracy) rather than on the relationship between benchmark scores and external criteria. The paper does not:

Correlate GENERAL365 scores with human evaluations of reasoning quality on the same problems.
Compare GENERAL365 performance to established cognitive assessments or real-world reasoning outcomes.
Conduct factor analysis to determine whether the eight categories measure distinct or overlapping constructs.
Validate that per-category scores have discriminant validity (i.e., that Semantic Interference scores predict performance on semantically-interfering tasks outside GENERAL365 better than other categories do).
Establish test-retest reliability beyond the four-run sampling variance analysis in Appendix D (which measures stochastic consistency, not construct stability).

The paper's claim that "models that achieve SOTA performance in specific challenge categories also tend to rank high in overall accuracy, suggesting that general reasoning proficiency is a composite of these foundational abilities" (Section 3.2) actually hints at a potential construct validity problem: if all category scores are highly correlated (as the positive manifold suggests), then the eight-category breakdown may be measuring a largely unidimensional construct with category-specific noise rather than eight distinct reasoning capabilities. This would undermine the diagnostic value of per-category analysis — the finding that Semantic Interference and Optimal Strategy are "bottlenecks" could simply reflect that those categories contain harder problems, not that they measure distinct cognitive deficits.

Mitigation status. The paper does not acknowledge construct validity as an open question, does not conduct construct validation analyses, and does not propose future work to establish the relationship between GENERAL365 scores and external reasoning outcomes. The benchmark is presented as measuring "general reasoning" by design rather than by validation — the taxonomy defines what "general reasoning" means, and the construction pipeline ensures problems align with the taxonomy, but the chain from taxonomy to construct validity is assumed rather than tested.

4. The Per-Category Diagnostic Analysis Is Confounded by Unequal Category Sizes, Multi-Label Problems, and Uncalibrated Difficulty

The assumption or constraint. The paper's fine-grained diagnostic analysis (Table 1, Figure 5) identifies "Semantic Interference" and "Optimal Strategy" as primary performance bottlenecks based on per-category accuracy scores that are consistently ~10 percentage points below models' overall averages. This analysis assumes that per-category scores are comparable — that a score of 50.8% on Optimal Strategy means the same thing (in terms of capability deficit) as a score of 65.3% on Complex Constraints. The analysis also assumes that category labels cleanly separate reasoning challenges, enabling attribution of performance differences to specific cognitive capabilities.

The consequence. Three structural features of GENERAL365's design undermine the comparability of per-category scores and complicate causal attribution of performance differences:

First, unequal category sizes (Figure 3a). Complex Constraints is the most prevalent category, while Probability & Uncertainty contains the fewest problems (over 20 seeds, but substantially fewer than Complex Constraints). Per-category scores are computed over different numbers of instances with different statistical reliability and potentially different difficulty distributions. A category with 50 problems may have higher variance in difficulty (some very easy, some very hard) while a category with 20 problems may have a tighter difficulty distribution, making per-category means less comparable. The paper does not report variance or confidence intervals for per-category scores, so readers cannot assess whether the ~10 percentage point gap between Complex Constraints and Optimal Strategy (for most top models) is statistically reliable or within sampling noise for the smaller categories.

Second, multi-label problem structure (Figure 3b). The paper states that "nearly 70% of the problems are annotated with two or more challenge tags." This means a single problem contributes to multiple category scores. If a problem is labeled as both "Complex Constraints" and "Semantic Interference," and the model fails, the failure is attributed to both categories — but the actual reason for failure might be semantic interference alone, complex constraints alone, an interaction between them, or neither (e.g., a simple calculation error). The multi-label structure inflates inter-category correlations and makes it impossible to attribute differential performance to specific cognitive deficits without additional analysis (e.g., comparing performance on single-label vs. multi-label problems within each category, or analyzing error patterns to identify which challenge caused the failure).

The finding that Semantic Interference and Optimal Strategy are "bottlenecks" is particularly vulnerable to this confound. If Optimal Strategy problems are disproportionately also labeled as Complex Constraints and Branching & Enumeration (because strategic planning under constraints inherently involves both constraint satisfaction and search), then low Optimal Strategy scores may reflect the compound difficulty of multiple overlapping challenges rather than a specific deficit in strategic planning per se. In that case, the diagnostic claim ("current LLMs struggle with long-term strategic planning") overstates the evidence.

Third, uncalibrated problem difficulty within categories. The paper does not establish that problems within each category have comparable difficulty or that difficulty differences between categories are controlled. It is possible that the specific Optimal Strategy problems in GENERAL365 are simply harder (require more reasoning steps, involve more entities, have more intricate constraint interactions) than the specific Complex Constraints problems, independent of the reasoning type. In that case, the "Optimal Strategy bottleneck" reflects the difficulty of the particular problems selected for that category rather than a general capability gap in strategic planning. Without independent difficulty calibration (e.g., expert human solve rates, item response theory difficulty parameters, or performance of a "slow" reasoning system given unlimited time), per-category score differences cannot be disentangled from per-category problem difficulty differences.

What evidence exists in the paper. The paper acknowledges these structural features (unequal category sizes: Figure 3a; multi-label distribution: Figure 3b) but does not discuss their implications for the diagnostic analysis. The multi-label structure is presented positively ("further underscoring that GENERAL365 is characterized by high-difficulty, composite challenges rather than isolated reasoning tasks," Section 2.3) without noting that composite challenges make per-category attribution ambiguous. The paper does not report per-category difficulty calibration, confidence intervals on per-category scores, or analyses that attempt to isolate category-specific effects (e.g., comparing single-label and multi-label instances within each category, or conducting ablation-style analyses where problems are sorted by labeling to identify unique category contributions).

Mitigation status. The paper does not treat the confounding of per-category analysis as a limitation. The multi-label structure is framed as a feature (real-world reasoning involves overlapping cognitive demands), which is defensible, but the paper does not acknowledge the tension between this design choice and the diagnostic use case — the more problems carry multiple labels, the less cleanly category scores reflect distinct capabilities. The paper does not propose analyses that could partially address this confound (e.g., reporting per-category scores on single-label subsets, conducting item-level analyses, or using statistical techniques to estimate unique category contributions), and does not flag this as a direction for future work.

5. Temperature Asymmetry Between Reasoning and Non-Reasoning Models Confounds Cross-Architecture Comparisons

The assumption or constraint. The evaluation protocol (Section 3.1) uses temperature T = 1.0 for reasoning models and T = 0.7 for non-reasoning models. The paper justifies the higher temperature for reasoning models as intended "to comprehensively explore the reasoning trajectories when facing the diverse tasks of GENERAL365" and notes that for models supporting adjustable reasoning effort, the "highest available setting" was used "to elicit their peak reasoning performance." The implicit assumption is that temperature differences do not systematically advantage one architecture over the other in ways that affect the headline finding that reasoning models outperform non-reasoning models.

The consequence. Temperature controls the randomness of token sampling — higher temperature produces more diverse outputs by flattening the token probability distribution, while lower temperature produces more deterministic outputs concentrated on high-probability tokens. The asymmetric temperature settings create two confounds for cross-architecture comparisons:

First, reasoning models benefit from more exploration. At T = 1.0, reasoning models sample from their full output distribution, enabling them to explore alternative reasoning paths that might be pruned at lower temperatures. GENERAL365 problems are specifically designed to require non-obvious reasoning approaches — the Semantic Interference category requires suppressing commonsense priors, the Optimal Strategy category requires evaluating multiple action sequences. Exploration is likely beneficial for these problem types because the most probable reasoning path (which a lower temperature would favor) may be the commonsense path that leads to the semantic interference trap. By giving reasoning models higher temperature, the evaluation protocol may advantage them on exactly the types of problems where GENERAL365 is most challenging.

Second, non-reasoning models are evaluated at a setting that may underestimate their capability. At T = 0.7, non-reasoning models produce less diverse outputs that may be insufficient for problems requiring exploration of alternative reasoning strategies. A non-reasoning model that could, in principle, solve a GENERAL365 problem through a non-obvious reasoning path might fail at T = 0.7 because it defaults to the higher-probability (but incorrect) path, while the same model at T = 1.0 might have sampled the correct path. The paper's conclusion that "some non-reasoning models, such as Qwen3-Max-Instruct and LongCat-Flash, outperform several dedicated reasoning models" (Section 3.2) is actually strengthened by this confound (non-reasoning models are succeeding despite the temperature disadvantage), but the magnitude of the gap between reasoning and non-reasoning architectures may be artificially inflated by the temperature asymmetry.

The practical consequence is that practitioners cannot use GENERAL365 scores to directly compare reasoning and non-reasoning architectures for their deployment decisions. A practitioner choosing between a reasoning model at T = 1.0 and a non-reasoning model at T = 0.7 on the basis of GENERAL365 scores would be comparing models under different evaluation conditions, making the comparison potentially misleading.

What evidence exists in the paper. The sampling variance analysis (Appendix D, Table 3) provides some temperature-related evidence but is incomplete for diagnosing this confound. The analysis shows that for four reasoning models at T = 1.0 (DeepSeek-V3.2-Thinking, Kimi-K2-Thinking, GLM-4.7-Thinking, LongCat-Flash-Thinking-2601), the maximum deviation across four runs is 0.9–2.0%. For one non-reasoning model at T = 0.7 (LongCat-Flash-2512), the deviation is 2.6%. This suggests that variance is acceptably low under both temperature settings, but it does not address the confound: the question is not whether scores are stable at each temperature but whether the mean scores differ systematically between temperatures for the same model. The paper does not report temperature ablation results — e.g., running a reasoning model at T = 0.7 or a non-reasoning model at T = 1.0 — that would quantify the effect of temperature on cross-architecture score differences.

Mitigation status. The paper does not acknowledge temperature asymmetry as a potential confound, does not conduct temperature ablation experiments, and does not discuss the implications of temperature choice for cross-architecture comparisons. The stability analysis (Appendix D) addresses sampling variance within a fixed temperature setting but does not address systematic effects of temperature on mean performance. A simple addition — evaluating one reasoning model at both T = 0.7 and T = 1.0, and one non-reasoning model at both temperatures — would have provided evidence for or against the temperature confound but is absent from the paper.

6. Single-Dataset Design Limits Generalizability and Prevents Understanding of How Reasoning Difficulty Transfers Across Domains

The assumption or constraint. GENERAL365 is a single benchmark comprising 365 human-curated seed problems with 1,095 evaluation instances. All validation analyses (diversity, difficulty, cross-benchmark comparison) are conducted within the scope of this single dataset, and all claims about LLM reasoning capabilities are supported by performance on GENERAL365 alone. The paper implicitly assumes that GENERAL365's eight-category taxonomy and K-12 knowledge constraint capture a sufficient range of "general reasoning" that findings from this benchmark generalize to reasoning in other contexts.

The consequence. The paper's core claims — that reasoning is heavily domain-dependent, that Semantic Interference and Optimal Strategy are primary bottlenecks, that Gemini-3-Pro's 62.8% ceiling represents the current frontier of general reasoning — are all contingent on GENERAL365's specific problem composition. Without evaluation on other reasoning benchmarks that also target knowledge-orthogonal reasoning (e.g., KOR-Bench, which the paper cites but does not benchmark against), there is no way to assess whether the observed patterns are properties of LLM reasoning in general or properties of GENERAL365's specific problem selection.

Several specific generalizability questions are unaddressed:

Does the difficulty hierarchy across categories replicate? If another benchmark tested the same eight reasoning categories with different problems, would Semantic Interference and Optimal Strategy still emerge as the hardest? Or is this finding an artifact of the specific problems GENERAL365 uses to represent those categories?
Does Gemini-3-Pro's efficiency advantage generalize? The finding that Gemini-3-Pro achieves top accuracy with ~14k tokens vs. competitors' ~25-30k tokens (Figure 4) is based on performance on GENERAL365. If Gemini-3-Pro achieves this efficiency through architectural features that are well-suited to the specific types of constraint-satisfaction puzzles in GENERAL365, its efficiency advantage may not transfer to other reasoning tasks (e.g., legal reasoning, scientific hypothesis evaluation, ethical deliberation).
Are the ranking differences between models stable across reasoning domains? The leaderboard (Figure 1) shows GLM-5-Thinking (59.9%) outperforming GPT-5-Thinking (58.6%) by 1.3 percentage points. If this gap is within GENERAL365's measurement noise (the sampling variance analysis in Appendix D shows Δ up to 2.0% for some models, but confidence intervals are not reported for the leaderboard), the ranking may reverse on a different reasoning benchmark. Without multi-benchmark evaluation, readers cannot distinguish genuine capability differences from benchmark-specific advantages.

The most significant consequence is for the paper's foundational claim about domain-dependence of reasoning. The paper argues that the gap between GENERAL365 scores (26.7–62.8%) and domain-specific benchmark scores (near-perfect on AIME, IMO, etc., as cited from prior work) demonstrates that reasoning is domain-dependent. But this gap is observed across different benchmarks with different problem compositions, not within a controlled experimental design. It is possible that GENERAL365 is simply a harder benchmark — regardless of domain — than AIME or IMO, and that the performance gap reflects difficulty rather than domain specificity. To demonstrate domain-dependence, one would need to show that the same models exhibit differential capability across domains of comparable difficulty. GENERAL365 alone cannot provide this evidence.

What evidence exists in the paper. The cross-benchmark comparisons in Section 3.3.2 (Figure 8) compare GENERAL365 to BBH and BBEH — other general reasoning benchmarks, not domain-specific ones. The paper's description of domain-specific benchmark performance (Section 1) cites external sources (OpenAI, 2026; Hubert et al., 2025; Huang, 2026) rather than conducting within-paper evaluations. The diversity validation (Section 3.3.1) confirms that GENERAL365 is more diverse than BBH and BBEH, but diversity is a property of the benchmark's internal structure, not evidence that its findings generalize to other reasoning contexts.

Mitigation status. The paper acknowledges the scope limitation to some extent by positioning GENERAL365 as complementing rather than replacing domain-specific benchmarks (Section 4) and by releasing a public subset "to facilitate community research and ensure reproducibility" (Appendix C). However, the paper does not discuss generalizability as a limitation of its claims, does not evaluate models on additional benchmarks to establish convergent validity, and does not propose multi-benchmark evaluation as future work. The single-dataset design is treated as sufficient for the claims being made, but the strength of those claims (about the nature of LLM reasoning in general, not just about performance on GENERAL365) exceeds what a single benchmark can establish without external validation.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper shifts the conversation around LLM reasoning evaluation from a domain-centric paradigm—where mathematical olympiad scores and physics competition performance serve as the primary evidence of reasoning capability—toward a construct-centric paradigm that asks not just "how well does the model reason?" but "what kind of reasoning are we actually measuring?" The distinction matters because the paper provides concrete, quantitative evidence that these two paradigms yield dramatically different answers: the same models that approach near-perfect scores on domain benchmarks and on template-homogeneous general reasoning benchmarks (94.4% on BBH for Gemini-3-Pro) achieve barely above 60% on GENERAL365, with the majority of models failing to reach even that threshold (Figure 1). This is not an incremental gap—it is a qualitative difference in what the evaluation reveals.

The magnitude of this shift should be understood as a reframing with diagnostic consequences, not a paradigm shift in the Kuhnian sense. The paper does not argue that domain-specific benchmarks are invalid or that their results are meaningless. Rather, it argues that they answer a different question than the one the field increasingly needs answered: "Can this model reason in contexts where it cannot lean on memorized domain patterns?" This reframing changes how benchmark results should be interpreted—not as unidimensional measures of "reasoning ability" but as measurements along specific dimensions that interact with model training distributions in complex ways. A model that scores 95% on AIME and 55% on GENERAL365 is not simply "good at math but bad at reasoning"—it is good at mathematical reasoning under conditions where its training distribution provides relevant patterns, and substantially weaker when those patterns are absent or actively misleading. Both numbers are true, and both matter for deployment decisions, but neither alone tells the full story.

The paper resolves a latent tension in the reasoning evaluation literature that has been building as model scores on existing benchmarks have compressed. BBH was introduced in 2023 specifically because BIG-Bench tasks were too easy for contemporary models; by 2026, BBH itself shows 89–94% accuracy for top models (Figure 8), rendering it nearly useless for discriminating between frontier systems. BBEH addressed the difficulty problem but amplified the diversity problem—the paper's similarity analysis (Figure 7) shows BBEH with μ = 4.80, meaning its instances are even more logically homogeneous than BBH's (μ = 4.71). The field was caught in a difficulty-diversity tradeoff: benchmarks could be difficult (BBEH) or diverse (BBH's original task variety), but seemed unable to be both simultaneously because the dominant construction paradigm—template-based expansion from a small set of task templates—inherently sacrifices logical diversity for instance count. GENERAL365 demonstrates that this tradeoff is not fundamental but methodological: human-curated seed problems with K-12 knowledge constraints can achieve both high difficulty (62.8% ceiling, Figure 8) and high logical diversity (μ = 2.16 similarity, Figure 7), but at the cost of a more expensive, labor-intensive construction process that produces fewer total instances.

This demonstration redirects research attention in several specific ways:

It makes template-based benchmark construction less attractive as a methodology. The paper's diversity analysis provides a replicable protocol for auditing benchmarks for logical redundancy, and its application to BBH and BBEH demonstrates that template-based expansion—which has been the dominant paradigm because it produces large datasets cheaply—yields benchmarks where the vast majority of instances test the same reasoning skeleton wearing different surface clothing. A benchmark of 6,511 instances that tests only a handful of distinct logical patterns (BBH, with 77.8% of nearest-neighbor pairs receiving maximum similarity scores) provides less genuine reasoning coverage than a benchmark of 365 human-curated seeds representing independent reasoning structures. Future benchmark designers now have both a conceptual argument and a quantitative methodology for justifying investment in human curation over template expansion, even at the cost of smaller instance counts.

It makes verifier and reward model research more urgent for general reasoning. The paper does not itself develop or test verifiers, but its finding that reasoning models (which internally implement some form of verification or search) substantially outperform chat models on categories like Implicit Information Reasoning (68.8% vs. 29.6% for Gemini-3-Pro vs. DeepSeek-V3.2-Chat, Table 1) and Recursive & Backtracking (66.8% vs. 50.3%) suggests that test-time search and self-verification are effective specifically for the reasoning patterns where GENERAL365 is most challenging. This connects to the test-time compute scaling literature that the prior example paper analyzed in depth—if GENERAL365's Semantic Interference and Optimal Strategy categories represent reasoning challenges where current architectures hit fundamental limits, then improving verifier robustness for exactly these types of reasoning becomes a high-priority research target.

It shifts the burden of proof for benchmark claims about "general reasoning." Before GENERAL365, a benchmark could claim to test general reasoning by selecting tasks that don't obviously require specialized knowledge (logic puzzles, word problems, constraint satisfaction). The paper's diversity validation methodology—embedding space visualization plus pairwise logical similarity scoring—provides a concrete, replicable standard for auditing such claims. A benchmark that asserts reasoning diversity can now be tested against that assertion using the same protocol: project its instances into embedding space, retrieve nearest-neighbor pairs, and have an LLM evaluator score those pairs for logical similarity. This makes "general reasoning" an empirically testable property of a benchmark rather than a design intention, raising the methodological bar for the entire subfield of reasoning evaluation.

It identifies specific capability boundaries that are actionable for model development. The finding that Semantic Interference and Optimal Strategy scores are consistently ~10 percentage points below models' overall averages across all architectures (Table 1, Figure 5) provides model developers with concrete targets. Improving Semantic Interference performance requires mechanisms for suppressing pretrained priors when task rules contradict them—a capability that may require architectural innovations beyond simply scaling model size or training data, since larger models trained on more data may actually have stronger pretrained priors that are harder to override. Improving Optimal Strategy performance requires mechanisms for evaluating multiple candidate action sequences and selecting the utility-maximizing one—a capability that connects to planning, search, and reinforcement learning in ways that current autoregressive architectures may not naturally support. These are not vague "improve reasoning" goals but specific engineering challenges with measurable success criteria: close the ~10 percentage point gap between these categories and models' overall performance.

However, the paper also makes certain research directions less attractive by demonstrating diminishing returns:

Further optimization on BBH or BBEH as a measure of reasoning progress is unlikely to yield meaningful signals, given that BBH is essentially saturated (89–94% for current models) and BBEH's logical homogeneity means improvements may reflect better template matching rather than genuine generalization.
Scaling model size alone, without architectural innovations targeting the specific failure modes GENERAL365 identifies, seems unlikely to close the gap—the fact that Gemini-3-Pro, the strongest model tested, still achieves only 62.8% suggests that current scaling paradigms may be approaching diminishing returns for the types of reasoning GENERAL365 tests.
Template-based benchmark construction as a cost-saving strategy now carries a demonstrated validity cost that must be weighed against the savings, given the paper's evidence that template expansion produces benchmarks with μ ≈ 4.7–4.8 logical similarity (Figure 7).

The paper's most enduring conceptual contribution may be the operationalization of reasoning diversity as a measurable benchmark property. The pairwise similarity scoring methodology is likely to outlast GENERAL365 itself—as models eventually solve the current benchmark (which the paper explicitly designs headroom for, with its 62.8% ceiling leaving substantial room for progress), the methodology for auditing logical redundancy in benchmarks will remain relevant for constructing their successors. This is a methodological contribution that transcends the specific dataset, analogous to how the BLEU score transcended any particular machine translation system by providing a general evaluation protocol.

Follow-Up Research This Work Enables

Direct quantification of the reasoning-knowledge confound by evaluating the same models on GENERAL365 and domain benchmarks. The paper's foundational claim—that domain-specific benchmark scores conflate reasoning with knowledge retrieval—is argued from cross-paper comparisons (citing prior work's results on AIME, IMO, IPhO) rather than demonstrated within a controlled experimental design. A natural follow-up would evaluate all 26 models from the GENERAL365 leaderboard on one or two representative domain benchmarks (e.g., AIME 2025 for mathematics, GPQA for graduate-level science) and analyze the correlation structure. The key question: is the correlation between GENERAL365 and domain-specific benchmarks weaker than the correlation between GENERAL365 and other general reasoning benchmarks (BBH, BBEH)? If GENERAL365 measures reasoning decoupled from knowledge, it should show lower cross-benchmark correlation with knowledge-heavy benchmarks than with knowledge-light ones, after controlling for difficulty. A strong follow-up would also include human expert solve rates on both GENERAL365 and the domain benchmarks to establish whether the performance gap reflects genuine difficulty differences or domain-specific knowledge requirements—if human experts (who possess the domain knowledge) perform similarly on both, but LLMs show a large gap, that would directly evidence the knowledge confound.

Training verifiers or process reward models specifically for Semantic Interference and Optimal Strategy reasoning patterns. GENERAL365 identifies these two categories as primary bottlenecks where top models score ~10 percentage points below their overall averages (Table 1: Gemini-3-Pro at 55.7% Semantic Interference, 50.8% Optimal Strategy vs. 62.8% overall). The test-time compute scaling literature has shown that verifier-guided search can substantially improve LLM reasoning on problems within a base model's capability range—but that literature has predominantly been tested on mathematical reasoning benchmarks (MATH, GSM8K) where the reasoning patterns are well-represented in training data. A natural experiment would apply PRM-guided beam search or best-of-N weighted selection specifically to GENERAL365's Semantic Interference and Optimal Strategy subsets, training the verifier on GENERAL365-style problems. The hypothesis: if Semantic Interference problems are hard because models default to commonsense priors rather than because they fundamentally cannot execute the required logic, then a verifier that penalizes commonsense-consistent-but-rule-violating reasoning steps should improve accuracy. The expected outcome: verifier-guided search should help more on Semantic Interference (where the model's error is choosing the wrong reasoning path from among available options) than on Optimal Strategy (where the model may genuinely lack the planning capability to generate the optimal path in the first place). A negative result—verifier search not helping on either category—would suggest that these bottlenecks reflect fundamental capability gaps rather than selection errors, redirecting research toward architectural innovations rather than test-time strategies.

Fine-grained item-level analysis to determine whether per-category score differences reflect genuine capability gaps or problem selection artifacts. The paper's diagnostic claim that Semantic Interference and Optimal Strategy are primary bottlenecks (Section 3.2, Table 1) is confounded by unequal category sizes (Figure 3a), multi-label problem structure (Figure 3b: ~70% of problems carry 2+ labels), and uncalibrated per-category difficulty. A follow-up analysis would decompose per-category scores into components attributable to the category label versus attributable to problem-specific difficulty. Concrete approaches: (1) For problems with a single category label (the ~30% minority), compute per-category scores and compare to the multi-label-inclusive scores—if the single-label Semantic Interference score is similarly low, the bottleneck interpretation is strengthened. (2) Apply item response theory (IRT) to estimate latent difficulty and discrimination parameters for each problem, then test whether category labels explain variance in difficulty beyond what problem-level features (length, number of constraints, entity count) explain. (3) Have human expert raters solve a random subset of GENERAL365 problems and report per-category solve rates—if humans also find Semantic Interference and Optimal Strategy substantially harder than other categories, the bottleneck reflects genuine cognitive difficulty rather than LLM-specific deficits. Without this item-level analysis, the diagnostic value of per-category scores remains suggestive rather than established.

Cross-architectural analysis of reasoning traces to understand why Gemini-3-Pro achieves superior reasoning efficiency. Figure 4 shows Gemini-3-Pro achieving top accuracy (62.8%) with ~14k average output tokens while competitors like GLM-5-Thinking require ~25-30k tokens for lower accuracy (59.9%). The paper attributes this to "superior reasoning efficiency" and "reasoning density" but does not analyze why—is Gemini-3-Pro producing shorter but equally effective reasoning chains, or is it producing fundamentally different types of reasoning (e.g., more structured constraint propagation rather than verbose exploration)? A follow-up would collect the full Chain-of-Thought outputs from multiple models on a shared subset of GENERAL365 problems and conduct qualitative coding of reasoning strategies: frequency of backtracking, use of intermediate abstractions, tendency to verify sub-conclusions, length of individual reasoning steps, and presence of irrelevant exploration. The hypothesis: efficient models achieve their efficiency through better reasoning structure (systematic constraint propagation, early pruning of dead ends) rather than merely shorter outputs. If so, reasoning trace analysis could identify specific reasoning strategies that correlate with both accuracy and efficiency, providing training targets for improving other models. A strong negative result—efficient models simply being more concise without structural differences in reasoning—would suggest that the efficiency advantage is an artifact of training objectives (e.g., reinforcement learning with a brevity penalty) rather than superior reasoning capability, and would weaken the paper's claim that efficiency reflects reasoning quality.

Longitudinal study tracking whether GENERAL365's difficulty persists as models improve, or whether the benchmark's challenge is specific to current architectural limitations. GENERAL365 was constructed by filtering out problems that were "easily solvable" by top models at construction time. This creates a potential circularity: the benchmark is hard for the generation of models that existed when it was built, but may not remain hard for future generations with different architectural properties. A planned follow-up would re-evaluate GENERAL365 on each new major model release (e.g., future Gemini, GPT, and open-source models) and track not just aggregate accuracy but per-category trajectories. The key diagnostic: if Semantic Interference scores improve faster than other categories as models advance, that suggests current Semantic Interference difficulty reflects a fixable architectural limitation (e.g., insufficient training on counterfactual reasoning) rather than a fundamental reasoning challenge. If Optimal Strategy scores remain stubbornly low even as other categories approach saturation, that suggests strategic planning under constraints is a genuinely hard capability that may require paradigm-level innovations (e.g., integration with explicit planning modules, search-based architectures) rather than incremental scaling. This longitudinal study would transform GENERAL365 from a static benchmark into a capability-tracking instrument, providing the field with a "reasoning progress report" that is more informative than tracking scores on already-saturated benchmarks.

Development of lightweight difficulty estimators for routing problems to appropriate models or inference strategies. The prior example paper on test-time compute scaling demonstrated that prompt difficulty estimation enables 4× efficiency gains by adaptively allocating inference compute. GENERAL365's per-category difficulty variation—and its wide dynamic range (26.7% to 62.8% across models, Figure 1)—makes it a natural testbed for extending that paradigm to general reasoning. A concrete follow-up: train a lightweight classifier on GENERAL365's public subset that predicts, from the problem text alone, whether a given model is likely to succeed or fail on that problem (binary classification) or which category of challenge it represents (multi-class). The classifier could use the problem embeddings from text-embedding-ada-002 (already demonstrated to capture semantic structure, Figure 6) as input features, with model correctness as the training label. Success would be measured by whether routing problems to the most capable model for that problem type (based on per-category scores from Table 1) improves aggregate accuracy-per-dollar compared to using a single model for all problems. This connects GENERAL365 to the broader test-time compute allocation literature while addressing a practical deployment question: given a heterogeneous set of reasoning problems arriving at inference time, can we cheaply estimate which problems need the expensive reasoning model and which can be handled by a cheaper chat model?

Practical Applications and Downstream Use Cases

Model selection for reasoning-intensive enterprise deployments. Organizations deploying LLMs for workflows that involve constraint satisfaction, logical deduction, or strategic planning—legal contract analysis, supply chain optimization, competitive strategy development, medical differential diagnosis—currently select models based largely on domain benchmark scores (AIME, GPQA, SWE-bench) that may not predict performance on their actual reasoning tasks. GENERAL365 provides a complementary selection criterion that specifically measures reasoning under conditions where domain knowledge is not the differentiating factor. The leaderboard (Figure 1) and per-category breakdowns (Table 1) enable fine-grained model selection: a legal tech company whose reasoning tasks predominantly involve interpreting novel regulatory constraints (Semantic Interference-like reasoning) should weigh models' Semantic Interference scores more heavily than their Complex Constraints scores, while a logistics company optimizing delivery routes under time windows and capacity limits (Optimal Strategy-like reasoning) should prioritize that category. The 25+ percentage point spread between top and bottom models on GENERAL365 means the choice of model has substantial practical consequences—a deployment using DeepSeek-V3.2-Chat (37.6%) rather than Gemini-3-Pro (62.8%) would fail on roughly 25% more reasoning problems, a difference that would be immediately visible in production metrics. The efficiency dimension (Figure 4) adds a cost consideration: Gemini-3-Pro achieves its top accuracy with approximately half the output tokens of similarly-accurate competitors, directly translating to lower per-query inference costs and latency for high-volume deployments.

Diagnostic auditing of model reasoning weaknesses before domain-specific fine-tuning. Practitioners who fine-tune LLMs for domain-specific reasoning tasks (e.g., training a model on medical case studies, legal precedents, or engineering design problems) currently lack tools for diagnosing why their fine-tuned model fails on held-out cases—is the failure due to insufficient domain knowledge, or to underlying reasoning limitations that fine-tuning cannot fix? GENERAL365's per-category diagnostic framework provides a pre-fine-tuning audit: evaluating a base model on GENERAL365 before fine-tuning reveals its reasoning capability profile, identifying categories where improvement is unlikely regardless of domain-specific training data. For example, a model that scores 40% on Optimal Strategy before fine-tuning is unlikely to become a strong strategic planner through fine-tuning on domain examples alone—that capability gap may require architectural improvements, test-time search strategies, or hybrid systems that pair the LLM with an explicit planner. Conversely, strong pre-fine-tuning scores on Complex Constraints and Recursive & Backtracking suggest the model has the underlying reasoning machinery to handle complex domain-specific problems once the knowledge is provided. This diagnostic use transforms model selection from "pick the highest-scoring model on our domain benchmark" to "pick the model whose reasoning profile matches our task's cognitive demands, then fine-tune for domain knowledge."

Benchmarking the reasoning impact of test-time compute scaling strategies. The test-time compute scaling literature has demonstrated that search against verifiers and iterative revision can substantially improve LLM performance on mathematical reasoning, but has not yet established whether these strategies transfer to the types of general reasoning GENERAL365 tests—where the challenge is suppressing commonsense priors (Semantic Interference) or evaluating multiple viable strategies (Optimal Strategy) rather than executing complex mathematical derivations. GENERAL365 provides a ready-made evaluation suite for testing whether existing test-time strategies help on these qualitatively different reasoning challenges. A practitioner could take a base model (e.g., Qwen3-Max-Instruct at 48.5%, Table 1), apply best-of-N weighted selection with a GENERAL365-trained verifier, and measure per-category improvement. If Semantic Interference improves substantially under verifier-guided search (suggesting the model can generate correct reasoning paths but fails to select them), while Optimal Strategy does not improve (suggesting the model genuinely cannot generate optimal plans), that differential result would inform whether to invest in verifier training versus architectural improvements for different reasoning use cases. This is a directly actionable research workflow that GENERAL365's category structure and difficulty headroom make newly feasible—existing general reasoning benchmarks are either too easy (BBH: no headroom for measuring improvement) or too logically homogeneous (BBEH: improvements may reflect template matching rather than genuine capability gains).

When to Prefer This Benchmark

The paper does not present GENERAL365 as a method to be preferred over alternatives in a deployment context—it is a benchmark for evaluation, and its "competitors" are other benchmarks (BBH, BBEH, KOR-Bench, ARC-AGI), not alternative approaches to solving reasoning problems. However, the paper implicitly argues for when GENERAL365 should be preferred over existing benchmarks for evaluating LLM reasoning capabilities, and this tradeoff can be made explicit:

Prefer GENERAL365 over BBH or BBEH for evaluating LLM reasoning when:

The goal is to discriminate between frontier models (BBH is saturated at 89–94% for top models, providing only ~5 percentage points of dynamic range, while GENERAL365 provides ~36 points; Figure 8).
The concern is genuine reasoning generalization rather than template matching (BBH and BBEH exhibit μ = 4.71–4.80 logical similarity, meaning most instances share identical reasoning skeletons; GENERAL365 achieves μ = 2.16, indicating independent reasoning patterns; Figure 7).
Diagnostic information about specific reasoning weaknesses is needed (BBH and BBEH provide task-level scores but not a structured capability taxonomy; GENERAL365's eight-category breakdown enables targeted model improvement; Table 1).
The deployment context involves problems where rules may conflict with commonsense defaults, or where optimal strategy selection matters (these are GENERAL365's hardest categories, and models' performance on them is not predicted by their aggregate BBH/BBeH scores).

Prefer domain-specific benchmarks (AIME, GPQA, SWE-bench) over GENERAL365 when:

The deployment context involves tasks where domain expertise is the primary requirement, and reasoning without that expertise is irrelevant (e.g., deploying an LLM to solve competition mathematics problems, where the model's mathematical knowledge is part of the capability being evaluated).
The goal is to track progress at the frontier of expert-level domain reasoning, where the K-12 knowledge constraint would artificially cap the difficulty that can be tested.

Prefer BBH or BBEH as a complement to GENERAL365 when:

Instance count per evaluation matters for statistical power (BBH provides 6,511 instances vs. GENERAL365's 1,095, enabling finer-grained measurement of small performance differences at the cost of logical diversity).
Reproducibility of results across research groups is paramount, and the full private portion of GENERAL365 is not accessible (only the 720-instance public subset is available, while BBH and BBEH are fully public).