ArXiv: 2512.16969
🎯 Pitch
Current LLMs show a surprising fragmentation: while they can execute dry experiment code smoothly over 90% of the time, their numerical accuracy collapses to below 37% (PassAll@5). This paper introduces SGI-Bench, a 1000+ sample benchmark spanning 10 disciplines and grounded in the four-quadrant Practical Inquiry Model, revealing that no model today approaches coherent scientific workflow intelligence.
1. Executive Summary
This paper proposes an operational definition of Scientific General Intelligence (SGI)—the capacity to autonomously navigate the full iterative cycle of scientific inquiry—grounded in the Practical Inquiry Model's four quadrants (Deliberation, Conception, Action, Perception) and instantiates it through SGI-Bench, a scientist-aligned benchmark of over 1,000 expert-curated samples spanning ten disciplines. Evaluating state-of-the-art LLMs reveals fragmented cognition: exact-match accuracy in deep research stalls at 10–20% despite step-level alignment, idea generation suffers from uniformly low feasibility (best model scoring only 22.90) despite high novelty, and dry experiments achieve high executability (>90% smooth execution rate) but collapse on numerical correctness (best PassAll@5 of only 36.64%), establishing that current models possess isolated fragments of scientific reasoning that fail to cohere into workflow-faithful intelligence. The paper further introduces Test-Time Reinforcement Learning (TTRL)—a mechanism that optimizes retrieval-augmented novelty rewards at inference without ground-truth labels—improving a base model's novelty score from 49.36 to 62.06, demonstrating that SGI is a dynamic capacity that can evolve through iterative self-improvement rather than a static property.
2. Context and Motivation
The Core Problem: We Cannot Measure What We Mean by "Scientific Intelligence"
The foundational problem this paper addresses is deceptively simple: the field lacks a coherent, operational definition of Scientific General Intelligence (SGI), and therefore cannot measure progress toward it. While terms like "scientific reasoning," "AI scientist," and "scientific discovery" appear frequently in contemporary AI discourse, the paper argues that the concept of SGI remains "frustratingly nebulous, often acting as a moving goalpost that hinders clear evaluation and progress" (Section 1). This is not merely a semantic issue — it has concrete consequences for research direction, resource allocation, and model development.
To see why this ambiguity is damaging, consider the paper's analogy to the AGI debate. Just as some researchers view LLMs as "early signals of AGI" while others dismiss them as "stochastic parrots" (Section 1, citing Bender et al., 2021), similar contradictory judgments plague discussions of AI in science. A model that excels at answering multiple-choice physics questions might be hailed as scientifically capable by one benchmark, while another that evaluates open-ended experimental design finds the same model incapable. Without a shared definition of what constitutes scientific intelligence, progress is measured against incompatible yardsticks, and claims of capability become untethered from empirical evidence.
The Practical Inquiry Model (PIM), adapted from Garrison, Anderson, and Archer (1999, 2001) — originally developed to study critical thinking in distance education — provides the taxonomic grounding for SGI. The model decomposes scientific cognition into four interdependent quadrants: Deliberation (evidence search, synthesis, and critical evaluation), Conception (generation of hypotheses and ideas), Action (implementation of experiments or simulations), and Perception (interpretation of empirical results). An AI exhibiting genuine SGI must demonstrate robust capabilities across this entire spectrum, not merely excel at isolated tasks within one quadrant.
The deep significance of the PIM framework is that it shifts the evaluation paradigm from what models know to how they think, plan, and adapt in pursuit of new knowledge (Section 2). A system that scores perfectly on multiple-choice knowledge tests but cannot design an experiment to test a novel hypothesis does not, under this definition, qualify as scientifically intelligent. This echoes the broader AGI discourse but anchors it in a specific, theoretically-motivated taxonomy that enables empirical operationalization.
Why This Problem Matters: The Consequences of Fragmented Evaluation
The importance of solving this measurement problem extends in several directions, each with substantial real-world stakes:
First, scientific discovery is arguably the most complex and structured of human cognitive activities, and the paper explicitly positions SGI as "a critical leap toward AGI, serving as a definitive testbed for advanced reasoning, planning, and knowledge creation capabilities" (Section 1). If we cannot systematically evaluate scientific reasoning, we cannot meaningfully assess progress toward general intelligence. The scientific process — with its demands for literature synthesis, hypothesis formation, experimental design, and evidence-based interpretation — constitutes a uniquely demanding integration of skills that subsumes many narrower benchmarks (mathematical reasoning, reading comprehension, code generation) within a coherent workflow.
Second, the resource allocation question is urgent. Organizations developing and deploying scientific AI systems — pharmaceutical companies, materials science labs, climate research institutes — face decisions about where to invest: better pretraining, more specialized data, tool integration, or agentic orchestration. Without a benchmark that measures integrated scientific workflow capability, these decisions are made blindly. The paper notes that existing benchmarks frame "inquiry as a static, closed-domain question-answering task" (Section 1), which bears little resemblance to how scientists actually work. A model that appears promising on such benchmarks may fail catastrophically when asked to perform the full literature-review-to-experimental-reasoning pipeline that real scientific problems demand.
Third, the paper's findings directly challenge assumptions about scale and access. The aggregate results (Table 5) reveal that the best closed-source system achieves an SGI-Score of approximately 33.83 out of 100, with leading open-source models only marginally behind (e.g., Claude-Sonnet-4.5 at 32.16 vs. Qwen3-Max at 31.97). The SGI-Score is computed as the average across all five core task categories, each represented by its strictest metric: Exact Match for Deep Research, the average of four subjective-plus-objective dimensions for Idea Generation, PassAll@5 for Dry Experiment, the average of Sequence Similarity and Parameter Accuracy for Wet Experiment, and Multi-Choice Accuracy for Experimental Reasoning. This near-parity between open and closed models indicates that "scale and access alone do not translate into robust scientific cognition" (Section 4.2). This finding has significant implications for resource allocation and for the debate about whether proprietary models hold an insurmountable advantage in AI-for-science applications.
Prior Approaches and Their Systematic Shortcomings
The paper catalogs an extensive landscape of existing scientific AI benchmarks (Section 7) and identifies a consistent pattern of fragmentation that falls short of measuring SGI as defined by the PIM framework. The critique can be organized along two axes:
Fragmentation by Scientific Discipline
Many benchmarks drill deeply into single fields but provide no cross-domain assessment:
- In physics: PhyBench (Qiu et al., 2025) tests multi-step reasoning, while PHYX (Shen et al., 2025) assesses physical reasoning and visual understanding, and PHYSICS (Feng et al., 2025) uses university-level problems. PhysUniBench (Wang et al., 2025) adds multimodal physics diagram interpretation.
- In chemistry: ChemBench (Mirza et al., 2024) provides domain-specific evaluation, ChemMLLM (Tan et al., 2025) extends to multimodal tasks, and specialized benchmarks like ChemSafetyBench (Zhao et al., 2024) and SpectrumWorld (Yang et al., 2025) target safety and spectroscopy.
- In life sciences: Benchmarks span molecular-level tasks (DeepSEA, Kathail et al., 2025; GenomicsLong-Range, Anonymous, 2024) to healthcare applications (BioASQ, Krithara et al., 2023; VQA-RAD, Lau et al., 2018) and neuroscience (BrainBench, Luo et al., 2024), with agricultural extensions (SeedBench, Ying et al., 2025).
- In earth sciences: OmniEarth-Bench (Wang et al., 2025) covers cross-domain tasks, EarthSE (Xu et al., 2025) builds multi-level evaluation, and MSEarth (Zhao et al., 2025) uses graduate-level scientific publications.
- In astronomy: AstroLLaMA and AstroMLab (Pan et al., 2024), along with specialized tools.
- In materials science: MoleculeNet (Wu et al., 2018) for molecular machine learning.
- In ocean and climate science: OceanBench (El Aouni et al., 2025) and ClimaQA (Manivannan et al., 2025).
The paper's criticism is not that these benchmarks are individually flawed — many are rigorous within their domains — but that they collectively "present a fragmented view of scientific intelligence" because "disciplinary scope remains narrow" and "challenges seldom reach expert-level reasoning" (Section 1). A model might excel on ChemBench but fail on BrainBench, and neither benchmark provides information about whether the model can connect chemical insights to neuroscientific hypotheses — precisely the kind of cross-domain synthesis that characterizes real scientific breakthroughs.
Fragmentation by Scientific Task Stage
A second category of benchmarks targets cross-disciplinary capabilities but captures only isolated stages of the scientific workflow:
For knowledge processing and analysis (mapping roughly to the Deliberation quadrant): SciAssess (Cai et al., 2024) decomposes literature analysis into memory, understanding, and reasoning layers; SFE (Zhou et al., 2025) probes multimodal performance on raw scientific data; SciReasoner (Wang et al., 2025) targets alignment of natural language with heterogeneous scientific representations.
For deep research and literature synthesis (Deliberation): DeepResearch Bench (Du et al., 2025) measures report quality and citation grounding; Manalyzer (Xu et al., 2025) focuses on hallucination mitigation in automated meta-analysis; Scientist-Bench (Tang et al., 2025) covers the workflow from review to paper generation; SciArena (Zhao et al., 2025) collects pairwise comparison preferences from domain researchers; DeepResearch Arena (Wan et al., 2025) evaluates multi-stage research orchestration through seminar-grounded tasks; AAAR-1.0 (Lou et al., 2025) evaluates AI-assisted research tool capabilities.
For idea generation (Conception): MOOSE-Chem2 (Yang et al., 2025) evaluates hypotheses against reference answers using multiple judges in a win/tie/lose framework; AI Idea Bench 2025 (Qiu et al., 2025) assesses novelty of agent-generated ideas using datasets derived from top-tier conference papers.
For planning and execution (Action): ToolBench (Qin et al., 2023) and ToolUniverse (Gao et al., 2025) explore API usage and standardization; SciCode (Tian et al., 2024) and ScienceAgentBench (Chen et al., 2025) assess scientific code generation within realistic workflows; MLE-bench (Chan et al., 2025) and TaskBench (Shen et al., 2024) evaluate general planning via Kaggle competitions and task decomposition.
For embodied scientific discovery: DISCOVERYWORLD (Jansen et al., 2024) provides a virtual environment for complete novel discovery cycles, but the paper notes it "focuses on a gamified simulation environment" whose "task scenarios and evaluation dimensions cannot fully reflect the complexity and high-level cognitive needs of real scientific research workflows" (Section 7.2). LLM-SRBench (Shojaee et al., 2025) evaluates scientific equation discovery capability but with "a relatively simple task and process" (Section 7.2).
The paper's central critique — which justifies the creation of SGI-Bench — is that "existing process-oriented benchmarks typically address only partial dimensions—such as knowledge understanding, data perception, or code generation—lacking a fine-grained, systematic evaluation of the entire scientific discovery lifecycle" (Section 7.2). Even ambitious benchmarks like ATLAS (Liu et al., 2025), which expanded cross-disciplinary coverage with high-difficulty standards, focus on "distinguishing frontier models through complex scientific reasoning and logical application tasks rather than the entire process of scientific discovery" (Section 7.1).
The Missing Link: Workflow-Faithful Evaluation
The paper identifies a deeper conceptual gap that goes beyond the coverage critique. Existing benchmarks, even when cross-disciplinary, frame scientific inquiry as a series of disconnected sub-tasks — answer this question, generate that hypothesis, complete this code — without evaluating whether a model can sustain coherent reasoning across the transitions between sub-tasks. The PIM framework reveals that genuine scientific cognition requires not just competence in each quadrant individually, but the ability to close the loop: literature review informs hypothesis generation, which specifies experimental design, whose results feed back into revised understanding.
The paper's position is that this loop-closing capacity has never been systematically measured. A model that performs well on Deep Research Bench (literature synthesis) and SciCode (code generation) separately might still fail when asked to read a paper, identify a knowledge gap, propose an experiment to fill it, write the computational analysis code, and interpret the results — precisely the integrated workflow that SGI-Bench's four-task architecture (Scientific Deep Research → Idea Generation → Dry/Wet Experiment → Experimental Reasoning) is designed to evaluate.
How This Paper Positions Itself
The paper explicitly frames its contribution not as incremental improvement over existing benchmarks, but as an operational instantiation of a theoretical framework that has been missing from the field. Several strategic positioning choices are evident:
SGI-Bench is not "just another benchmark." The paper asserts that SGI-Bench "functions as an operational instantiation of the SGI framework" (Section 1). Its goal is not to replace ChemBench or SciCode but to provide the missing integrative evaluation that sits above them — measuring whether a model can string together the isolated competencies that existing benchmarks measure into a coherent scientific workflow.
The four task categories are deliberately aligned with the PIM quadrants, not chosen for convenience: Scientific Deep Research operationalizes Deliberation, Idea Generation operationalizes Conception, Dry/Wet Experiment operationalizes Action, and Experimental Reasoning operationalizes Perception. This mapping provides theoretical grounding for what might otherwise appear as an arbitrary selection of tasks.
Scientist-alignment is a core design principle, not a buzzword. The paper describes a multi-stage construction pipeline involving over 100 Master's and PhD holders, multi-round expert review, and a scientist-in-the-loop approach (Section 2.3). The raw materials — drawn from the 125 Big Questions published in Science and from leading journals (Nature, Science, Cell) — are intended to ensure that benchmark tasks "authentically mirror real scientific workflows" (Section 2). The difficulty filtering step, in which questions solvable by more than half of six high-performance models (GPT-5, Gemini-2.5-Pro, DeepSeek-R1, Kimi-k2, etc.) are removed, aims to maintain expert-level challenge.
The benchmark is designed to be extensible and customizable, not a one-time static dataset. The SGIEvalAgent framework (Section 3) allows researchers to define custom metrics, filter questions by discipline or task type, and generate evaluation reports aligned with specific research questions — positioning SGI-Bench as an ongoing evaluation platform rather than a fixed leaderboard.
TTRL is presented as an initial probe into SGI as a dynamic capacity, not as a fully developed solution to the benchmark's challenges. The paper is careful to note that TTRL, as implemented, optimizes only for novelty — leaving feasibility, rigor, and experimental constraints as future work (Section 6.2). This positioning is strategic: it demonstrates that SGI can improve through test-time learning (validating the dynamic-capacity framing) while acknowledging that the current implementation is only a first step.
The paper also draws an implicit parallel to the development of AGI benchmarks: just as the field moved from narrow task evaluation (e.g., GLUE, SuperGLUE) to more integrative assessments (e.g., BIG-bench, MMLU, HLE) as models improved, the paper argues that scientific AI evaluation must similarly evolve from discipline-specific, stage-isolated benchmarks to workflow-faithful, multi-stage assessment. SGI-Bench is positioned to fill this evolutionary niche for scientific reasoning, providing a framework that can accommodate future model improvements without becoming obsolete — since the tasks are drawn from open scientific questions and are designed to be continuously refreshed through expert curation.
3. Technical Approach
3.1 Reader Orientation
This paper constructs SGI-Bench, a benchmark and evaluation framework that measures large language models' ability to perform complete scientific workflows — from literature review through hypothesis generation, experimental execution, and results interpretation — rather than testing isolated scientific knowledge. The system solves the problem of fragmented scientific AI evaluation by operationalizing a theoretical model of scientific cognition (the Practical Inquiry Model) into four concrete task categories, each with multi-dimensional scientist-aligned metrics, and wrapping them in an agentic evaluation framework that enables customizable, transparent, and reproducible assessment.
3.2 Big-Picture Architecture (Diagram in Words)
The SGI-Bench system comprises five major components:
1. The Practical Inquiry Model (PIM) Framework — the theoretical backbone that defines SGI as competence across four interdependent cognitive quadrants: Deliberation, Conception, Action, and Perception. This framework determines what gets measured and why.
2. SGI-Bench Task Construction Pipeline — a multi-stage, scientist-in-the-loop process that transforms raw scientific materials (from Science's 125 Big Questions and leading journals) into over 1,000 expert-curated benchmark samples across 10 disciplines. This pipeline ensures that tasks authentically mirror real scientific workflows rather than artificial test problems.
3. Four Task Categories with Multi-Dimensional Metrics — each task category is specifically designed to instantiate one PIM quadrant, with evaluation metrics that capture both final-answer correctness and intermediate reasoning quality:
- Scientific Deep Research (Deliberation): literature-inquiry tasks requiring multi-step retrieval, quantitative reasoning, and evidence synthesis, evaluated via Exact Match and Step-Level Accuracy.
- Idea Generation (Conception): methodology-design tasks requiring structured research proposals with implementation steps, evaluated via hybrid subjective-objective metrics for Effectiveness, Novelty, Detailedness, and Feasibility.
- Dry/Wet Experiment (Action): Dry experiments are code-completion tasks with masked scientific functions, evaluated via unit test passing rates, execution time, and smooth execution rate. Wet experiments are protocol-planning tasks requiring atomic action sequencing and parameter specification, evaluated via Sequence Similarity and Parameter Accuracy.
- Experimental Reasoning (Perception): multimodal multiple-choice questions requiring interpretation of scientific images (process diagrams, observations, simulations, experiments, visualizations), evaluated via Multi-Choice Accuracy and Reasoning Validity.
4. SGIEvalAgent Evaluation Framework — an agentic system that orchestrates the evaluation pipeline through four stages: Question Selection (filtering benchmark questions by discipline, task, or user intent), Metric Customization (generating novel evaluation metrics from user queries), Prediction & Evaluation (running model inference and applying metrics with tool-augmented verification), and Report Generation (compiling results with contextualized analysis).
5. Test-Time Reinforcement Learning (TTRL) Module — an experimental component that optimizes a language model's idea generation capability at inference time without ground-truth labels, using retrieval-augmented novelty rewards and the GRPO algorithm. This module probes whether SGI can emerge dynamically through self-improvement rather than being a static property acquired during pretraining.
Information flows as follows: raw scientific materials → expert annotation and quality control → benchmark questions with ground-truth answers → model inference on selected questions → multi-dimensional metric computation by evaluation agents → aggregated performance reports and SGI-Score.
3.3 Roadmap for the Deep Dive
- First, the PIM-grounded operational definition of SGI — because all subsequent task design, metric selection, and evaluation philosophy derives from this theoretical commitment about what scientific intelligence is.
- Second, the scientist-aligned data construction pipeline — because benchmark quality and validity depend on how questions are sourced, annotated, and filtered, and the multi-stage expert-in-the-loop process is a core methodological contribution.
- Third, the four task categories with their input/output specifications, evaluation metrics (including all mathematical definitions), and design rationales — because these constitute the operational core of SGI-Bench, and understanding each task's formulation is essential for interpreting the evaluation results.
- Fourth, the SGIEvalAgent evaluation framework — because it introduces the agentic, customizable evaluation paradigm that distinguishes SGI-Bench from static benchmarks and enables the user-customized metrics demonstrated in the analysis.
- Fifth, the Test-Time Reinforcement Learning (TTRL) methodology — because it represents the paper's experimental probe into dynamic SGI and introduces a novel reward mechanism (retrieval-based novelty) applied during inference rather than training.
3.4 Detailed, Sentence-Based Technical Breakdown
This is primarily a benchmark construction and evaluation paper whose core idea is that Scientific General Intelligence must be measured across the complete cognitive cycle of Deliberation → Conception → Action → Perception using scientist-aligned tasks and multi-dimensional metrics, and that current LLMs exhibit only fragmented, non-integrated capabilities across this cycle.
The PIM-Grounded Operational Definition of SGI
The paper anchors its definition of Scientific General Intelligence in the Practical Inquiry Model (PIM), originally developed by Garrison, Anderson, and Archer (1999, 2001) to study critical thinking in text-based educational environments. The adaptation to scientific cognition is the paper's foundational theoretical move.
The PIM decomposes the scientific process into four interdependent cognitive activities, organized in a cyclic structure (Figure 1):
Deliberation is defined as "the search, synthesis, and critical evaluation of knowledge" (Section 2). This corresponds to what scientists do during literature review: they retrieve existing findings, assess their reliability, integrate across sources, identify contradictions, and determine what is known versus unknown. The paper operationalizes this through the Scientific Deep Research task.
Conception is defined as "the generation of ideas" (Section 2). This captures the creative, synthetic aspect of scientific cognition: forming novel hypotheses, designing methodological approaches, and proposing experiments that could resolve identified knowledge gaps. The paper operationalizes this through the Idea Generation task.
Action is defined as "the practical implementation via experiments" (Section 2). This represents the execution phase where theoretical designs meet physical or computational constraints: running simulations, conducting laboratory procedures, collecting data. The paper operationalizes this through the Dry/Wet Experiment task.
Perception is defined as "the awareness and interpretation of results" (Section 2). This captures the analytical phase where raw observations are transformed into scientific conclusions: identifying patterns in data, comparing conditions, inferring causal mechanisms, and updating beliefs in light of evidence. The paper operationalizes this through the Experimental Reasoning task.
The critical property of the PIM is that these quadrants form a cycle, not a linear sequence. Interpretation of results (Perception) feeds back into revised understanding (Deliberation), which generates new hypotheses (Conception), which motivate new experiments (Action), whose outcomes require interpretation (Perception). The paper's operational definition translates this cyclic structure into four criteria: an AI system exhibits SGI if it can "(1) retrieve, synthesize, and critically evaluate knowledge; (2) generate scientifically grounded and novel ideas; (3) plan and execute experimental procedures; (4) interpret empirical outcomes with causal and contextual awareness" (Section 2).
The paper is explicit that all four capabilities are necessary — competence in three out of four quadrants does not constitute SGI. This is the key departure from existing benchmarks, which the paper argues capture at most two quadrants (typically Deliberation via knowledge tests or Action via code generation) without assessing whether a model can string together the complete cycle.
The significance of grounding SGI in an established educational theory rather than an ad-hoc task taxonomy is that it provides falsifiable criteria: if a model excels at literature retrieval but cannot design experiments, it fails the SGI test not because of arbitrary benchmark design, but because it lacks a necessary component of the cognitive cycle that defines scientific intelligence.
Scientist-Aligned Data Construction Pipeline
The benchmark construction process is described in Section 2.3 and illustrated in Figure 2(G). It proceeds through four sequential stages, each designed to enforce scientific validity and maintain alignment with real research practice.
Stage 1: Raw Corpus Collection
The starting point is a set of 75 research directions spanning ten scientific domains (astronomy, chemistry, earth science, energy, information science, life science, materials science, neuroscience, physics, mathematics), curated through "multiple discussions with experts from diverse scientific disciplines" and drawing from both "the 125 important scientific questions published in Science" (Sanders, 2021) and "prominent research directions in various disciplines with significant scientific impact" (Section 2.3). The complete research directions are listed in Appendix A.2 (Table 9) and visualized in Figure 8, showing specific topics such as "Gravitational Wave Detection and Parameter Estimation" (astronomy), "Target Based Drug Design" (chemistry), "Ocean Heat Content" (earth science), "Optimal Power Flow Calculation" (energy), "Code Generation" (information science), "Tumor Neoantigen Discovery" (life science), "Polymer Thermoelectric" (materials science), "Differential Privacy" (mathematics), "Visual Decoding" (neuroscience), and "Computational Condensed Matter Physics" (physics).
For each research direction, domain experts provided raw source materials consisting of "scientific texts and images across the various disciplines" (Section 2.3). The texts cover four categories: knowledge introduction, methodological design, experimental procedures, and data analysis — deliberately spanning the PIM quadrants. The images include experiment figures, data visualizations, and observational images, each with detailed descriptions. Experts also provided seed questions — initial example questions demonstrating the desired format and difficulty — and annotation requirements specifying task-specific constraints for the annotation process.
This stage establishes the scientist-alignment principle: the raw materials come directly from practicing scientists rather than being scraped from general web corpora, ensuring that downstream questions reflect genuine scientific concerns rather than textbook-style knowledge tests.
Stage 2: Question Construction
The paper recruited "over 100 Master's and PhD holders from different disciplines" to construct benchmark questions according to the task definitions specified in Section 2.1. The annotation process follows a specific two-annotator protocol: "one of whom is responsible for generating complex draft questions, and the other is responsible for refining them" (Section 2.3), as illustrated in Figure 2(G) where "Annotator A" produces a draft and "Annotator B" performs refinement.
Several rules enforce scientific validity and traceability:
- Annotators must "reference the original data source and paragraph for each question, ensuring traceability to scientist-provided data" (Section 2.3). Every question can be traced back to a specific piece of source material provided by domain experts.
- Questions must conform to the task-specific input/output specifications defined in Sections 2.1.1 through 2.1.4, ensuring structural consistency across the benchmark.
- Experts continuously reviewed questions during construction: "Each question was immediately submitted to the relevant expert for evaluation, who assessed its scientific value. For instance, a question with an experiment configuration that lacks general applicability would be deemed scientifically invalid. Experts provided feedback to annotators, who then revised the questions accordingly" (Section 2.3).
This iterative expert review loop is the key mechanism for maintaining scientist-alignment during construction — it prevents annotators from drifting toward artificial or textbook-style problems that might be easier to construct but would not reflect genuine scientific reasoning.
Stage 3: Data Cleaning
After all questions were constructed, three layers of cleaning were applied:
Rule-based cleaning used task-specific criteria to filter structurally invalid questions. Examples provided include: "for Scientific Deep Research, steps must be short sentences forming a list, each representing one step; for Wet Experiments, each action must exist in the predefined action pool" (Section 2.3). This layer catches format violations that would make questions unanswerable or unevaluatable.
Model-based cleaning used "large language models to detect and remove questions with semantic errors or potential logical inconsistencies" (Section 2.3). The paper does not specify which models were used for this step or the exact filtering criteria, but the intent is to catch content-level errors that rule-based checks would miss — such as contradictory constraints, impossible experimental conditions, or questions that cannot be answered from the provided information.
Expert quality check had "all questions reviewed by the original data-providing scientists, removing incomplete questions, questions with non-unique answers, or questions whose research direction did not align with the source data" (Section 2.3). For Dry Experiments specifically, "Python environments were used to test all code snippets to ensure executability" — meaning that every code-completion question has been verified to have an executable reference solution.
This three-layer cleaning is designed to progressively filter from structural invalidity (layer 1), to semantic/logical invalidity (layer 2), to scientific invalidity (layer 3), with each layer catching errors that the previous layers cannot detect.
Stage 4: Difficulty Filtering
The final stage ensures that SGI-Bench remains challenging for frontier models. The procedure evaluated each question with "six high-performance models (e.g., GPT-5, Gemini-2.5-Pro, DeepSeek-R1, Kimi-k2) under a setup allowing web search and deep-reasoning modes" and removed "questions that more than half of the models could correctly answer" (Section 2.3). This is a relative difficulty threshold: a question must be unsolvable by at least 4 out of 6 frontier models to remain in the benchmark.
The filtering criterion is strategically chosen. By using multiple models with web search and deep reasoning enabled, the paper ensures that questions solvable through straightforward retrieval or surface-level reasoning are excluded, leaving only those requiring genuine multi-step scientific inference. The threshold of "more than half" being correct (rather than "any model being correct") is a pragmatic choice: it allows questions that are accidentally easy for one model due to training data contamination to still be retained, while filtering questions that are genuinely easy across the frontier of current capability.
The final benchmark composition (Section 2.4) comprises: 318 Scientific Deep Research questions, 315 Idea Generation questions, 271 Dry Experiment questions, 68 Wet Experiment questions, and 291 Experimental Reasoning questions, for a total of 1,263 samples. The discipline distributions (Figure 9) show that Scientific Deep Research, Idea Generation, and Experimental Reasoning share identical discipline coverage, while Dry and Wet Experiments have different distributions — Wet Experiments covering only a subset of disciplines such as Biology and Chemistry, reflecting the domain-specific nature of laboratory protocol planning.
Task Category 1: Scientific Deep Research (Deliberation)
Task Design Philosophy
Scientific Deep Research is designed to instantiate the Deliberation quadrant of the PIM — the literature review and meta-analysis stage where scientists retrieve, synthesize, and critically evaluate existing knowledge. The paper positions this task as a hybrid between "AI's deep research, which often relies on multi-hop searches to gather diverse information across multiple sources" and "the methodology of meta-analysis from the scientific community, which synthesizes existing literature to derive precise, data-driven conclusions" (Section 2.1.1).
A critical scoping decision is that the benchmark focuses on literature-inquiry-centric deep research rather than "report-style reasoning" or open-ended synthesis. The paper provides a specific rationale: "by constraining the task to literature inquiry rather than broader report-generation settings, we ensure greater reproducibility and more reliable evaluation, while still probing a core component of scientific inquiry" (Section 2.1.1). This is a deliberate trade-off between coverage and measurability — open-ended scientific writing would require extensive expert adjudication, which the paper defers to future versions.
Task Input Specification
Each Scientific Deep Research question provides five input components:
- Background (B): "A detailed background of the research topic, including the scientific field and subfields, to avoid ambiguities in terminology" (Section 2.1.1). This establishes the disciplinary context and disambiguates field-specific terms.
- Constraints (C): "Constraints such as experimental settings, scientific assumptions, and data sources that frame the problem appropriately" (Section 2.1.1). These bound the search space and prevent models from providing technically correct but contextually inappropriate answers.
- Data (D): "Any experimental or empirical data directly mentioned in the task, which might be either explicitly provided or inferred" (Section 2.1.1). This can include numerical values, experimental parameters, or observational data embedded in the question text.
- Question (Q): "A specific, focused question that the task aims to address, such as determining a particular quantity or its variation over time" (Section 2.1.1). The questions are designed to have short, unique, and verifiable answers — typically numerical values, specific quantities, or short phrases.
- Response Requirements (R): "Specifications for the answer, including the required units and whether the answer should be an integer or a decimal with a specified number of decimal places" (Section 2.1.1). These requirements enable automated exact-match scoring by constraining answer format.
Task Output Specification
The model is required to produce two outputs:
- Steps (S): "A detailed, step-by-step approach that the system uses to retrieve and process data or perform reasoning" (Section 2.1.1). Each step should be a short sentence representing one discrete reasoning or retrieval operation.
- Answer (A): "A precise numerical or string-based response, such as a specific value or a phrase" (Section 2.1.1).
The task formulation is formally expressed as:
where the model or agent takes the five input components and produces both intermediate reasoning steps and a final answer.
Four Question Types
To capture the diversity of real-world literature inquiries, the paper classifies questions into four types (Table 1):
- Data: "Focused on retrieving or analyzing structured datasets, such as event counts, statistical summaries, or dataset-specific attributes." These questions test the model's ability to locate and extract specific numerical values from scientific literature.
- Properties: "Concerned with identifying or inferring material, molecular, or system properties, often requiring interpretation of experimental results or theoretical knowledge." These questions require the model to determine physical, chemical, or biological properties of systems described in the literature.
- Micro-experiments: "Small-scale controlled experiments, often involving chemical reactions, physical transformations, or laboratory processes under specific conditions." These questions provide simulated reasoning over experimental procedures and outcomes at laboratory scale.
- Macro-experiments: "Large-scale or natural experiments, such as astronomical events, climate observations, or geophysical phenomena." These questions extend literature review to global or long-term observations.
The type distribution (Figure 9d) shows that Data and Properties questions are most common, followed by Micro- and Macro-experiments. The paper's later analysis reveals that Data and Properties questions are substantially harder for models (Section 4.3, Figure 14), suggesting that unstructured retrieval and property inference pose greater challenges than reasoning over structured experimental protocols.
Evaluation Metric 1: Exact Match (EM)
Exact Match is the primary, strictest metric for Deep Research evaluation. The definition is straightforward:
\text{EM} = \mathbb{1}(\text{model_answer} = \text{reference_answer})
The model receives 1 if the output exactly matches the reference answer (after accounting for the specified formatting requirements), and 0 otherwise. The paper notes that EM is applicable because "Scientific Deep Research tasks are designed to have short, unique, and easily verifiable answers" (Section 2.2.1).
The design choice of EM (rather than fuzzy matching or semantic equivalence) is justified by the task construction: because Response Requirements specify exact formatting (e.g., "two decimal places," "integer format"), ambiguity is eliminated. A model that produces "3.14" when the answer is "3.14159" has demonstrably failed to follow instructions and/or correctly compute the answer. This strictness is a deliberate choice — it means EM directly measures end-to-end scientific correctness rather than "approximately right" understanding.
Evaluation Metric 2: Step-Level Accuracy (SLA)
Step-Level Accuracy addresses the limitation that EM provides only a binary final-answer signal, losing information about partial reasoning correctness. The metric is computed as:
The evaluation procedure uses "an LLM-based judge to compare each model-generated step against the reference solution steps. For each step, the judge determines whether it is correct and provides reasoning" (Section 2.2.1). This is a fine-grained evaluation that avoids binary correctness judgments for the entire solution.
The relationship between SLA and EM is theoretically informative: if SLA is high but EM is low, the model is producing locally correct steps but failing to maintain global reasoning coherence — a specific failure mode that indicates reasoning chain collapse rather than knowledge absence. If SLA and EM are both low, the model lacks foundational understanding. If both are high, the model has mastered the task. The empirical results (Section 4.3) consistently show SLA substantially exceeding EM, validating the paper's claim that the primary failure mode is end-to-end coherence rather than step-level competence.
Task Category 2: Idea Generation (Conception)
Task Design Philosophy
Idea Generation instantiates the Conception quadrant — the creative stage where scientists synthesize existing knowledge to formulate novel hypotheses, methods, or frameworks. The paper acknowledges a fundamental tension: "because idea generation is an open-ended and highly creative task, its evaluation is inherently challenging" (Section 2.1.2).
A crucial scoping decision narrows the task to methodology design rather than open-ended hypothesis generation. The paper explains: "Fully open-ended hypothesis generation involves substantial conceptual freedom and requires extensive expert adjudication to achieve reliable judgments. Due to practical constraints, our current evaluation focuses on the method-design component of scientific ideas" (Section 6.4, Limitations). This means that models are asked to propose how to investigate a research question (including implementation steps, data, evaluation metrics) rather than what the high-level hypothesis is — a more constrained task that enables partially automatable evaluation.
Task Input Specification
Each Idea Generation question provides six input components designed to simulate the pre-ideation knowledge state of a researcher who has completed a literature review:
- Related Work (RW): "A summary of existing research relevant to a certain research direction, providing context for new ideas."
- Challenge (C): "The current challenges in the field and the limitations of existing solutions."
- Limitation (L): "Specific shortcomings or constraints of current research that new ideas need to address."
- Motivation (M): "The perspective and motivation of addressing the limitations in this research direction."
- Task Objective (TO): "The primary goal of the task, such as generating ideas that solve identified challenges or improve existing solutions."
- Existing Solutions (ES): "A description of the current approaches or solutions available in the field."
The input structure is deliberately cumulative: Related Work establishes context, Challenges and Limitations identify gaps, Motivation provides the rationale, and Task Objective defines the target. Together, these components constrain the idea generation space to a specific, well-defined research problem — making evaluation tractable while preserving scientific authenticity.
Task Output Specification
The model must produce a structured idea decomposed into six interrelated components:
- Core Idea (CI): "The central novel idea or concept generated to address the research challenge."
- Implementation Steps (IS): "The steps or procedures required to implement the core idea."
- Implementation Order (IO): "The sequence in which the implementation steps should be executed."
- Data (D): "The data that will be used to implement the idea or evaluate its effectiveness."
- Evaluation Metrics (EM): "The criteria for assessing the success or relevance of the generated idea."
- Expected Outcome (EO): "The anticipated result or contribution the idea is expected to achieve."
The decomposition into these six components is what makes the evaluation "more systematic and tractable" (Section 2.1.2). Rather than judging a holistic idea narrative — which would require extensive expert review — each component can be assessed against specific criteria. The task formulation is:
Hybrid Subjective-Objective Evaluation Framework
The evaluation framework for Idea Generation is the most complex in SGI-Bench, combining subjective pairwise comparisons by LLM judges with objective computable metrics across four dimensions.
Subjective Evaluation via LLM Judges
For each dimension, model-generated ideas are compared against expert-written reference ideas through pairwise comparisons. The procedure uses "three different LLM judges, each casting two independent votes, resulting in a total of six votes per dimension" (Section 2.2.2). The pairwise win rate against the reference idea is used as the subjective component. Using three judges with two votes each (six total) rather than a single judge reduces the impact of judge-specific biases and provides a more robust signal. The specific LLMs used as judges are not named in the paper.
Objective Dimension 1: Effectiveness
Effectiveness measures whether the model-generated idea addresses the intended research challenge. The objective component uses keyword matching:
The procedure is: "For each reference idea, human experts extract its 3–5 most essential keywords. We compute the hit rate of these keywords in the model-generated idea, allowing semantic matches to avoid underestimating effectiveness" (Section 2.2.2). The semantic match allowance is important — it prevents penalizing models that express the same concept with different terminology. The averaging with the LLM judge win rate balances objective content coverage with subjective quality assessment.
Objective Dimension 2: Novelty
Novelty measures how different the generated idea is from existing work. The objective component quantifies dissimilarity:
The dissimilarity score is computed by "measuring the dissimilarity between the model-generated idea and prior related work. Lower similarity indicates that the model proposes ideas not present in existing literature and therefore exhibits higher creativity" (Section 2.2.2). The paper does not specify the exact embedding model or similarity metric used, but the principle is clear: ideas that are semantically distant from existing literature receive higher novelty scores.
Objective Dimension 3: Detailedness
Detailedness measures whether the idea contains sufficient implementation specifics. The objective component has two sub-components:
The completeness score checks "whether the idea contains required components (Core Idea, Implementation Steps, Implementation Order, Dataset, Evaluation Metrics, Expected Outcome)" — in other words, whether all six output components are present. The redundancy penalty is "computed via sentence-level semantic similarity. Ideas with many repetitive sentences are penalized, as verbosity without substance does not constitute genuine detail" (Section 2.2.2). This penalty is crucial: it prevents models from gaming the completeness score by generating long, repetitive text that nominally contains all required components but lacks substantive detail.
Objective Dimension 4: Feasibility
Feasibility measures whether the proposed implementation can actually be executed. The objective component uses graph matching:
The procedure is: "For each research direction, domain experts provide a standardized implementation graph containing the essential nodes and their execution order. We extract an implementation graph from each model-generated idea and compute its similarity to the expert template. A low similarity indicates that the proposed idea does not align with accepted solution workflows and is therefore infeasible" (Section 2.2.2).
The implementation graph approach is a specific, falsifiable way to measure feasibility: if the model proposes steps that cannot be ordered into a valid execution graph (because of missing dependencies, circular dependencies, or incompatible operations), the graph similarity will be low. The expert template encodes domain knowledge about what constitutes a valid experimental workflow.
Design Rationale for the Hybrid Framework
The averaging of objective and subjective scores for each dimension reflects a deliberate epistemological stance: neither purely automated metrics (which might miss qualitative aspects of scientific quality) nor purely human evaluation (which is expensive and potentially inconsistent) is sufficient alone. The hybrid approach attempts to capture both structural properties (via objective metrics) and scientific judgment (via LLM judges), with the averaging ensuring that neither dominates. The paper acknowledges limitations of this approach in future work discussions, noting that "fully open-ended hypothesis generation... requires extensive expert adjudication to achieve reliable judgments" (Section 6.4).
Task Category 3: Dry/Wet Experiment (Action)
Dry Experiment: Code Completion for Computational Science
Task Design Philosophy
Dry experiments operationalize the Action quadrant for computational and simulation-based science. The paper chooses a code-completion formulation: "where the model fills in missing components of an existing scientific script rather than generating an entire project from scratch" (Section 2.1.3). This is a deliberate scoping choice for reproducibility — generating complete scientific pipelines from scratch would produce highly variable outputs that are difficult to evaluate automatically, while filling in masked functions within a known context provides a controlled test of scientific code understanding.
Task Input Specification
Each Dry Experiment provides three components:
- Background (B): "Information from relevant scientific code, providing context for the dry experiment." This typically includes a natural-language description of the scientific problem, the computational approach, and relevant domain concepts — analogous to the introduction section of a scientific computing paper.
- Data Code (D): "The data used in the experiment, including any code snippets or predefined inputs." This is a complete, executable script that generates or loads the data needed by the main analysis code. The model does not modify this component.
- Main Code (M): "The core experimental code where some functions may be masked or missing." This is an analysis script with certain functions replaced by stubs containing function signatures, docstrings (including argument types, return types, functional descriptions, and example usage), and tags indicating the function category (e.g., "[Numerical calculation]", "[Statistical analysis]", "[Simulation]").
Figure 16 provides a concrete example: a quantum chemistry experiment where the construct_trial_wavefunction function is masked. The function header is preserved: def construct_trial_wavefunction(positions, params, state_index, wavefunction_type='gaussian') along with the docstring describing its purpose and arguments. The model must infer the algorithmic logic from the scientific context and the function's interface specification.
Task Output Specification
The model must produce:
- Functions (F): "The missing functions in the main code M, which the system is tasked with generating or completing" (Section 2.1.3).
The task formulation is:
Six Function Categories
To characterize the scope of computational science tested, the paper defines six function categories (Table 2):
- Numerical Calculation: "Basic mathematical computations required to support physical or chemical modeling." Examples include integration, differentiation, solving equations, and computing physical quantities.
- Statistical Analysis: "Processing experimental data using descriptive or inferential statistics to identify trends and distributions."
- Simulation: "Running computational simulations (e.g., molecular dynamics, finite element analysis) and filtering results for relevant conditions."
- Metric Calculation: "Computing evaluation metrics such as accuracy, error, or performance indicators for validating experiments."
- Data Processing: "Handling raw data before and after experiments, including normalization, cleaning, and feature extraction."
- Predictive Modeling: "Applying machine learning methods to categorize, predict, or group experimental results."
Each problem contains 5 unit tests. The distribution across categories (Figure 9e) shows Numerical Calculation as the largest category, consistent with its foundational role in scientific computing.
Evaluation Metric 1: Pass All k Unit Tests (PassAll@k)
PassAll@k measures the proportion of problems where the generated code passes at least k of the 5 unit tests. The mathematical definition distinguishes it from the standard Pass@k metric:
The paper explicitly contrasts this with Pass@k: "While Pass@k requires only one successful attempt out of k trials, PassAll@k demands that at least k attempts pass the unit tests. Consequently, PassAll@5 represents the most challenging criterion" (Section 2.2.3).
This distinction is important for scientific code: Pass@k (common in general code generation benchmarks) allows a model to succeed by generating many attempts and having one work by chance. In contrast, PassAll@k requires the generated code to pass multiple diverse tests simultaneously, which better reflects the rigor needed for scientific computation where a single correct output under lucky conditions is insufficient.
The paper reports PassAll@1, PassAll@3, and PassAll@5 (Table 7), with PassAll@5 being the strictest and used for the SGI-Score aggregation. The gap between PassAll@1 and PassAll@5 reveals how many problems have partial but not complete correctness — a phenomenon the paper finds to be substantial.
Evaluation Metric 2: Average Execution Time (AET)
AET quantifies code efficiency:
where $t_i$ is the execution time of the $i$-th test case and $N$ is the total number of test cases (Section 2.2.3). This metric captures whether the model generates computationally efficient implementations, which matters for large-scale scientific simulations.
Evaluation Metric 3: Smooth Execution Rate (SER)
SER measures basic code robustness:
This metric is independent of output correctness: it captures whether the code runs without crashing, raising exceptions, or producing runtime errors, regardless of whether the computed values are correct. The empirical results (Section 4.5.1) show that many models achieve SER > 90% while PassAll@5 remains below 37%, confirming that "syntactic fluency ≠ scientific computational reasoning" (Section 6.1).
Wet Experiment: Protocol Planning for Laboratory Science
Task Design Philosophy
Wet experiments operationalize the Action quadrant for laboratory science. The paper chooses a protocol-design formulation with a predefined action space: "where the model composes a sequence of experimental actions and parameters from a predefined action space" (Section 2.1.3). This is again a scoping choice for reproducibility — rather than generating free-form experimental protocols (which would be extremely difficult to evaluate automatically), the model selects and sequences from a finite set of standardized atomic actions.
Task Input Specification
Each Wet Experiment provides:
- Background (B): "Information from relevant experimental procedure." This is a natural-language description of the scientific objective, the experimental context, and any constraints — analogous to the methods section of a scientific paper.
- Action Pool (AP): "A predefined set of atomic actions that can be used in the experiment, along with explanations and corresponding input/output definitions" (Section 2.1.3).
The action pool is a critical design element. Each action is defined with explicit input/output types, parameter specifications, and functional descriptions. Appendix cases (Sections A.3.4) show examples: for a life science wet experiment, actions include <Fix_tissue_in_formalin>(tissue, fixative), <Embed_tissue_in_paraffin>(fixed_tissue), <Section_tissue>(tissue_block, thickness), <Stain_with_antibody>(tissue_section, antibody, concentration), and so on. Each action specifies what it consumes (inputs), what it produces (outputs), and the parameters it requires.
The action pool approach enforces several desirable properties:
- Combinatorial control: the space of possible protocols is finite and well-defined, enabling systematic evaluation.
- Grounding in real practice: actions correspond to actual laboratory operations, not abstract descriptions.
- Type checking: input/output types enable verification that action sequences are type-consistent (e.g., an action requiring a "tissue section" as input cannot precede the action that produces tissue sections).
Task Output Specification
The model must produce:
- Atomic Action Order (AAO): "The order in which atomic actions should be executed."
- Atomic Action Parameters (AAP): "The parameters associated with each atomic action (e.g., reagents, temperature)" (Section 2.1.3).
The task formulation is:
Evaluation Metric 1: Sequence Similarity (SS)
Sequence Similarity measures how well the model's action ordering matches the reference protocol. The metric uses inversion counting:
where $\text{Inv}(\text{seq}_{\text{model}}, \text{seq}_{\text{ref}})$ is the number of discordant pairs between the sequences, $n$ is the sequence length, and $\frac{n(n-1)}{2}$ is the maximum possible number of inversions.
The metric is normalized to [0, 1]: SS = 1 means the sequences are identical (zero inversions), and SS = 0 means maximal disorder (every pair is inverted relative to the reference). This metric captures ordering errors without requiring exact position matching — two sequences that differ by a single misplaced step receive a high but not perfect score, proportional to how many pairwise order relationships are violated.
Evaluation Metric 2: Parameter Accuracy (PA)
Parameter Accuracy measures correctness of the parameters assigned to each action:
The paper notes an important evaluation detail: "permutation-equivalent parameter groups are treated as identical (e.g., ⟨action 1⟩(B, C) and ⟨action 1⟩(X, Y) are identical when B = X and C = Y)" (Section 4.5.2). This prevents penalizing models for parameter ordering when the order is not semantically meaningful. The evaluation counts a parameter as correct if it matches the reference value, accounting for the permissible equivalence classes.
The SGI-Score for Wet Experiments is computed as the average of Sequence Similarity and Parameter Accuracy, reflecting that both ordering and parameterization are necessary for correct protocol execution.
Task Category 4: Experimental Reasoning (Perception)
Task Design Philosophy
Experimental Reasoning instantiates the Perception quadrant — the data analysis stage where scientists interpret empirical evidence to draw conclusions. The paper focuses on data-analysis-oriented reasoning: "where the model must extract relevant visual or numerical cues from multi-modal sources, compare conditions, and identify causal or descriptive patterns" (Section 2.1.4).
The task uses a multiple-choice design with a minimum of 10 options per question. The paper explains this choice: "We employ a multiple-choice design to ensure objective, automatable evaluation. While practical, this structure constrains the model's ability to express diverse reasoning paths and limits assessment of open-form scientific explanations" (Section 6.4, Limitations). This is an explicit trade-off: the multiple-choice format sacrifices the richness of open-ended scientific reasoning for the reliability of automated scoring.
Task Input Specification
Each Experimental Reasoning question provides:
- Multiple Experimental Images (MEI): "A set of images representing various experimental outcomes or data collected from instruments."
- Question (Q): "A specific question or hypothesis related to the experimental data that requires reasoning or analysis."
The images span five modalities (Table 3):
- Process Images: "Graphical symbols + text describing workflows or variable relations." These capture experimental logic rather than raw data.
- Observation Images: "Raw data from instruments (e.g., telescope, satellite, microscope)." These are direct measurements from scientific instruments.
- Experiment Images: "Photos of instruments, setups, or lab operations." These document experimental configurations.
- Simulation Images: "Generated from computational models/software." These visualize theoretical predictions.
- Visualization Images: "Processed structured data into charts/plots." These reveal patterns in data.
The modality distribution (Figure 9f) shows that visualization and observation images are most common, followed by process, experiment, and simulation images.
Task Output Specification
The model must produce:
- Reasoning (R): "The specific steps in the reasoning process, including calculation, thinking, analysis, etc."
- Answer (A): "The conclusion drawn from analyzing the experimental data, answering the specified question or hypothesis."
The task formulation is:
Four Reasoning Paradigms
The paper classifies the reasoning required into four paradigms (Table 4):
- Signal Perception: "Direct extraction of information from visual signals without heavy prior knowledge." Examples include identifying patterns in telescope images or microscope slides.
- Attribute Understanding: "Requires disciplinary background to interpret key features and scientific attributes." Examples include recognizing crystalline structures in materials science images.
- Comparative Reasoning: "Integrates and contrasts information across multiple images, often cross-domain." Examples include comparing climate model simulations with satellite observations.
- Causal Reasoning: "Goes beyond correlation to infer mechanisms or propose hypotheses." Examples include inferring causal pathways in gene expression from multi-modal experimental data.
The paradigm distribution (Figure 9g) shows attribute understanding and causal reasoning as the most common, followed by signal perception and comparative reasoning. The paper's later analysis reveals that comparative reasoning is the most challenging paradigm (Section 4.6, Figure 25).
Evaluation Metric 1: Multi-Choice Accuracy (MCA)
Multi-Choice Accuracy is the strictest metric for Experimental Reasoning:
where $N$ is the total number of test samples. The model receives 1 for each question where the selected option exactly matches the reference answer, and 0 otherwise. With at least 10 options per question, random guessing would yield approximately 10% accuracy, making this a stringent test of genuine reasoning.
Evaluation Metric 2: Reasoning Validity (RV)
Reasoning Validity measures the quality of the model's intermediate reasoning, independent of whether the final answer is correct. The evaluation uses "an LLM-based judge to assess the model-generated reasoning against a reference reasoning" (Section 2.2.4). The judge assigns a score from 0 to 10 on a continuous scale:
- 0: "completely invalid, contradictory, or irrelevant"
- 10: "fully rigorous, logically coherent, and perfectly aligned with the reference reasoning"
The judge also provides textual justifications for the assigned score, enabling qualitative analysis of reasoning failures. The final RV score is:
where the judge score is the continuous 0–10 rating per sample.
The relationship between MCA and RV mirrors the EM-SLA relationship in Deep Research: a model with high RV but low MCA can produce plausible reasoning chains that lead to incorrect conclusions, indicating a specific failure mode where the model understands the general scientific principles but fails to apply them correctly to the specific evidence. The empirical results confirm that RV consistently exceeds MCA across models (Figure 24).
SGIEvalAgent: The Agentic Evaluation Framework
Design Rationale
The paper argues that "conventional 'LLM-as-a-judge' paradigms are insufficient to handle the diverse and specialized metrics required by SGI assessment" (Section 3). The SGIEvalAgent framework addresses three limitations of simpler evaluation approaches: (1) the need for tool-augmented verification (e.g., web search for fact-checking, Python execution for code testing, PDF parsing for scientific document analysis), (2) the diversity of metrics across tasks (binary exact match, continuous similarity scores, unit test pass rates, LLM-judge ratings), and (3) the desire for user-customizable evaluation that can adapt to specific research questions beyond the predefined metrics.
The framework is structured into four interconnected stages, each orchestrated by a specialized agent (Figure 10).
Stage 1: Question Selection (Questioning Agent)
The questioning agent takes three inputs: a User Query (which can be text, keywords, or questions expressing the evaluation intent), the complete SGI-Bench dataset (tagged with discipline and research area metadata), and a K-value specifying how many questions to select. It outputs Selected Indices pointing to the relevant questions.
The agent "filters questions according to multiple criteria, including disciplinary domain, task category, and evaluation intent specified in the input query" (Section 3.1). When no user query is provided, it defaults to selecting all questions, enabling comprehensive benchmarking. This stage effectively defines the evaluation scope — a user can, for example, request evaluation of only chemistry-related Idea Generation questions at a specific difficulty level.
Stage 2: Metric Customization (Customization Agent)
The customization agent takes: the User Query, the selected SGI-Bench data subset, the Selected Indices, a Tool Pool (web search, PDF parser, Python interpreter), and a Metric Pool (the predefined task-specific metrics from Section 2.2). It outputs Metrics for Evaluation — an augmented set that includes both the predefined metrics and any novel user-specified metrics.
The key capability is that the agent "parses the evaluation intent from user input to formalize customized metric instructions with advanced tools like web search and PDF parser, enabling flexible prioritization of metrics or integration of novel evaluation dimensions" (Section 3.2). Section 5.3.1 provides a concrete example: a user asks "which model has the best rigor in idea generation across science disciplines," and the customization agent formalizes "Rigor (scientific strictness)" as a multi-aspect rubric with six criteria (logical self-consistency, theory/literature grounding, precise problem definition, discipline-aligned terminology, methodological soundness, risk awareness) scored 0–10 per aspect and aggregated with equal weights.
Stage 3: Prediction & Evaluation (Evaluation Agent)
The evaluation agent takes: the selected SGI-Bench data, Selected Indices, the model's generated Responses, the Tool Pool, and the Metrics for Evaluation. It outputs Scores (0–10 per question) and Rationales ("a brief explanation of why the response is correct or incorrect with respect to accuracy, completeness, clarity, and supporting evidence") (Section 3.3).
The agent first executes inference for the target model on the selected questions, then applies each metric to score the responses. For each score, it generates a grounded rationale using the tools to verify factual claims, execute code assertions, or retrieve supporting evidence from external sources. This tool-augmented approach distinguishes SGIEvalAgent from simpler LLM-as-judge pipelines: the evaluation agent can actively verify claims rather than relying solely on its internal knowledge.
Stage 4: Report Generation (Reporting Agent)
The reporting agent takes: the Score List, Rationale List, and User-customized Metrics. It outputs a comprehensive evaluation Report that "both visualizes and quantifies the performance of different LLMs and agents across the selected questions and metrics" (Section 3.4). The report "contextualizes the findings within the broader landscape of scientific discovery capabilities, thereby enabling users to extract actionable insights" (Section 3.4).
Section 5.3.2 provides an example report for the user-customized rigor metric, highlighting that "closed-source models generally exhibit higher rigor under this rubric," "intra-family iterations capture measurable gains," and "higher rigor reflects more structured, well-grounded, and verifiable research plans rather than merely fluent narratives."
Integration with Predefined Metrics
A critical design choice is that customized metrics are "reported alongside SGI-Bench's predefined task metrics rather than replacing them, preserving standardized comparability while highlighting the user's domain-specific focus" (Section 5.3.1). This means that the customization framework augments rather than supplants the standardized evaluation — users can explore novel evaluation dimensions without losing the ability to compare against the benchmark's canonical metrics.
Test-Time Reinforcement Learning (TTRL)
Motivation and Core Idea
TTRL addresses a fundamental challenge in scientific AI: "the inherent absence of ground truth" for open-ended discovery tasks (Section 5.1). Unlike closed-domain tasks (math, code) where solutions can be verified against correct answers, "the generation of novel research ideas is an open-ended problem with no pre-existing 'gold standard' labels. This limitation renders traditional offline training pipelines insufficient for adapting to dynamic and unexplored scientific territories" (Section 5.1).
The research question TTRL probes is: "How can we enhance a model's capability during the inference phase in the absence of ground-truth supervision?" The answer is to optimize the model's policy against reward signals derived from the model's own outputs or environmental feedback — specifically, novelty measured as semantic dissimilarity from retrieved related work.
The paper distinguishes TTRL from the original implementation (Zuo et al., 2025) which "primarily leveraged consensus-based consistency as a reward mechanism for logical reasoning tasks." Here, "we establish novelty as our core optimization objective" because "in the current context" of scientific ideation, innovation matters more than consistency (Section 5.1.1).
TTRL Training Framework (Figure 26)
The framework operates as follows:
-
For a given query
$Q$(a scientific idea generation prompt from SGI-Bench), the policy model$\pi_\theta$generates a group of$k$candidate outputs$\{o_1, \ldots, o_k\}$. The paper uses$k = 8$generations per query. -
Simultaneously, real-time web search retrieves a set of related works
$\mathcal{W} = \{w_1, \ldots, w_n\}$where$n = 4$(the number of retrieved papers per query). -
Each candidate output receives a composite reward:
where $R_{\text{format}}$ is the format reward and $R_{\text{novelty}}$ is the novelty reward. The two components are unweighted (effectively 1:1 weighting).
Format Reward
The format reward enforces structural compliance. The model must encapsulate reasoning within thinking... XML tags and the proposal within <answer>...</answer> tags:
This is a binary reward: 1 if the output follows the required XML structure, 0 otherwise. The format constraint ensures that outputs are parseable and that the model separates its reasoning process from its final proposal — mirroring the separation of reasoning and answer in SGI-Bench's Idea Generation task definition.
Novelty Reward
The novelty reward is the core innovation of TTRL. It quantifies how semantically different the generated idea is from existing literature.
First, the generated idea and each retrieved paper are embedded into a vector space. Let $\mathbf{e}_{\text{idea}}$ be the embedding of the generated answer and $\{\mathbf{e}_{w_j}\}_{j=1}^{n}$ be the embeddings of the $n$ retrieved papers. The average cosine similarity is computed:
where $\cdot$ denotes dot product and $\|\cdot\|$ denotes the Euclidean norm.
Second, this similarity is converted to an innovation score:
where the clip function constrains the output to the range $[0, 10]$. When $S_{\text{avg}} = 1$ (maximum similarity to existing work), $S_{\text{inn}} = 0$. When $S_{\text{avg}} = 0$ (complete dissimilarity), $S_{\text{inn}} = 10$. When $S_{\text{avg}} < 0$ (negative cosine similarity, indicating opposed semantic directions), $S_{\text{inn}}$ would exceed 10 but is clipped to 10.
Third, a gating threshold $\tau = 5$ converts the continuous innovation score to a binary novelty reward:
Why this form matters: The binary reward with threshold $\tau = 5$ encodes a specific inductive bias: ideas must be substantially different from existing work (innovation score above the midpoint of the 0–10 range) to receive positive reinforcement. This prevents the model from receiving reward for marginal modifications that are technically different but not meaningfully novel. A continuous reward proportional to $S_{\text{inn}}$ would reward even trivial deviations, potentially leading to "novelty hacking" where the model learns to produce superficially different but substantively similar ideas that happen to use different vocabulary.
Training Algorithm: Group Relative Policy Optimization (GRPO)
TTRL uses GRPO (Guo et al., 2025) as the training backbone. GRPO is a reinforcement learning algorithm that optimizes a policy by comparing the rewards of multiple outputs generated for the same query, rather than requiring an absolute value function. For the group of $k = 8$ outputs per query, the optimization updates the policy to increase the probability of outputs that received higher rewards relative to the group average.
The paper does not provide the full GRPO loss function, but the core mechanism is: outputs with $R(o) = 2$ (both format and novelty rewards) receive the strongest positive signal, outputs with $R(o) = 1$ (format only, meaning the idea was not sufficiently novel) receive a weaker or neutral signal, and outputs with $R(o) = 0$ (format violation) receive negative signal.
Training Hyperparameters (Table 8)
| Hyperparameter | Value |
|---|---|
| Base Model | Qwen3-8B |
| RL Algorithm | GRPO |
| Precision | bfloat16 |
| Learning Rate | $5 \times 10^{-7}$ |
| Max Length | 2048 tokens |
| Generations (G) | 8 |
| Temperature | 1.0 |
| Batch Size | 4 |
| Related Works (n) | 4 |
| Weights | 1:1 (format:novelty) |
The high temperature (1.0) is notably different from the temperature-0 setting used for benchmark evaluation — it encourages diverse exploration during training, essential for discovering novel ideas. The low learning rate ($5 \times 10^{-7}$) reflects the need for stable optimization when the reward signal is noisy (novelty is measured relative to variable web search results) and the base model's capabilities must be preserved rather than catastrophically overwritten.
Training Dynamics (Figure 27)
The training curves show a two-phase process over approximately 800 steps. The Format Reward "rises rapidly and saturates near 1.0 within the first few steps, indicating that the model quickly adapts to the rigid XML structural constraints" (Section 5.1.3). Once the format is stabilized (around step 50–100), the Idea Reward (novelty) "exhibits a consistent upward trend throughout the training steps, driving the total reward to converge at a higher value" (Section 5.1.3).
Quantitatively, the average novelty score of the model's outputs increased from a baseline of 49.36 to 62.06 — an improvement of approximately 12.7 points — "achieved entirely without ground-truth labels" (Section 5.1.3).
Why TTRL matters for SGI
The TTRL experiment is not just a method contribution but a philosophical probe into the nature of SGI. By demonstrating that a model can self-improve on open-ended scientific ideation without labeled supervision, the paper provides empirical support for viewing "Scientific General Intelligence not as a static property, but as a dynamic capacity that can evolve through iterative, self-reflective reasoning cycles" (Section 5.1). The improvement from 49.36 to 62.06 represents a model learning to be more scientifically creative through interaction with its environment (web search) and reinforcement from novelty — a primitive form of the self-corrective loop that defines the PIM cycle.
However, the paper is careful about scope: TTRL optimizes only novelty, and the qualitative case study (Figure 28) shows that improvements manifest as more structurally specific technical proposals (e.g., "dual-branch transformer" instead of generic "transformer-based architecture") rather than demonstrated feasibility. Combining novelty rewards with feasibility, rigor, or experimental constraint rewards is flagged as "a crucial next step for reliable scientific ideation" (Section 6.2).
4. Key Insights and Innovations
Innovation 1: SGI as a Measurable, Workflow-Faithful Construct Grounded in Educational Theory
Prior to this work, the term "scientific intelligence" in AI evaluation was deployed informally — a label attached to whatever a particular benchmark happened to measure, whether that was multiple-choice physics questions (PhyBench), chemical property prediction (ChemBench), literature comprehension (SciAssess), or code generation (SciCode). Each benchmark carved out a slice of scientific cognition and implicitly treated that slice as representative of the whole, but no framework provided principled criteria for determining whether the union of all slices constituted genuine scientific intelligence. The dominant assumption was additive: if a model passes enough discipline-specific and stage-specific tests, it can be described as scientifically capable.
This paper makes a fundamental conceptual move that distinguishes it from the additive paradigm: it grounds the definition of Scientific General Intelligence in the Practical Inquiry Model (PIM), an established educational theory of critical thinking (Garrison, Anderson, and Archer, 1999, 2001), and argues that SGI requires competence across an interdependent cycle — Deliberation, Conception, Action, and Perception — rather than a collection of independent competencies. The theoretical insight is that the PIM's cyclic structure implies a stricter requirement than coverage: a model must not only perform each quadrant's tasks individually but maintain coherence across the transitions between them. A system that retrieves literature brilliantly but designs infeasible experiments has failed the SGI test not because of an arbitrary benchmark design choice, but because it cannot close the loop from Deliberation to Action — a failure mode that additive benchmarks cannot detect because they never demand cross-quadrant integration.
The significance of this move extends beyond benchmark construction. It provides falsifiable boundary conditions for claims about AI scientists. Under the PIM-grounded definition, showing that a model can write research papers (DeepResearch Bench) or generate hypotheses (MOOSE-Chem2) is not sufficient to claim SGI — the model must also execute experiments and interpret their outcomes within a coherent workflow. Conversely, the definition provides a roadmap for what capabilities must be developed: improvements in any single quadrant (e.g., better code generation for Dry Experiments) are necessary but not sufficient for SGI progress. This reframing transforms SGI from a vague aspiration — "models are getting better at science" — into a measurable construct with four jointly necessary dimensions.
The empirical architecture of SGI-Bench — four task categories explicitly mapped to four PIM quadrants, with separate metrics per task but an aggregate SGI-Score that equally weights all categories (Table 5) — is a direct operationalization of this theoretical stance. The choice to weight all quadrants equally encodes the claim that deliberation without execution is as incomplete as execution without interpretation. The finding that the best model achieves an SGI-Score of only 33.83/100, despite individual models achieving relatively strong performance in isolated categories (e.g., GPT-5 achieving 55.40 on Idea Generation average), validates the theoretical claim: fragmentary competence does not aggregate into integrated scientific intelligence.
Innovation 2: The Diagnosis That Feasibility — Not Novelty — Is the Bottleneck in LLM Scientific Ideation
Prior work on AI-driven scientific ideation has overwhelmingly focused on novelty as the primary evaluation axis. Benchmarks like AI Idea Bench 2025 and MOOSE-Chem2 evaluate whether generated ideas are new relative to existing literature, and systems are rewarded for proposing concepts that human judges find creative or surprising. The implicit assumption — inherited from the broader creativity-in-AI literature — is that generating novel ideas is the hard part, and that feasibility is a downstream concern that can be handled by human scientists or later-stage tool assistance.
SGI-Bench's Idea Generation results (Table 6) provide a striking empirical refutation of this assumption. Across all evaluated models — open and closed — feasibility is uniformly the lowest-scoring dimension. The best model for novelty (GPT-5, 76.08) achieves only 18.87 on feasibility; the best model for feasibility (o3, 22.90) is substantially below its own novelty score of 73.74. Even open-source models show the same pattern: Qwen3-8B achieves 20.58 on feasibility but 49.36 on novelty. The gap between novelty and feasibility is not a marginal difference — it is a factor of 3–4× for most models.
This diagnosis is intellectually significant because it inverts the field's assumed difficulty ordering. The bottleneck in LLM scientific ideation is not generating creative concepts — models are demonstrably capable of recombining distant ideas from their heterogeneous training corpora into plausible-sounding novel proposals. The bottleneck is realization: translating those concepts into executable plans with concrete parameters, resource specifications, dependency-consistent step ordering, and interface definitions. The paper's feasibility metric — based on graph similarity between model-generated implementation plans and expert-provided workflow templates — specifically captures this realizability gap. Models propose steps that cannot be ordered into a valid dependency graph, reference data without acquisition plans, and specify modules without concrete parameter choices — failures that are invisible to novelty-focused evaluation but would prevent a human scientist from executing the proposed research.
The paper identifies specific failure patterns underlying the feasibility deficit (Section 6.1): "(i) data references without acquisition or preprocessing plans; (ii) training and optimization loops that omit concrete hyperparameters or resource assumptions; (iii) algorithmic modules named but not grounded in precise choices (e.g., solver type, training objective, evaluation protocol); (iv) integration steps that fail to specify interfaces, ordering, or data flow." These patterns collectively define a realization bottleneck — a failure mode where linguistic fluency masks planning incompetence. Models can describe what should happen at a high level without specifying how to make it happen — precisely the skill that distinguishes a research proposal from a research plan.
This finding has direct implications for research prioritization. If feasibility rather than novelty is the binding constraint, then efforts to make models "more creative" (e.g., better novelty rewards in TTRL, Section 5.1) address the wrong bottleneck for practical scientific impact. The paper's candid acknowledgment that TTRL improves novelty (49.36 → 62.06) without addressing feasibility (Section 6.2: "novelty alone is insufficient for SGI") implicitly makes this point: the field needs constraint-aware planning, tool-augmented feasibility verification, and training signals that reward executable specificity rather than conceptual innovation alone.
Innovation 3: The Over-Execution/Under-Correction Gap as a Signature of Fragmented Scientific Cognition
A recurring empirical pattern across SGI-Bench tasks is that models achieve relatively strong performance on metrics that measure partial competence or structural validity, but collapse on metrics that demand end-to-end correctness or global coherence. The paper systematically documents this gap across multiple task categories, revealing it as a signature of fragmented scientific cognition rather than a task-specific artifact.
In Scientific Deep Research, Step-Level Accuracy substantially exceeds Exact Match across all evaluated systems: multiple agentic systems reach approximately 50–65% SLA while EM remains below 20% (Figures 12, 13). This gap quantifies a specific failure mode: models produce "locally correct steps but cannot maintain global coherence across long reasoning chains" (Section 6.1). The model correctly executes individual retrieval or computation operations but fails to chain them into a verifiable final claim — a form of reasoning trajectory collapse that is invisible to metrics that only evaluate intermediate steps.
In Dry Experiments, the Smooth Execution Rate (SER) — measuring whether code runs without errors, independent of correctness — exceeds 90% for many top models (GPT-5.1: 96.53, Gemini-3-Pro: 98.85), while PassAll@5 (the strictest correctness metric) peaks at only 36.64% (Table 7). This gap, which the paper characterizes as "syntactic fluency ≠ scientific computational reasoning" (Section 6.1), reveals that models have mastered the surface grammar of scientific programming (importing correct libraries, calling functions with valid types, avoiding syntax errors) while failing to select appropriate algorithms, maintain numerical stability, or satisfy domain constraints. The case study in Figure 18 — where o4-mini uses naïve np.cumsum integration instead of adaptive scipy.integrate.quad, producing cascading scientific errors — concretely illustrates this pattern.
In Experimental Reasoning, Reasoning Validity consistently exceeds Multi-Choice Accuracy across models (Figure 24). Models can "often produce partially coherent narratives even when selecting the wrong option" (Section 6.1), demonstrating understanding of general scientific principles without the precision needed to apply them to specific multi-modal evidence.
The intellectual contribution is not merely the observation of these gaps individually — individual benchmarks have noted that execution rate exceeds correctness or that step-level reasoning is easier than final-answer accuracy. The contribution is the identification of the over-execution/under-correction gap as a cross-task invariant that characterizes the current frontier of LLM scientific capability. Across deliberation (Deep Research), action (Dry Experiment), and perception (Experimental Reasoning), the consistent pattern is: models achieve moderate-to-strong performance on metrics that reward local correctness or structural validity, but collapse on metrics that demand global correctness. This pattern diagnoses a specific kind of fragmentation — not between tasks (models can do each task somewhat), but within tasks (models cannot sustain correctness from beginning to end of a single scientific reasoning chain).
The practical implication is that metric selection is not a neutral choice but encodes assumptions about what constitutes scientific competence. If a benchmark reports only SLA without EM, or only SER without PassAll@5, it will systematically overestimate model capability by capturing local success while missing global failure. SGI-Bench's design of pairing strict (EM, PassAll@5, MCA) with lenient (SLA, SER, RV) metrics for each task category is a methodological innovation that makes this gap diagnostic rather than hidden — enabling researchers to distinguish "model has partial competence" from "model has integrated competence," a distinction that most scientific AI benchmarks collapse.
Innovation 4: Test-Time Reinforcement Learning as a Probe into the Dynamic Nature of Scientific Intelligence
The dominant paradigm in scientific AI evaluation treats capability as a static property: a model is trained (via pretraining, fine-tuning, or RLHF), frozen, and then evaluated on a benchmark. Improvements require retraining with better data, larger scale, or improved architectures. This paradigm implicitly models scientific intelligence as something that is acquired during training and then deployed at inference — a fixed capacity that either exists or doesn't.
The paper's TTRL experiment (Section 5.1) probes an alternative hypothesis: that scientific intelligence — specifically, the ability to generate novel ideas — can emerge dynamically at test time through interaction with the environment, without access to ground-truth labels. The mechanism is simple in concept (generate candidates, retrieve related work via web search, reward dissimilarity, optimize policy via GRPO), but the philosophical implication is profound: if a model can improve its idea generation capability at inference time through self-guided exploration, then SGI is not merely a property of training but a capacity that can be cultivated through process.
The quantitative result — novelty score improvement from 49.36 to 62.06 without labeled supervision — is modest in absolute terms but striking in context. The improvement is achieved entirely through the model's interaction with an open-ended environment (web search results are variable and uncurated), guided only by a reward function that the model itself operationalizes (dissimilarity from retrieved literature). This is a primitive form of the self-corrective loop that defines the PIM cycle: the model generates (Conception), evaluates against external evidence (Deliberation), and updates its policy — closing a loop without human intervention.
The paper's positioning of TTRL as an experimental probe rather than a method contribution is strategically significant. The authors acknowledge that novelty-only optimization is insufficient ("combining novelty with rigor- or feasibility-based rewards is a crucial next step," Section 6.2), and the negative result with ReST^EM-trained revision models (Appendix K, Figure 16) demonstrates that online optimization can backfire when training signals are misaligned. The contribution is not "TTRL solves scientific ideation" but "TTRL demonstrates that SGI is dynamic, opening a research program around multi-objective test-time self-improvement."
This reframing connects SGI evaluation to the broader test-time compute scaling literature (Snell et al., 2024; Muennighoff et al., 2025) but with a crucial distinction: whereas most test-time scaling work optimizes for correctness on tasks with verifiable answers (math, code), TTRL optimizes for novelty on a task where no ground-truth exists. The retrieval-augmented novelty reward mechanism (Section 5.1.1, Equations 3–5) is a specific solution to the challenge of defining rewards for open-ended scientific exploration — using the existing literature as a dynamic baseline against which novelty is measured. The binary gating with threshold τ = 5 encodes the inductive bias that only substantially novel ideas receive positive reinforcement, preventing "novelty hacking" through superficial rephrasing. This mechanism, while simple, represents a conceptual advance: it converts the philosophical problem of "what counts as a good new scientific idea?" into an operational optimization target that can be computed at test time without expert adjudication.
The training dynamics (Figure 27) further support the dynamic-capacity framing. The two-phase optimization — format stabilization followed by steady novelty improvement — suggests that the model is not merely memorizing patterns from its training data but actively learning to explore more distant regions of idea space in response to environmental feedback. The qualitative case study (Figure 28) shows the pre-training output as "standard combination of existing components" (contact map prediction + Rosetta energy functions) while the post-training output proposes structurally specific concepts ("dual-branch transformer," "differentiable physics engine") — suggesting that the improvement is not cosmetic but reflects genuine exploration beyond the model's default generation distribution.
5. Experimental Analysis
Evaluation Methodology
-
Dataset. All experiments use SGI-Bench, comprising 1,263 expert-curated samples across four task categories: 318 Scientific Deep Research questions, 315 Idea Generation questions, 271 Dry Experiment questions, 68 Wet Experiment questions, and 291 Experimental Reasoning questions (Section 2.4). The benchmark spans ten scientific disciplines (Figure 8, Table 9), with questions constructed from raw materials provided by domain experts and drawn from leading journals (Nature, Science, Cell) and Science's 125 Big Questions. Each task category has distinct input-output specifications detailed in Section 2.1. The data construction pipeline (Section 2.3) includes multi-round expert review, rule-based cleaning, model-based cleaning, and difficulty filtering — any question solvable by more than half of six frontier models (GPT-5, Gemini-2.5-Pro, DeepSeek-R1, Kimi-k2, etc.) under web-search and deep-reasoning modes was removed to maintain expert-level challenge.
-
Base model(s). The paper evaluates a wide spectrum of models spanning both open-weight and closed-source families (Section 4.1). Open-weight LLMs include DeepSeek-V3.2, DeepSeek-R1 (Guo et al., 2025), Intern-S1 and Intern-S1-mini (Bai et al., 2025), Kimi-k2 (Kimi Team et al., 2025), Qwen3-VL-235B-A22B, Qwen3-235B-A22B, Qwen3-Max, and Qwen3-8B (Yang et al., 2025), and Llama-4-Scout (Meta AI, 2025). Closed-source LLMs include GPT-4o (OpenAI, 2024), GPT-4.1, GPT-5 (OpenAI, 2025), GPT-5.1, GPT-5.2-Pro, o3 and o4-mini (OpenAI, 2025), Gemini-2.5-Flash and Gemini-2.5-Pro (Comanici et al., 2025), Gemini-3-Pro (Google DeepMind, 2025), Claude-Opus-4.1 and Claude-Sonnet-4.5 (Anthropic, 2025), Grok-3 (xAI, 2025), and Grok-4 (xAI, 2025). For open-source agents, the paper tests SmolAgents (Roucher et al., 2025) with both GPT-4.1 and Gemini-2.5-Flash backends, Owl with both backends (Hu et al., 2025), WebThinker (Li et al., 2025), XMaster (Chai et al., 2025), and InternAgent (NovelSeek Team et al., 2025). For closed-source agents, the paper evaluates OpenAI DeepResearch with o3 and o4-mini backends (OpenAI, 2025), Kimi-Search with Kimi-k2 backend (Moonshot AI, 2025), Doubao-Search with Seed-1-6, Grok-Search with Grok-4 backend (xAI, 2025), and Perplexity with Sonar-Pro backend (Perplexity AI, 2025). Models are not fine-tuned for SGI-Bench; all are evaluated zero-shot. The paper argues the selection is "representative of the capabilities of many contemporary LLMs" (Section 4.1), covering the frontier of both open and closed systems. For benchmarking consistency, temperature is set to 0 for all configurable models, and a standard zero-shot, task-specific prompt template is used across all tasks.
-
Metrics. SGI-Bench employs multi-dimensional, task-specific metrics designed to capture both final-answer correctness and intermediate reasoning quality (Section 2.2). For Scientific Deep Research, Exact Match (EM) measures whether the model's final answer exactly matches the reference answer (binary 0/1), and Step-Level Accuracy (SLA) measures the proportion of model-generated steps deemed correct by an LLM judge relative to reference solution steps. For Idea Generation, four dimensions are assessed via a hybrid subjective-objective framework: Effectiveness (average of keyword hit rate and LLM-judge pairwise win rate), Novelty (average of dissimilarity score from prior work and LLM-judge win rate), Detailedness (average of a completeness score with redundancy penalty and LLM-judge win rate), and Feasibility (average of graph similarity to expert implementation templates and LLM-judge win rate). Each dimension uses three independent LLM judges casting two votes each (six total), and scores are reported as win rates against expert-written reference ideas. For Dry Experiment, PassAll@k measures the proportion of problems where the generated code passes at least k of 5 unit tests (distinct from standard Pass@k in requiring simultaneous correctness across multiple tests), Average Execution Time (AET) measures code efficiency as
$\frac{1}{N}\sum_{i=1}^N t_i$, and Smooth Execution Rate (SER) measures the proportion of code executions without runtime errors. For Wet Experiment, Sequence Similarity (SS) measures ordering fidelity via$\text{SS} = 1 - \frac{\text{Inv}(\text{seq}_{\text{model}}, \text{seq}_{\text{ref}})}{n(n-1)/2}$where Inv counts discordant pairs, and Parameter Accuracy (PA) measures the proportion of correctly specified parameters across actions (with permutation-equivalent parameter groups treated as identical). For Experimental Reasoning, Multi-Choice Accuracy (MCA) measures the proportion of questions where the selected option matches the reference answer (with ≥10 options per question), and Reasoning Validity (RV) measures an LLM judge's 0–10 continuous assessment of the model's reasoning quality relative to reference reasoning. The aggregate SGI-Score (Table 5) is the unweighted average across all five core task categories, where each category is represented by its strictest metric: EM for Deep Research, the average of four idea dimensions for Idea Generation, PassAll@5 for Dry Experiment, the average of SS and PA for Wet Experiment, and MCA for Experimental Reasoning. For multimodal models that lack text-only counterparts (e.g., Qwen3-VL-235B-A22B used in Experimental Reasoning), the paper uses the closest available variant and marks these with an asterisk in Table 5. -
Baselines. The paper evaluates models directly against SGI-Bench's reference answers and expert-written idea proposals rather than against a specific baseline model. For Deep Research, the reference solutions provide both the correct final answer and annotated reasoning steps against which model outputs are compared via EM and SLA. For Idea Generation, model-generated structured ideas are compared against expert-written reference ideas through pairwise LLM-judge evaluations, with objective metrics (keyword matching, dissimilarity, completeness, graph similarity) providing complementary signals. For Dry Experiments, each problem contains 5 unit tests that serve as the ground-truth correctness signal, with PassAll@k reporting what fraction of these tests the generated code satisfies simultaneously. For Wet Experiments, the reference protocol (expert-annotated action sequences and parameters) provides the ground truth against which SS and PA are computed. For Experimental Reasoning, expert-annotated correct answers and reasoning chains provide the reference for MCA and RV. The SGI-Score aggregates performance across all five tasks, enabling model-to-model comparison without a designated "baseline" model. The evaluation framework does not include random-guessing or majority-class baselines in the main results, though for Experimental Reasoning (≥10 options per question), random guessing would yield approximately 10% MCA.
-
Generation budget / compute accounting. The paper does not use a fixed "generation budget" across tasks in the manner of best-of-N sampling studies, since each task category has a distinct formulation: Deep Research requires multi-step reasoning with web search and computation, Idea Generation produces a single structured proposal, Dry and Wet Experiments produce single completions, and Experimental Reasoning selects from multiple-choice options. For agent evaluations (Deep Research), the compute cost includes tool calls (web_search, visit_webpage, python_interpreter, wikipedia_search) whose frequency and latency are analyzed in Section 5.2. For TTRL experiments (Section 5.1), the generation budget is explicitly specified: 8 candidate outputs per query (k=8), with training running for approximately 800 steps at batch size 4 and max length 2048 tokens (Table 8). For benchmark evaluation (Section 4), models produce single outputs per question at temperature 0. The paper does not conduct FLOPs-matched comparisons between models of different scales, unlike the test-time compute scaling literature. This is a meaningful gap: the SGI-Score comparison across model families (Table 5) does not account for differences in model size, inference cost, or tool-use overhead, so "GPT-5 outperforms Qwen3-8B on SGI-Score" is a valid claim, but "GPT-5 is more compute-efficient for scientific reasoning" cannot be inferred from the reported results.
-
Cross-validation / statistical protocol. For Idea Generation's subjective evaluation, the paper employs three independent LLM judges, each casting two votes per pairwise comparison, yielding six total votes per dimension per idea (Section 2.2.2). This multiple-judge design reduces the impact of individual judge biases compared to single-judge evaluation. The LLM judges used are not explicitly named. For benchmark construction, difficulty filtering was performed using six high-performance models; questions that more than half could correctly answer were removed (Section 2.3), establishing a consistent difficulty threshold but not a statistical protocol. The evaluation itself is single-pass (temperature 0, one response per question), so confidence intervals, standard errors, or statistical significance tests are not reported for the main SGI-Score comparisons in Table 5. For Experimental Reasoning's Reasoning Validity metric, the LLM judge assigns a 0–10 continuous score per sample, and the final RV is reported as the mean across samples — but without measures of inter-judge reliability. For Deep Research's Step-Level Accuracy, the LLM judge evaluates each step as correct/incorrect, and SLA is computed as the proportion of correct steps — again without reporting judge consistency metrics. The small size of certain task subsets (e.g., Wet Experiment has only 68 questions total, split across 3 disciplines; Section 2.4) means that per-category or per-discipline breakdowns (Tables 18, 19) are based on very small samples, potentially introducing high variance that is not quantified. The TTRL results report the improvement from baseline novelty score 49.36 to post-training 62.06 (Section 5.1.3) without confidence intervals or significance testing, though the training dynamics in Figure 27 show a clear upward trend across 800 steps.
Main Quantitative Results
Overall SGI-Score: Fragmented Cognition Across the Frontier
The SGI-Score, computed as the unweighted average across all five task categories using each category's strictest metric, reveals that contemporary LLMs achieve only roughly 30/100, with the best model (Gemini-3-Pro) reaching 33.83 (Table 5). This low ceiling holds despite the evaluated models including the most capable systems publicly available as of late 2025 (GPT-5, GPT-5.2-Pro, Claude-Sonnet-4.5, Gemini-3-Pro). The distribution is compressed: most models cluster in the 28–32 range, with only a 5-point spread between the best (33.83) and the weakest frontier model (Qwen3-8B at 19.15). This compression is informative: it suggests that scale alone — Qwen3-8B is orders of magnitude smaller than GPT-5 — does not translate into proportionally better integrated scientific cognition.
A striking pattern is the near-parity between open-source and closed-source systems. Claude-Sonnet-4.5 (closed-source) achieves 32.16, while Qwen3-Max (open-source) achieves 31.97 — a difference of less than 0.2 points. Similar proximity appears throughout the ranking: Grok-4 (closed, 28.68) vs. Intern-S1 (open, 28.10); GPT-4o (closed, 26.87) vs. Qwen3-VL-235B-A22B (open, 28.32). This near-parity challenges the assumption that proprietary models hold an insurmountable advantage in scientific reasoning. However, the SGI-Score masks substantial task-level variation: GPT-5 achieves 55.40 on Idea Generation average (dramatically outperforming all open models) but only 14.47 on Deep Research EM (roughly comparable to DeepSeek-V3.2 at 12.70 and Qwen3-Max at 15.38). The SGI-Score's equal task weighting means that a model dominating one quadrant while failing another can appear "average" overall — a deliberate design choice reflecting the paper's claim that all four PIM quadrants are jointly necessary for SGI, but also a choice that may understate genuine capability advances in specific scientific activities.
The SGI-Score should be interpreted with the understanding that task difficulties are not calibrated to be equal: Deep Research EM of 18.48 (the best result) represents near-complete failure on 81.5% of questions, while Idea Generation average of 55.40 (the best result) represents winning roughly 55% of pairwise comparisons against expert-written ideas. The SGI-Score's equal weighting implicitly treats "gets 18% of deep research questions right" as equally informative as "wins 55% of idea generation comparisons," even though these metrics operate on fundamentally different scales and capture different aspects of performance. This is not a flaw — it follows from the PIM framework's equal weighting of quadrants — but it means the SGI-Score conflates capability levels that may be at very different points on their respective mastery curves.
Scientific Deep Research: High Step-Level Alignment, Collapsed End-to-End Accuracy
The headline finding for Deep Research is a consistent, large gap between Step-Level Accuracy (SLA) and Exact Match (EM) across all evaluated systems (Section 4.3, Figures 12 and 13).
Exact Match is uniformly low. For standalone LLMs (Figure 12), the best EM is 18.48 (Gemini-3-Pro), with most models achieving 10–16%. This means that even the strongest model gets the final answer wrong on over 81% of Deep Research questions. For tool-augmented agents (Figure 13), EM is similarly constrained: the best agent result is SmolAgents(GPT-4.1) at roughly 15–16%, and several agent systems underperform the best standalone LLMs. The paper notes that "EM differences are marginal with the best LLMs matching or exceeding the best agents" (Section 4.3), implying that tool access alone does not resolve the end-to-end accuracy bottleneck.
Step-Level Accuracy is substantially higher. Multiple systems — including several agents and LLMs — achieve SLA above 50%, with the best results around 65% (Figures 12, 13). The paper interprets this gap as evidence of a specific failure mode: "models frequently produce partially correct or locally consistent reasoning steps but struggle to maintain coherence and correctness across the full reasoning chain" (Section 4.3). This is not a knowledge-absence failure (if it were, SLA would also be low) but a reasoning trajectory collapse failure: the model correctly executes individual retrieval or computation operations but cannot chain them into a verifiable final claim.
Difficulty varies by question type. Figure 14 breaks down performance by the four question categories defined in Table 1. Performance across all four categories "rarely exceeds 30%" (Section 4.3), with Data and Properties questions being the weakest. The paper attributes this to the nature of the information required: "data- and property-related questions often rely on detailed numerical specifications or contextual descriptions scattered across disparate sources in the literature, demanding precise retrieval, cross-referencing, and aggregation" (Section 4.3). In contrast, Micro- and Macro-experiment tasks "tend to provide more structured protocols or clearer experimental outcomes, enabling LLMs and agents to reason with fewer retrieval uncertainties" (Section 4.3). This difficulty gradient validates the task type taxonomy and reveals that unstructured numerical retrieval from heterogeneous sources is the hardest sub-skill within Deliberation.
Newer models do not universally outperform predecessors. The paper notes that Grok-4 "exhibits lower EM and SLA than Grok-3 on this benchmark, suggesting that large-scale training may introduce regressions or reduce retention of specialized scientific knowledge" (Section 4.3). This is a notable counterexample to the assumption that larger, newer models monotonically improve on all capabilities. The specific numbers are visible in Figure 12: Grok-3 achieves roughly 13.5% EM and approximately 47% SLA, while Grok-4 achieves roughly 13.3% EM and approximately 42% SLA — a regression of about 5 percentage points in SLA.
Subject-wise analysis reveals substantial domain variation. Table 14 (Appendix A.4) reports LLM Deep Research scores across ten scientific disciplines. Performance is highest in Math (e.g., Qwen3-235B-A22B at 77.78, Gemini-2.5-Pro at 52.00, o4-mini at 48.00) and Information (e.g., Gemini-3-Pro at 35.29, Qwen3-VL-235B-A22B at 16.67), while several domains show near-zero performance from most models: Energy (most models at 0.00), Life (most below 13%), Neuroscience (most at 0.00–8.33%), and Physics (most at 0.00–9.38%). This extreme domain variation suggests that Deep Research capability is not a general skill but is highly dependent on whether the domain's question patterns align with the model's training distribution. The Math outlier (scores up to 77.78) is particularly striking and may reflect contamination from math competition datasets in pretraining corpora rather than genuine scientific deep research capability.
Idea Generation: Novelty Is Not the Bottleneck — Feasibility Is
The headline finding for Idea Generation (Section 4.4, Table 6) is an inversion of the field's assumed difficulty ordering: models are substantially better at generating novel ideas than at making them executable.
Novelty is relatively high across models. GPT-5 achieves 76.08 on Novelty, o3 achieves 73.74, and even open-source models score competitively (Kimi-k2: 69.49, DeepSeek-R1: 63.64). The paper interprets this as evidence that "modern LLMs possess a robust capacity for generating conceptually novel scientific ideas" (Section 4.4), likely stemming from "their broad pretraining over heterogeneous scientific corpora, which enables them to recombine distant concepts across domains" (Section 4.4).
Feasibility is uniformly low. The highest Feasibility score is o3 at 22.90, followed by o4-mini at 22.01, with most models clustering in the 14–21 range. GPT-5 — the overall Idea Generation leader with 55.40 average — achieves only 18.87 on Feasibility, a gap of over 57 points from its Novelty score (76.08). Open-source feasibility peaks at Qwen3-8B (20.58), which is comparable to many closed-source models but still below the best. The paper characterizes this as a realization bottleneck: "conceptual richness does not reliably translate into implementation-ready plans" (Section 4.4).
Detailedness is high for top models but not universal. GPT-5 achieves 85.72 on Detailedness and GPT-5.2-Pro achieves 78.03, but many models score substantially lower (DeepSeek-V3.2: 47.34, Llama-4-Scout: 43.08). The paper notes that "execution details are often underspecified," with typical failures including "(i) data references without acquisition or preprocessing plans; (ii) training and optimization loops that omit concrete hyperparameters or resource assumptions; (iii) algorithmic modules named but not grounded in precise choices (e.g., solver type, training objective, evaluation protocol); (iv) integration steps that fail to specify interfaces, ordering, or data flow" (Section 4.4). The Detailedness metric's redundancy penalty — which penalizes repetitive sentences — means that high scores require substantive specificity, not merely verbose descriptions.
Subject-wise variation exists but is less extreme than Deep Research. Table 16 (Appendix A.4) shows Idea Generation scores across ten disciplines. GPT-5 leads in most subjects with scores in the 47–65 range, while weaker models like Llama-4-Scout score in the 22–30 range. The subject variation is moderate compared to Deep Research's extreme swings (0–77%), suggesting that idea generation capability is more transferable across domains — consistent with the hypothesis that it relies on general recombination and linguistic fluency skills rather than domain-specific knowledge retrieval.
Implications of the novelty-feasibility gap. The finding challenges the research direction implied by prior work on AI ideation (MOOSE-Chem2, AI Idea Bench 2025) which focused on measuring and improving novelty. If feasibility rather than novelty is the binding constraint, then efforts to make models "more creative" address the wrong bottleneck for practical scientific impact. The paper's TTRL experiment (Section 5.1), which optimizes novelty and improves it from 49.36 to 62.06, does not address feasibility — a gap the paper explicitly acknowledges in Section 6.2.
Dry Experiment: Executable Code, Incorrect Answers
The headline finding for Dry Experiments (Section 4.5.1, Table 7) is a systematic gap between code executability and scientific correctness.
PassAll@5 is low even for the best models. Under the strictest criterion (PassAll@5, requiring all 5 unit tests to pass), the best result is Gemini-3-Pro at 36.64%, followed by Claude-Sonnet-4.5 at 35.79% and o4-mini at 35.79%. Under the most lenient criterion (PassAll@1, requiring at least 1 of 5 tests to pass), the best result rises to only 42.07% (Claude-Sonnet-4.5). These numbers mean that the best model fails to produce any correct test case on 58% of problems, and fails to produce fully correct code on 63.4% of problems.
Smooth Execution Rate is high but misleading. Many top models achieve SER above 90%: GPT-5.1 (96.53%), Gemini-3-Pro (98.85%), GPT-5.2-Pro (96.60%). The paper emphasizes that "high execution rates do not guarantee correctness" (Section 4.5.1): the gap between SER (>90%) and PassAll@5 (<37%) quantifies the distance between "code that runs without crashing" and "code that computes the right answer." Several models show noticeably lower SER despite competitive accuracy: Gemini-2.5-Flash (44.65% SER) and Gemini-2.5-Pro (44.65% SER) have the lowest execution rates, meaning their generated code frequently crashes — a robustness issue distinct from the correctness issue affecting models like GPT-5 (75.50% SER, 29.89% PassAll@5) which runs more reliably but is comparably inaccurate.
Closed-source models have a modest but not dominant advantage. The top PassAll@5 scores are: Gemini-3-Pro (36.64), Claude-Sonnet-4.5 (35.79), o4-mini (35.79), Claude-Opus-4.1 (34.69), GPT-4.1 (34.32). The top open-source scores are: Qwen3-Max (33.21), DeepSeek-R1 (33.33), Grok-4 (33.71). The gap between best closed (36.64) and best open (33.71) is less than 3 percentage points — a difference that could plausibly be bridged by improved prompting or fine-tuning rather than representing a fundamental architectural advantage.
Numerical Calculation and Simulation functions are the hardest. Figure 17 breaks down PassAll@5 by function category. Models perform best on Data Processing and Predictive Modeling, where "multiple valid implementations exist and errors are less amplified" (Section 4.5.1). In contrast, "Numerical Calculation and simulation-oriented functions prove substantially more difficult. These tasks typically require precise numerical stability, accurate discretization, or careful handling of domain-specific constraints, all of which amplify small reasoning inconsistencies" (Section 4.5.1). The specific numbers in Figure 17 confirm: Numerical Calculation scores range from roughly 15% to 35% across models, while Data Processing scores range from roughly 24% to 52%. The case study in Figure 18 concretely illustrates this: o4-mini uses np.cumsum (forward Euler approximation) instead of scipy.integrate.quad (adaptive integration) for computing comoving volume, producing a significant deviation in the final volume estimate. The error is not syntactic (both functions are valid Python) but algorithmic — the model fails to recognize that coarse discretization in cosmological integration produces unacceptable cumulative errors.
Wet Experiment: Protocol Planning Remains Brittle
The headline finding for Wet Experiments (Section 4.5.2, Figure 20) is uniformly low Sequence Similarity (SS) and moderate but inconsistent Parameter Accuracy (PA).
Sequence Similarity is low across all models. The best closed-source SS is approximately 35.5 (a model not explicitly named in the text, but visible in Figure 20), while the best open-source SS is below 30. This means that even the strongest model's action ordering shares only about 35% of pairwise order relationships with the reference protocol — the model gets roughly two-thirds of step pairs in the wrong relative order. The paper identifies recurring error patterns: "insertion of unnecessary steps, omission of essential steps, and incorrect ordering of valid steps" (Section 4.5.2).
Parameter Accuracy is moderate but variable. The top PA result is approximately 40.6 (closed-source), with some open-source models competitive. However, parameter accuracy collapses for certain models: GPT-5 achieves only 20.7 (Figure 20), suggesting that strong performance on other SGI tasks does not transfer to wet-lab parameter specification. The paper notes that "PA appears slightly more optimistic also since permutation-equivalent parameter groups are treated as identical" (Section 4.5.2) — meaning that the PA metric is more lenient than it appears, and the true parameter error rate may be worse than reported.
Temporal and branch-aware planning is the core failure mode. The case study in Figure 21 illustrates the qualitative nature of wet experiment failures. The ground-truth protocol for an NSCLC anti-PD-1 immunotherapy workflow features "a deeply branched structure with precisely coordinated timing and sample-handling procedures" (Section 4.5.2). The o4-mini generated workflow "collapses longitudinal sampling into a single blood draw," executes PBMC isolation only once rather than per time point, performs flow cytometry once without capturing temporal variation, and conflates genomic and immunophenotyping workflows by extracting genomic DNA in parallel with PBMC isolation. These are not minor parameter errors — they represent fundamental failures in experimental design reasoning. The model treats the protocol as a flat list of operations rather than a temporally-structured, multi-branch workflow, revealing that "while they can enumerate plausible wet experiment actions, they struggle to construct experimentally valid, temporally consistent, and branch-aware protocols" (Section 4.5.2).
Subject-wise analysis is limited by small sample size. Table 18 (Appendix A.4) reports SS and PA for Life, Material, and Physics wet experiments. The small total sample (68 questions) means each discipline subset is very small, making per-discipline comparisons potentially unreliable. The pattern that emerges is that Grok-3 achieves the highest Material-SS (41.93) and Physics-PA (58.32), while Intern-S1-mini shows near-zero performance on Material-SS (4.67) and Physics-SS (0.00), indicating that basic protocol planning capability is not universally present.
Experimental Reasoning: Comparative Reasoning Is the Hardest Frontier
The headline finding for Experimental Reasoning (Section 4.6, Figure 24) is that models perform substantially better on Reasoning Validity than on Multi-Choice Accuracy, with closed-source models maintaining a modest lead.
Multi-Choice Accuracy is moderate, with the best at 41.92. Gemini-3-Pro achieves 41.92% MCA, followed by Gemini-2.5-Pro at 41.24%, GPT-5.2-Pro at 39.18%, and GPT-5 at 38.14% (Table 5, Figure 24). The best open-source model is Qwen3-Max at 37.80% (marked with an asterisk indicating a multimodal variant). With a minimum of 10 options per question, random guessing would yield ≤10%, so all models substantially exceed chance — but 41.92% still means the best model selects the wrong answer on more than half of questions.
Reasoning Validity consistently exceeds Multi-Choice Accuracy. Across models, RV scores are typically 10–30 points higher than MCA (Figure 24). The best RV is approximately 71.3, while the best MCA is 41.92 — a gap of nearly 30 points. This pattern parallels the SLA-EM gap in Deep Research and the SER-PassAll@5 gap in Dry Experiments, suggesting it reflects a general property of current LLM scientific reasoning: models can produce locally plausible reasoning chains that lead to wrong conclusions. The paper notes that "even when the final choice is incorrect, explanations often preserve partial logical coherence" (Section 4.6), indicating that models have learned to produce scientifically-sounding narratives without the precision needed to discriminate between closely related hypotheses.
Comparative reasoning is the hardest paradigm. Figure 25 (left panel) breaks down Experimental Reasoning MCA across four reasoning paradigms: Signal Perception, Attribute Understanding, Comparative Reasoning, and Causal Reasoning. The paper reports that "LLMs perform consistently well in causal reasoning and perceptual recognition. In contrast, comparative reasoning emerges as a persistent weakness" (Section 4.6). The specific values visible in Figure 25 and Table 13 confirm: models achieve their highest MCA in Causal Reasoning (many above 40%) and Signal Perception (many 35–50%), while Comparative Reasoning scores are uniformly lower, typically 28–38%. The paper interprets this as evidence that "models struggle when required to contrast subtle quantitative or qualitative differences, a cognitive operation fundamental to scientific evaluation and hypothesis discrimination" (Section 4.6). This is consistent with the broader finding that current models are better at generating plausible narratives (causal reasoning, perceptual description) than at precise, evidence-based discrimination.
Subject-wise variation is substantial. Figure 25 (right panel) and Table 19 (Appendix A.4) reveal a domain gradient: "models achieve their highest accuracy in astronomy, followed by chemistry, energy science, and neuroscience. Conversely, performance declines substantially in materials science, life sciences, and Earth sciences" (Section 4.6). For example, GPT-5 achieves 70.59 MCA in Astronomy but 20.00 in Physics and 30.00 in Energy. The paper attributes this pattern to training data representation: "astronomy and chemistry are easier, while materials science, life science, and Earth science remain hardest, reflecting the mismatch between real scientific visual heterogeneity and training priors" (Section 6.1). Astronomy images (telescope observations, spectra) and chemistry images (molecular structures, reaction diagrams) have more standardized visual conventions that may appear in pretraining data, while materials science micrographs and Earth science satellite imagery present more heterogeneous visual patterns that models have not learned to interpret systematically.
Tool-Integrated Reasoning Analysis (Section 5.2)
For agent systems evaluated on Deep Research, the paper provides a detailed analysis of tool usage patterns, latency, and efficiency.
Retrieval dominates tool calls. Figure 29 (left) shows that web_search is the most frequently invoked tool (539 calls, 33.98% of all), followed by visit_webpage (385, 24.27%), final_answer (358, 22.57%), python_interpreter (200, 12.61%), and wikipedia_search (104, 6.56%). The paper interprets this distribution as evidence that "an external 'retrieve-then-browse' loop remains the dominant path for contemporary agentic systems, reflecting persistent limits in time-sensitive and domain-specific knowledge available to base LLMs" (Section 5.2.1).
Model efficiency varies dramatically in retrieval. The paper highlights a specific comparison: "GPT-4.1 issues large volumes of web_search (168) and visit_webpage (110) that frequently land in slow tiers, whereas Qwen3-Max completes comparable coverage with far fewer retrieval and browsing steps (61 and 59, respectively)" (Section 5.2.1). This implies that retrieval strategy — not just retrieval access — is a significant differentiator in agentic scientific reasoning. The practical recommendation is that "reducing redundant retrieval iterations — via better query formulation and higher-quality extraction on the first pass — has immediate leverage on end-to-end latency, often exceeding gains from marginal improvements to raw model inference" (Section 5.2.1).
Tool latency dominates end-to-end time. Figure 29 (right) visualizes per-tool latency across models. The primary bottleneck is visit_webpage, which exhibits "a 21.28× spread" from 5.37s (Llama-4-Scout) to 114.29s (GPT-4.1). The python_interpreter tool shows a 9.65× range (5.48s to 52.94s), with DeepSeek-R1 at the slow end (52.94s), "consistent with more frequent multi-step error analysis and correction" (Section 5.2.2). The paper identifies a strategic trade-off: "systems optimized for first-attempt correctness minimize tool time but may forgo deeper self-correction, whereas systems favoring iterative refinement accrue longer tool-side latency while potentially achieving more robust final solutions" (Section 5.2.3). This trade-off is directly relevant to the SGI framing: models that self-correct more aggressively (closer to the PIM's iterative cycle) incur higher latency, and the current benchmark does not account for this latency in its scoring — a model that achieves higher accuracy through more extensive self-correction is not penalized for the additional compute, potentially overstating its practical superiority.
Ablation Studies and Robustness Checks
Difficulty filtering effectiveness: The paper does not conduct a formal ablation of the difficulty filtering threshold, but the empirical results provide indirect validation. The consistently low scores across all tasks (e.g., EM 10–20%, PassAll@5 17–37%) confirm that the filtering successfully removed questions solvable by frontier models, since even the best models fail on most questions. However, the absence of a comparison against an unfiltered version of SGI-Bench means we cannot quantify how much harder the filtering made the benchmark. The paper also does not report what fraction of initially constructed questions were removed by the filtering step, which would be informative about the initial difficulty distribution.
Temperature sensitivity: All benchmark evaluations use temperature 0 (Section 4.1), which maximizes determinism but may not reflect typical deployment settings where some stochasticity is desired. The paper does not ablate the effect of temperature on benchmark performance, meaning the reported scores represent a best-case determinism scenario that may not generalize to higher-temperature sampling. For tasks involving creative generation (Idea Generation), temperature effects could be substantial — higher temperature might improve novelty at the cost of feasibility or might degrade all dimensions. This ablation is absent.
Prompt template sensitivity: The paper uses "a standard zero-shot, task-specific prompt template across all tasks" (Section 4.1) but does not compare alternative prompt formulations. Few-shot prompting or chain-of-thought prompting could potentially improve performance on tasks like Deep Research and Experimental Reasoning, but these comparisons are not conducted. The sensitivity of results to prompt engineering choices — a known concern in LLM evaluation — remains unquantified for SGI-Bench.
Idea Generation judge reliability: The Idea Generation evaluation uses three unnamed LLM judges each casting two votes (six total per dimension). The paper does not report inter-judge agreement statistics (e.g., Cohen's kappa, intraclass correlation), leaving open the question of whether different judge choices would produce substantially different scores. The reliance on LLM judges for evaluating scientific creativity and feasibility — domains where even human expert judgment can be unreliable — is a significant methodological choice that is not validated through comparison with human expert ratings. The paper explicitly notes this limitation for open-ended hypothesis evaluation (Section 6.4) but does not extend the concern to methodology-design evaluation, where the same reliability questions apply.
Wet Experiment action pool coverage: The wet experiment action pools are domain-specific and finite. The paper does not analyze whether some model failures stem from the action pool being incomplete (i.e., models want to select actions not present in the pool) versus genuine planning errors. An ablation comparing performance with expanded vs. restricted action pools would clarify whether models are bottlenecked by the action space or by planning capability.
Multimodal model variants in Experimental Reasoning: Table 5 notes that asterisked scores (Qwen3-Max: 37.80, Qwen3-8B: 23.37) "indicate results from different versions of the same series of multimodal models." This means the Experimental Reasoning comparison is not perfectly like-for-like across model families — some models use vision-language variants while others use text-only variants with potentially different capabilities. The paper does not quantify the impact of this confound on cross-model comparisons.
TTRL reward ablation: The paper's TTRL implementation uses an unweighted sum of format and novelty rewards (Equation 1). No ablation of this weighting (e.g., prioritizing format more heavily, adding a third reward component for feasibility or coherence) is reported. The binary novelty reward with threshold τ = 5 is a specific design choice; ablating the threshold (e.g., τ = 3 or τ = 7) or comparing binary vs. continuous novelty rewards would clarify how sensitive the results are to this hyperparameter. The paper also does not report whether TTRL degrades non-novelty dimensions (Effectiveness, Detailedness, Feasibility) — it only reports the novelty improvement (49.36 → 62.06). If novelty gains come at the cost of feasibility losses, the practical value of TTRL for scientific ideation would be limited.
TTRL model scale: TTRL is tested on a single base model (Qwen3-8B). The paper does not evaluate whether the same approach works on larger models or on models from different families. The finding that TTRL improves novelty from 49.36 to 62.06 on an 8B model raises the question of whether larger models would show similar relative gains, smaller gains (if they are already near a novelty ceiling), or larger gains (if they can explore more effectively). This scaling analysis is absent.
Negative result: ReST^EM revision model degrades performance (Appendix K). The paper reports an attempted optimization of the revision model using ReST^EM (Singh et al., 2024) that backfired: "additional sequential revisions substantially hurt performance with this model. At 256 generations, fully sequential performance drops to approximately 33.5% compared to roughly 38.5% at the optimal ratio" (Section 5.3, paraphrased from Appendix K, Figure 16). The paper hypothesizes that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly" (Appendix K). This negative result is included in the appendix rather than the main text, but it is methodologically informative: it demonstrates that naive self-improvement in scientific domains can backfire, validating the paper's caution that TTRL with single-objective rewards (novelty only) is insufficient for robust SGI.
Critical Assessment
Claim 1: "SGI-Bench reveals that LLMs exhibit fragmented scientific cognition with no model achieving integrated competence across all four PIM quadrants."
Does the evidence support this? Yes, with important nuance about what "integrated competence" means. The SGI-Score ceiling of 33.83/100 (Table 5) clearly demonstrates that no model achieves high performance across all five tasks simultaneously. However, the SGI-Score is computed by averaging separately-evaluated tasks — it measures whether a model performs well on Deliberation and Conception and Action and Perception in separate test instances, not whether the same model can string these capabilities together in a single end-to-end workflow. A model could score 50 on Deep Research, 50 on Idea Generation, 50 on Dry Experiment, and 50 on Experimental Reasoning by solving different questions in each category, without ever demonstrating that it can take the output of one stage and feed it into the next. The SGI-Bench tasks are parallel instantiations of PIM quadrants, not sequential instantiations — each task is self-contained, and no task requires the model to chain across quadrants within a single problem. This means the fragmentation diagnosis (models have "isolated fragments" but fail "the iterative loop") is partially an artifact of benchmark design: the benchmark measures isolated capabilities and then interprets the aggregate as revealing fragmentation, rather than directly measuring cross-quadrant integration. The paper's claim about integration failure would be stronger if there existed a task type that required models to perform a complete PIM cycle (e.g., given a literature summary, generate a hypothesis, design an experiment, and interpret mock results — all in one multi-turn interaction). Such a task is absent from SGI-Bench.
Specific strength: The consistent gap between lenient and strict metrics within individual tasks (SLA vs. EM, SER vs. PassAll@5, RV vs. MCA) provides strong evidence for a specific kind of fragmentation — not between quadrants, but within individual cognitive operations. Models can produce locally correct steps without globally correct outcomes, and this pattern replicates across Deliberation, Action, and Perception. This is a genuine empirical contribution that does not depend on the cross-quadrant integration claim.
Missing evidence: The paper does not report correlations between task-level scores. Do models that score high on Deep Research also score high on Idea Generation? Is Dry Experiment performance predictive of Experimental Reasoning performance? Without these correlations, the claim that models exhibit "fragmented" cognition (implying uncorrelated or anti-correlated capabilities) versus "uniformly weak" cognition (implying correlated but low capabilities) cannot be distinguished. If all models score in the 20–40 range across all tasks with high inter-task correlation, the pattern is "uniformly weak" rather than "fragmented." The SGI-Score's equal weighting assumes fragmentation (otherwise differential weighting by task difficulty would be more informative), but this assumption is untested.
Claim 2: "Exact-match accuracy in deep research stalls at 10–20% despite step-level alignment above 50%, revealing reasoning trajectory collapse as the primary failure mode."
Does the evidence support this? Yes, strongly. The SLA-EM gap is large and consistent across all evaluated systems (Figures 12, 13), and the gap direction (SLA > EM) is theoretically informative: it means models can execute individual steps correctly but cannot maintain correctness across the full chain. The qualitative example in Figure 11 (the Chua's circuit deep research case) illustrates the mechanism: each step is a plausible retrieval or computation operation, but the model must correctly chain seven such steps to reach the final answer. If any step introduces an error (wrong paper retrieved, miscalculated constant, misidentified experimental parameter), the final answer is wrong despite most steps being correct. The paper's interpretation — "the failure mode is therefore not mere knowledge absence, but reasoning trajectory collapse under long-horizon scientific inference" (Section 6.1) — is well-supported.
Specific weakness: The SLA evaluation uses an LLM judge to compare model steps against reference steps, but the judge's own accuracy is not validated against human annotations. If the judge systematically overestimates step correctness (e.g., marking a step as correct if it sounds plausible even when the calculation is wrong), SLA would be inflated, and the true SLA-EM gap would be smaller than reported. The paper would be strengthened by a human validation study on a subset of SLA judgments.
Additional concern: The step-level annotation in the reference solutions may not capture all valid reasoning paths. A model might reach the correct final answer through steps that differ from the reference but are scientifically valid; SLA would penalize these steps as "incorrect" if the judge performs literal comparison against the reference. This would artificially depress SLA relative to a more generous evaluation that accepts alternative valid reasoning paths. The paper does not report whether SLA was computed with semantic matching (accepting equivalent but differently-worded steps) or strict literal matching.
Claim 3: "Idea generation suffers from uniformly low feasibility (best model scoring only 22.90) despite high novelty, revealing a realization bottleneck."
Does the evidence support this? Yes, this is one of the paper's strongest empirical findings. The gap between Novelty and Feasibility is large and consistent: GPT-5 achieves 76.08 Novelty vs. 18.87 Feasibility (a 57-point gap), and this pattern holds across all models (Table 6). The feasibility metric's objective component (graph similarity to expert implementation templates) provides a falsifiable, non-subjective measure that does not depend on LLM-judge reliability. If a model proposes implementation steps that cannot be ordered into a dependency graph matching the expert template, that is a structural failure independent of stylistic quality.
Specific nuance: The paper does not disentangle whether the feasibility gap reflects (a) the models' inability to generate feasible plans or (b) the evaluation's stringency in requiring alignment with a specific expert template. A model might propose a novel but genuinely feasible implementation that differs from the expert template in graph structure — this would receive a low Feasibility score despite being scientifically valid. The paper acknowledges this tension implicitly by averaging graph similarity with LLM-judge win rate, but the LLM judge may share the same template-bias. Without human expert evaluation of a sample of model-generated ideas for actual feasibility (independent of template matching), the Feasibility metric's construct validity remains uncertain.
Missing comparison: The paper does not report whether human scientists (e.g., the PhD annotators) can generate implementation plans that score highly on the Feasibility metric. If even human experts struggle to match the template (e.g., because real research plans have legitimate variation), then the Feasibility ceiling may be inherently low, and the models' performance may be less damning than it appears. A human baseline on the Idea Generation task would substantially strengthen the feasibility finding.
Claim 4: "Dry experiments achieve high executability (>90% SER) but collapse on numerical correctness (best PassAll@5 of only 36.64%)."
Does the evidence support this? Yes, with strong quantitative support. The SER-PassAll@5 gap is large and consistent (Table 7). GPT-5.1 achieves 96.53% SER but only 31.00% PassAll@5; Gemini-3-Pro achieves 98.85% SER but only 36.64% PassAll@5. The function category breakdown (Figure 17) further supports the claim by identifying Numerical Calculation and Simulation as the hardest categories — precisely those requiring algorithmic correctness rather than just syntactic validity. The case study in Figure 18 (comoving volume integration) provides a concrete, interpretable example of the failure mechanism.
Specific nuance: PassAll@5 requires all 5 unit tests to pass simultaneously. This is a substantially stricter criterion than Pass@1 (passing at least one test) or even PassAll@1 (which reaches 42.07%). The gap between PassAll@1 and PassAll@5 reveals that many problems have partial correctness — the model gets some but not all test cases right. The paper's choice to report PassAll@5 as the primary metric for the SGI-Score (and to emphasize it in the executive summary) is a deliberate stringency choice that maximizes the apparent gap between executability and correctness. If PassAll@1 were used instead, the best performance would be 42.07% — still low, but less dramatic than 36.64%.
Missing baseline: The paper does not report what PassAll@5 a human scientist with typical programming skills would achieve on these problems. Are the Dry Experiment tasks genuinely solvable by domain experts, or are they intrinsically difficult programming challenges that even humans would struggle with? Without a human baseline, it is unclear whether 36.64% represents "models are far from human-level scientific programming" or "these are genuinely hard problems and 36.64% is actually quite good." The SciCode benchmark (Tian et al., 2024) — which SGI-Bench cites — provides human baseline comparisons for scientific code completion; SGI-Bench's failure to do so is a meaningful gap.
Claim 5: "TTRL demonstrates that SGI is a dynamic capacity that can evolve through test-time self-improvement."
Does the evidence support this? The evidence supports a narrower claim: TTRL improves novelty scores on idea generation from 49.36 to 62.06 on a single 8B model (Section 5.1.3, Figure 27). This is a genuine self-improvement result achieved without ground-truth labels. However, the paper's framing — "SGI is a dynamic capacity" — overclaims relative to the evidence. TTRL improves one dimension (Novelty) on one task (Idea Generation) within one quadrant (Conception) of the PIM cycle. It does not demonstrate that Deliberation, Action, or Perception capabilities improve through test-time learning; it does not demonstrate that novelty improvements translate to better overall SGI-Scores; and it does not demonstrate that the improvement is sustained when the model is evaluated on held-out scientific domains. The qualitative case study (Figure 28) shows the post-TTRL model proposing more structurally specific concepts ("dual-branch transformer," "differentiable physics engine") — but without independent validation that these concepts are scientifically sound, the improvement could reflect the model learning to generate superficially novel-sounding proposals that are no more feasible than its pre-TTRL outputs.
Critical gap: The paper does not report whether TTRL degrades performance on Feasibility, Effectiveness, or Detailedness. If novelty gains come at the cost of feasibility losses (plausible, since the reward function only incentivizes dissimilarity from existing work, not executability), then TTRL may make models worse at the dimensions that SGI-Bench has identified as the primary bottleneck. This is not merely a missing ablation — it is a necessary condition for claiming that TTRL advances SGI rather than gaming one SGI sub-metric. The paper's own acknowledgment ("combining novelty with rigor- or feasibility-based rewards is a crucial next step," Section 6.2) implicitly concedes this limitation but does not quantify its severity.
Additional concern: The TTRL experiment uses web search for retrieving related works, but the retrieved results depend on the search engine, query formulation, and temporal factors (index freshness). The paper does not control for or report the stability of web search results across training. If the retrieved papers change significantly between training steps, the novelty reward becomes noisy in ways that could affect reproducibility. The training dynamics (Figure 27) show steady improvement, suggesting stability, but without controlling for retrieval variability, it is unclear whether the same experiment would produce the same improvement if rerun.
Claim 6: "Closed-source models show only a marginal edge over leading open-source systems."
Does the evidence support this? Yes, for the aggregate SGI-Score (Table 5): the best closed-source model (Gemini-3-Pro, 33.83) leads the best open-source model (Claude-Sonnet-4.5, 32.16) by only 1.67 points. However, this near-parity does not hold uniformly across tasks. On Idea Generation, GPT-5 (closed) achieves 55.40 while the best open-source model (Kimi-k2) achieves 43.17 — a 12.23-point gap. On Deep Research, the gap is smaller: Gemini-3-Pro (closed, 18.48) vs. Intern-S1 (open, 15.74) — a 2.74-point gap. On Dry Experiment, the gap is also modest: Gemini-3-Pro (closed, 36.64) vs. Qwen3-Max (open, 33.21) — a 3.43-point gap. The finding of marginal closed-source advantage is therefore task-dependent: true for Deep Research, Dry Experiment, and Experimental Reasoning; false for Idea Generation, where closed models substantially outperform open models.
Important caveat: The paper's model selection is a snapshot of late-2025 capabilities and does not control for model size, training data, or compute budget. Claude-Sonnet-4.5 is described as "closed-source" but its architecture and training details are partially public; Qwen3-Max is described as "open-source" but its training data and exact parameter count may not be fully disclosed. The open/closed distinction blurs at the frontier, making claims about "marginal edge" ambiguous — are we comparing the best available models regardless of openness, or models matched for scale and then compared on openness? The paper does not make this distinction clear.
Missing analysis: The paper does not investigate whether the modest closed-source advantage in aggregate SGI-Score reflects genuine capability parity or ceiling effects. If all models are near a minimum performance floor on certain tasks (e.g., EM ~10%, Wet Experiment SS ~20–35%), the SGI-Score's compression may reflect task difficulty rather than model similarity. A task with a very low ceiling compresses model differences; a task with a wide performance spread (like Idea Generation, ranging from 29.72 to 55.40) reveals genuine capability gaps. The SGI-Score's equal task weighting means that tasks with low ceilings (where all models perform similarly) dilute the signal from tasks with high variance (where models meaningfully differ) — potentially masking genuine capability advantages of specific models or model families.
Additional Strengths Not Claimed by the Paper
The multi-metric design enables differential diagnosis of failure modes. By pairing strict and lenient metrics for each task, the paper provides a diagnostic toolkit that goes beyond "model X is better than model Y." The SLA-EM gap localizes Deep Research failures to end-to-end chain coherence rather than step-level knowledge. The SER-PassAll@5 gap localizes Dry Experiment failures to algorithmic correctness rather than syntactic fluency. The RV-MCA gap localizes Experimental Reasoning failures to answer selection rather than reasoning narrative plausibility. This differential diagnosis approach is a genuine methodological contribution that could inform targeted model improvements — even though the paper itself does not demonstrate such improvements.
The scientist-in-the-loop construction pipeline is described in sufficient detail to enable replication. Unlike many benchmarks that provide only vague descriptions of their data sources ("curated from the web," "sourced from textbooks"), SGI-Bench specifies the exact research directions, expert involvement, annotation protocol, and quality control stages. The 75 named research directions (Table 9) and the multi-stage cleaning pipeline (Section 2.3) provide a replicable template for extending the benchmark to new disciplines or refreshing existing questions.
Missing Experiments That Would Strengthen the Paper
-
Human baseline on all tasks. The paper evaluates frontier AI systems but never establishes what performance level constitutes "scientifically competent." Are PhD scientists achieving 80% EM on Deep Research? 90% PassAll@5 on Dry Experiment? Without human baselines, the SGI-Score of 33.83/100 is uninterpretable in absolute terms — we don't know whether this represents "models are 34% of the way to human-level SGI" or "models are 34% of the way to the benchmark's ceiling, which is well below human performance."
-
Cross-quadrant chaining evaluation. The PIM framework claims that SGI requires closing the loop across quadrants, but no task in SGI-Bench evaluates this. A multi-turn task where a model must: (1) read a literature summary, (2) identify a knowledge gap, (3) propose an experiment to fill it, (4) evaluate mock experimental results, and (5) update its hypothesis — all in sequence — would directly test the integration claim.
-
Correlation analysis between task scores. Are the same models strong across all tasks, or do different models specialize in different quadrants? This would distinguish "uniformly weak" from "fragmented" capability and would inform whether the SGI-Score's equal weighting is appropriate.
-
Scaling analysis for TTRL. Would TTRL produce larger gains on larger base models? Would the novelty improvement generalize to held-out scientific domains? These questions are essential for assessing whether TTRL is a practical path toward SGI or a curiosity that works only under specific conditions.
-
Prompt engineering sensitivity. The paper uses a single zero-shot prompt per task. Evaluating performance across a range of prompt formulations (few-shot, chain-of-thought, role-play as scientist) would establish whether the reported performance floor reflects genuine capability limits or suboptimal prompting.
-
Test-retest reliability for LLM-judge metrics. The Idea Generation subjective evaluation and the Deep Research SLA evaluation both rely on LLM judges. Running the evaluation multiple times with different judge models or with the same judge at different temperatures would quantify the reliability of these metrics and establish confidence intervals for the reported scores.
6. Limitations and Trade-offs
6.1 The SGI-Score Measures Independent Task Performance, Not Cross-Quadrant Integration
The assumption or constraint. The PIM framework defines SGI as the capacity to "navigate the complete, iterative cycle of scientific inquiry," where interpretation of results feeds back into revised understanding, which generates new hypotheses, which motivate new experiments. However, SGI-Bench instantiates each PIM quadrant as a separate, self-contained task. As the paper acknowledges in Section 6.4, "the four stages in our benchmark function as probes for different components of scientific inquiry rather than a complete representation of real-world scientific practice." No task in the benchmark requires a model to chain across quadrants within a single problem — to take the output of a literature review and use it to motivate a hypothesis, then design an experiment based on that hypothesis, then interpret its results. Each task provides its own inputs (background, constraints, data) and expects its own outputs (answers, ideas, code, reasoning), with no cross-task state carried forward.
The consequence. The SGI-Score (the unweighted average across all five task categories, Table 5) measures whether a model possesses capabilities in Deliberation, Conception, Action, and Perception simultaneously — but not whether it can integrate them sequentially. A model could score 50 on Deep Research, 50 on Idea Generation, 50 on Dry Experiment, and 50 on Experimental Reasoning by solving different questions in each category, without ever demonstrating that it can sustain reasoning coherence across the transitions between quadrants. This means the paper's central diagnostic claim — that models exhibit "fragmented scientific cognition" failing to "close the iterative loop spanning Deliberation, Conception, Action, and Perception" (Section 6.1) — is partially an interpretation of aggregate independent-task performance rather than a direct measurement of integration failure. The true integration capability of current models — whether they can chain literature synthesis into hypothesis formation into experimental design into results interpretation within a single coherent workflow — is not evaluated by any task in SGI-Bench. A practitioner deciding whether to deploy an LLM for end-to-end scientific research assistance cannot infer from the SGI-Score alone whether the model will maintain coherence across workflow stages or will produce internally inconsistent outputs (e.g., an experimental design that does not actually test the proposed hypothesis).
What evidence exists in the paper. The paper's own task definitions (Section 2.1) and the formulation equations for each task confirm this limitation. For Scientific Deep Research (Section 2.1.1): S, A = LLM/Agent(B, C, D, Q, R) — the model receives the background, constraints, data, question, and response requirements as part of the task input. For Idea Generation (Section 2.1.2): CI, IS, IO, D, EM, EO = LLM/Agent(RW, C, L, M, TO, ES) — the model receives the related work, challenges, limitations, motivation, objective, and existing solutions as pre-packaged inputs, not as outputs of its own prior deep research. The inputs to later tasks are not the outputs of earlier tasks, even for questions drawn from the same research direction. The paper acknowledges in Section 6.4 that "many aspects of scientific work—such as integration across scientific disciplines and risk and safety assessment—remain outside our current scope," but does not explicitly flag the absence of cross-quadrant chaining as a limitation of the SGI-Score's construct validity.
Mitigation status. Not addressed in the current benchmark. The paper suggests future work on "integration across scientific disciplines" (Section 6.4) but does not propose cross-quadrant chaining tasks within the existing PIM framework. Section 6.3, Future Directions, discusses "planning-aware conception" and "branch- and time-aware wet-lab protocol reasoning" but frames these as improvements to individual quadrants rather than as integration challenges.
6.2 Difficulty Estimation Cost Dominates the Practical Utility of the Benchmark Architecture
The assumption or constraint. The paper's evaluation framework treats each SGI-Bench question as an independent unit with a known ground-truth answer (Deep Research, Dry/Wet Experiment, Experimental Reasoning) or an expert-constructed reference (Idea Generation). For a practitioner wanting to use SGI-Bench to evaluate their own model, the cost of running the evaluation is the cost of model inference on 1,263 questions — roughly the product of question count, average output length, and per-token inference cost. This cost is not reported but is conceptually straightforward.
However, interpreting the SGI-Score or any per-task metric as a measure of SGI — as opposed to a measure of benchmark-specific performance — depends on the underlying assumption that SGI-Bench's questions are representative of the distribution of scientific problems that a deployed system would encounter. The paper's difficulty filtering (Section 2.3) — removing questions solvable by more than half of six frontier models — makes this assumption unverifiable without knowing (a) what fraction of initially constructed questions were removed, (b) how the difficulty of retained questions relates to the difficulty of real scientific problems, and (c) whether the filtering introduced selection bias (e.g., preferentially retaining questions that are hard for reasons unrelated to scientific reasoning, such as requiring obscure numerical values or depending on specific paper retrieval).
The consequence. The headline numbers — SGI-Score of 33.83/100, EM of 10–20%, PassAll@5 of 17–37% — are interpretable only relative to the difficulty of the retained questions. If the filtering removed 90% of initially constructed questions, then the benchmark measures performance on an extremely difficult subset that may not be representative of typical scientific workflow tasks. A model scoring 33.83 on SGI-Bench might perform substantially better on the full, unfiltered distribution of scientific problems — or substantially worse on problems of different types not captured by the benchmark's construction pipeline. Practitioners cannot gauge whether the low absolute scores indicate "models are far from useful for scientific work" or "the benchmark is calibrated to be extremely difficult, and 33.83 represents non-trivial capability on challenging problems." The paper's own finding that the best model (Gemini-3-Pro, 33.83) is only marginally ahead of much smaller open-source models (e.g., Intern-S1, 28.10) could reflect either genuine capability parity or a benchmark difficulty ceiling that compresses model differences — without knowing the difficulty distribution, these interpretations cannot be distinguished.
What evidence exists in the paper. Section 2.3 describes the difficulty filtering procedure — "questions that more than half of the models could correctly answer were removed" — but does not report what fraction of questions were removed, what the pre-filtering score distribution looked like, or whether the filtering affected different task categories or disciplines differently. The data distribution visualizations (Figure 9) report only post-filtering counts (318, 315, 271, 68, 291). The paper also does not report the raw (pre-filtering) performance of any model on any task, which would enable computation of the filtering-induced difficulty shift.
Mitigation status. The paper does not address this directly. The difficulty filtering is presented as a quality assurance step to "ensure that the benchmark remains highly challenging" (Section 2.3), but the trade-off between challenge and representativeness is not discussed. The suggestion in Section 6.3 for "retrieval-conditioned quantitative reasoning" and "verification-aware step planning" implicitly acknowledges that current models fail on the benchmark's difficult questions, but does not address whether these failures reflect genuine scientific reasoning gaps or benchmark-specific difficulty artifacts.
6.3 Single Benchmark, Single Evaluation Regime: Generalization to Other Scientific Domains, Modalities, and Workflow Styles Is Unmeasured
The assumption or constraint. SGI-Bench, despite spanning 10 disciplines and 75 research directions (Table 9, Figure 8), remains a single benchmark constructed through a specific pipeline (Section 2.3): topics drawn from Science's 125 Big Questions, raw materials from leading journals (Nature, Science, Cell), annotation by over 100 Master's and PhD holders, and evaluation under a specific regime (zero-shot, temperature 0, task-specific prompt templates). All experiments are conducted on this single benchmark with these fixed evaluation settings. The paper explicitly acknowledges that SGI-Bench covers only "a subset" of real scientific workflows: "Many aspects of scientific work—such as integration across scientific disciplines and risk and safety assessment—remain outside our current scope" (Section 6.4), and the inductive paradigm of scientific discovery ("which relies on data-driven pattern formation and hypothesis emergence") is explicitly excluded (Section 6.4).
The consequence. There are at least four distinct generalization gaps, none of which are quantified:
-
Discipline generalization: The paper's own subject-wise results (Tables 14, 16, 17, 18, 19) show extreme performance variation across disciplines. Deep Research EM ranges from near-zero in Energy, Neuroscience, and Physics to >50% in Math for some models (Table 14). These gaps suggest that SGI-Bench's disciplinary coverage — while broader than most prior benchmarks — does not capture a uniform "scientific reasoning" capability. A model's SGI-Score may be dominated by performance on disciplines where it happens to have strong training data coverage (e.g., Math, Astronomy) and may not predict performance on under-represented disciplines. The paper does not report discipline-stratified SGI-Scores or analyze whether the aggregate score is robust to discipline weighting.
-
Evaluation regime generalization: All experiments use temperature 0 (Section 4.1). At temperature 1.0 (used in TTRL training, Table 8), model outputs would be more stochastic, and performance on correctness-critical tasks (Deep Research EM, Dry Experiment PassAll@5) would likely degrade. The paper does not ablate temperature effects. Similarly, all experiments use a single zero-shot prompt per task; few-shot or chain-of-thought prompting might substantially alter performance, especially for reasoning-heavy tasks like Deep Research and Experimental Reasoning. The sensitivity of SGI-Bench scores to prompting strategy is not measured.
-
Workflow style generalization: SGI-Bench's task formulations represent specific scoping choices: Deep Research is literature-inquiry-centric rather than report-generating (Section 2.1.1), Idea Generation is methodology-design-focused rather than open-ended hypothesis generation (Section 2.1.2), Dry Experiments are code-completion rather than full pipeline construction (Section 2.1.3), Wet Experiments are protocol-design from fixed action pools rather than free-form protocol generation (Section 2.1.3), and Experimental Reasoning is multiple-choice rather than open-form explanation (Section 2.1.4). Each of these scoping choices was made for reproducibility and evaluability, but each also excludes a substantial portion of real scientific practice. A model that excels at completing masked scientific functions might fail at designing a computational experiment from scratch; a model that selects the correct multiple-choice answer in Experimental Reasoning might be unable to produce a free-form interpretation of experimental results. The paper explicitly acknowledges several of these limitations (Section 6.4: "a multiple-choice design... constrains the model's ability to express diverse reasoning paths"), but does not quantify the capability gap between the constrained task formulation and the corresponding unconstrained real-world task.
-
Model family generalization: The TTRL experiment is conducted on a single model (Qwen3-8B, Section 5.1.2). It is unknown whether TTRL's novelty improvements (49.36 → 62.06) would replicate on larger models, on models from different families, or on models with different pretraining data distributions. Similarly, the finding that closed-source models show "only a marginal edge" on SGI-Score (Table 5) is based on a specific set of late-2025 models and may not hold for future model generations or for models with different training paradigms.
What evidence exists in the paper. The subject-wise breakdowns (Appendix A.4, Tables 10–19) provide the only evidence of generalization variance, but this evidence is purely descriptive — no formal analysis of variance components, no confidence intervals on per-discipline scores, and no attempt to predict held-out discipline performance from in-discipline performance. For the evaluation regime and workflow style gaps, the paper provides no ablation experiments.
Mitigation status. The paper acknowledges several of these scoping limitations in Section 6.4: "Deep Research currently emphasizes literature-inquiry–centric tasks" (acknowledged), "Idea Generation evaluation focuses on methodology design" (acknowledged), "Dry Experiment tasks currently support only Python" (acknowledged), "Experimental reasoning in enclosed spaces" via multiple-choice format (acknowledged). The paper frames these as choices made for "standardized, reproducible, and scalable evaluation" and defers broader coverage to "future versions." However, the cumulative effect of these scoping choices on the construct validity of the SGI-Score — whether it measures what it claims to measure ("the ability to autonomously navigate the complete, iterative cycle of scientific inquiry") — is not analyzed.
6.4 LLM-Judge Reliability in Idea Generation and Reasoning Evaluation Is Unvalidated Against Human Expert Judgment
The assumption or constraint. The Idea Generation evaluation (Section 2.2.2) relies on three unnamed LLM judges, each casting two independent votes per pairwise comparison (six total votes per dimension), to produce the subjective component of the Effectiveness, Novelty, Detailedness, and Feasibility scores. The Step-Level Accuracy metric for Deep Research (Section 2.2.1) uses an LLM judge to compare model-generated steps against reference steps. The Reasoning Validity metric for Experimental Reasoning (Section 2.2.4) uses an LLM judge to assign 0–10 continuous scores to model-generated reasoning. In all three cases, the paper assumes — but does not validate — that LLM judges produce evaluations that correlate with human expert judgment.
The consequence. If LLM judges are systematically biased relative to human experts, the reported scores are misaligned with the paper's own scientist-alignment principle. Several specific biases are plausible and unmeasured:
- Novelty inflation: If LLM judges are impressed by fluent, confident-sounding proposals that use technical terminology but are scientifically shallow, they may rate model-generated ideas as more novel than human experts would. This would inflate the Novelty scores in Table 6, potentially masking a larger feasibility gap or making the novelty-feasibility disparity appear smaller than it actually is.
- Step-level leniency: If the Deep Research SLA judge marks steps as "correct" when they sound plausible even if the underlying calculation or retrieval is wrong, SLA would be inflated relative to a human annotator's assessment. The SLA-EM gap (Section 4.3) would then partly reflect judge leniency rather than genuine reasoning trajectory collapse.
- Judge model dependence: The paper does not name the LLMs used as judges, making it impossible to assess whether judge-specific biases (e.g., a judge model that prefers verbose responses, or one that penalizes non-standard terminology) affect the reported scores. If the judge models are from the same families as some evaluated models, there is also a risk of self-preference bias.
The absence of inter-judge reliability statistics (e.g., Cohen's kappa, intraclass correlation) further prevents assessment of whether the six-vote protocol (three judges × two votes) provides sufficiently consistent evaluation, or whether additional judges or votes would substantially alter the results.
What evidence exists in the paper. The paper reports that the Idea Generation subjective evaluation uses pairwise comparisons with "three different LLM judges, each casting two independent votes" (Section 2.2.2), but provides no validation of these judges against human experts. For Deep Research SLA, the judge "determines whether it is correct and provides reasoning" (Section 2.2.1) but the judge's accuracy is not validated. For Experimental Reasoning RV, the judge assigns a 0–10 score "accompanied by justifications for the assigned score" (Section 2.2.4) but the alignment of these scores with human scientific judgment is not measured. The few-shot LLM-judge evaluation paradigm has known reliability issues (Zheng et al., 2023; Wang et al., 2024), including position bias, verbosity bias, and self-enhancement bias — none of which are controlled for or discussed in the paper.
Mitigation status. Not addressed. The paper positions the hybrid subjective-objective framework as a strength — "a robust, interpretable, and comprehensive assessment" (Section 2.2.2) — but does not validate the subjective component's alignment with the scientist-aligned principle that grounds the benchmark. Section 6.4 acknowledges that "fully open-ended hypothesis generation... requires extensive expert adjudication to achieve reliable judgments" and defers this to future work, but does not extend this concern to the methodology-design evaluation where LLM judges are already used. The paper would be substantially strengthened by a human validation study on a representative sample of Idea Generation pairwise comparisons, SLA judgments, and RV scores — even on a small subset (e.g., 50–100 samples) to establish correlation between LLM-judge and human-expert assessments.
6.5 Wet Experiment Evaluation Is Constrained by Small Sample Size and Action-Pool Artifacts
The assumption or constraint. Wet Experiments constitute only 68 questions (5.4% of SGI-Bench's total 1,263 samples), covering only a subset of disciplines (primarily Life, Material, and Physics; Figure 9c). Each question provides a predefined action pool from which the model must select and sequence atomic actions. The evaluation metrics — Sequence Similarity (SS) and Parameter Accuracy (PA) — compare model outputs against expert-annotated reference protocols. The paper notes that "permutation-equivalent parameter groups are treated as identical" in PA computation (Section 4.5.2).
The consequence. The small sample size makes Wet Experiment scores statistically fragile. With 68 questions split across at least three disciplines, per-discipline subsets (Table 18) may contain as few as ~10–20 questions each. A single question where the model happens to select the correct action ordering by chance can swing a per-discipline score by 5–10 percentage points. The reported differences between models — e.g., Grok-3 (37.92 average of SS and PA) vs. GPT-5 (16.31) in Table 5 — may partly reflect variance from small samples rather than genuine capability differences. The paper does not report confidence intervals or standard errors for Wet Experiment metrics, making it impossible to assess whether model-to-model differences are statistically meaningful.
Beyond sample size, the action-pool evaluation paradigm introduces two artifacts:
-
Action pool coverage: The action pools are domain-specific and finite. If a model wants to specify an action that is scientifically appropriate but not present in the provided pool, it must either select a suboptimal action (harming SS and PA) or fail to produce a valid output. The paper does not analyze whether model failures stem from wanting to use actions outside the pool versus genuine planning errors. This confound is particularly relevant for models with strong scientific knowledge that might propose protocols using actions not anticipated by the expert annotators.
-
Template rigidity: SS and PA both compare model outputs against a single expert reference protocol. A model might propose a genuinely valid but different protocol (e.g., a different ordering of independent steps, or different but equally valid parameter choices) that would receive low SS and PA despite being scientifically correct. The PA metric partially addresses this with permutation-equivalence for parameter groups, but SS provides no such flexibility — any deviation from the reference ordering, even for order-independent steps, reduces the score. This penalizes legitimate procedural variation in cases where multiple valid protocols exist.
What evidence exists in the paper. The small sample is documented: "Wet Experiments covering only a subset of disciplines, such as Biology and Chemistry" (Section 2.4), and the data distribution (Figure 9c) shows only three colored segments. The per-discipline breakdown in Table 18 confirms the small subset sizes. The case study in Figure 21 illustrates the qualitative nature of wet experiment failures — the o4-mini generated protocol collapses temporal sampling and conflates genomic/immunophenotyping branches — but this single example does not establish whether the failure is representative or whether alternative model-generated protocols judged as incorrect by SS/PA might be scientifically valid.
Mitigation status. The paper does not discuss sample size as a limitation for Wet Experiments. The action pool constraint is acknowledged in Section 6.4: "the action space for Wet Experiments is an early-stage abstraction; scaling it requires constructing a large, standardized library of atomic actions grounded in real laboratory protocols." However, this frames the limitation as a future scaling issue rather than a current evaluation validity issue. The paper does not propose any mitigation for template rigidity (e.g., evaluating against multiple expert-annotated reference protocols, or using a relaxed SS metric that ignores order for parallelizable steps).
6.6 TTRL Optimizes Novelty Without Measuring the Cost to Other SGI Dimensions
The assumption or constraint. The TTRL framework (Section 5.1) optimizes a composite reward R(o) = R_format(o) + R_novelty(o, W) where R_format is a binary format compliance check and R_novelty is a binary reward gated by a novelty threshold (τ = 5). The reward function contains no terms for Effectiveness, Detailedness, Feasibility, or any other dimension of scientific idea quality that SGI-Bench itself identifies as important. The training objective is to maximize semantic dissimilarity from retrieved related work, subject only to format compliance.
The consequence. TTRL demonstrably improves novelty (49.36 → 62.06 on the base model's intrinsic novelty score, Section 5.1.3), but the paper does not report what happens to the other three Idea Generation dimensions — Effectiveness, Detailedness, and Feasibility — during TTRL training. Several adverse outcomes are plausible and unmeasured:
- Feasibility degradation: The reward function incentivizes ideas that are semantically different from existing work. Ideas that are genuinely novel may also be less feasible (if they were feasible and valuable, existing work might already have explored them). Without a feasibility penalty, TTRL could shift the model's generation distribution toward less feasible ideas — precisely the dimension that SGI-Bench identifies as the primary bottleneck in LLM scientific ideation (Section 4.4, Table 6).
- Effectiveness reduction: Novelty-maximizing proposals may drift away from effectively addressing the stated research challenge. An idea that is highly dissimilar from existing work might be novel because it is irrelevant to the problem — a form of "novelty hacking" where the model learns to propose technically different but scientifically inappropriate approaches.
- Detailedness collapse: The format reward enforces XML structure but does not penalize vague or underspecified implementation steps. A model could maximize novelty while reducing detailedness (e.g., proposing high-level concepts without concrete parameters) to more easily satisfy the dissimilarity threshold without committing to specifics that might overlap with existing work.
The paper's qualitative case study (Figure 28) shows the post-TTRL model proposing a "dual-branch transformer" and "differentiable physics engine" — structurally more specific than the pre-TTRL output's "transformer-based architecture" and "Rosetta energy functions." However, this single example does not establish whether the increased specificity represents genuine scientific improvement or superficial technical vocabulary substitution. Without measuring all four SGI dimensions pre- and post-TTRL, the practical value of TTRL for scientific ideation remains unknown — it could be improving one dimension at the expense of others, resulting in no net gain (or a net loss) in overall idea quality as defined by SGI-Bench's own evaluation framework.
What evidence exists in the paper. The paper reports only the novelty improvement (Section 5.1.3) and the training dynamics of the composite reward and its components (Figure 27). Effectiveness, Detailedness, and Feasibility are not measured for TTRL-trained models. The paper's own Discussion acknowledges this limitation implicitly: "optimizing for novelty in isolation risks ungrounded or implausible ideas; combining novelty with rigor- or feasibility-based rewards is a crucial next step for reliable scientific ideation" (Section 6.2). However, this acknowledgment does not quantify the severity of the trade-off — it is possible that TTRL as currently implemented reduces overall idea quality relative to the base model, and this possibility is not tested.
Mitigation status. The paper identifies multi-objective TTRL as a future direction (Section 6.3): "Future TTRL systems should optimize a portfolio of scientist-aligned rewards (novelty, rigor, feasibility, safety, and experimental cost), and incorporate retrieval trustworthiness and contradiction penalties to prevent spurious innovation." This is a clear research agenda, but it means the current TTRL results — while demonstrating the possibility of test-time self-improvement for scientific ideation — do not demonstrate useful self-improvement in the context of SGI-Bench's multi-dimensional evaluation framework. A practitioner cannot deploy the current TTRL implementation and expect improved idea generation according to the benchmark's own quality criteria.
7. Implications and Future Directions
How This Work Changes the Landscape
This paper does not introduce a new model architecture, training paradigm, or inference technique that directly improves scientific reasoning. Its contribution is methodological and conceptual: it provides the first theoretically-grounded operational definition of Scientific General Intelligence and a benchmark architecture that can measure it. This is an enabling contribution rather than a performance advance — it changes what the field can claim and measure about scientific AI systems, rather than what those systems can do.
The magnitude of the shift is best understood by contrasting it with the status quo ante. Before SGI-Bench, the field's evaluation of scientific AI capability was fragmented across discipline-specific benchmarks (ChemBench, PhyBench, BrainBench) and stage-specific benchmarks (SciCode for code generation, MOOSE-Chem2 for hypothesis evaluation, DeepResearch Bench for literature synthesis). Each benchmark implicitly defined "scientific ability" as the narrow slice of cognition it happened to measure, and there was no framework for determining whether success across multiple benchmarks constituted genuine scientific intelligence or merely a collection of independent skills. The paper's diagnosis — that existing benchmarks "present a fragmented view of scientific intelligence... what is currently measured as 'scientific ability' reflects only a limited slice of true Scientific General Intelligence" (Section 1) — is both accurate and consequential. It implies that claims of scientific capability based on any single existing benchmark are necessarily incomplete, and that the field has been systematically overestimating progress toward AI scientists by evaluating against benchmarks that capture only isolated fragments of the scientific workflow.
SGI-Bench does not resolve this fragmentation — the paper's own results show that no model achieves integrated competence, with the best SGI-Score at only 33.83/100 (Table 5) — but it provides the diagnostic toolkit for identifying and quantifying it. The pairing of strict and lenient metrics within each task category (EM vs. SLA for Deep Research, PassAll@5 vs. SER for Dry Experiments, MCA vs. RV for Experimental Reasoning) enables differential diagnosis of failure modes that were previously invisible. The consistent gap between lenient and strict metrics across tasks — models produce locally correct steps but fail globally, generate executable code that computes wrong answers, articulate plausible reasoning that leads to incorrect conclusions — is a signature of fragmented scientific cognition that no single prior benchmark could detect because none paired metrics in this way. This gap is the paper's most empirically robust and conceptually significant finding, and it has direct implications for research prioritization: efforts to improve "scientific reasoning" that focus on individual component skills (better retrieval, better code syntax, better explanation generation) may improve lenient metrics without moving strict metrics — a possibility that SGI-Bench can now measure, and that prior benchmarks could not.
The paper also reconciles a latent tension in AI-for-science research: the simultaneous existence of impressive demonstrations (models writing research papers, generating drug candidates, solving competition math problems) and persistent skepticism about whether these represent genuine scientific capability. The resolution is that these demonstrations capture individual quadrant competencies — Deliberation for literature synthesis, Conception for idea generation, Action for code generation — while genuine SGI requires cross-quadrant integration. A model can excel at Deep Research and still fail to design a feasible experiment; it can generate novel hypotheses that cannot be implemented. The apparently contradictory assessments ("models are becoming AI scientists" vs. "models are stochastic parrots") are both partially correct when evaluated against different quadrants of the PIM cycle — a reconciliation that the field lacked before SGI-Bench provided a common framework for locating claims within the four-quadrant space.
The research directions this work makes more attractive are those that target the integration gaps rather than isolated capabilities: multi-objective training that jointly optimizes novelty and feasibility rather than novelty alone (Section 5.1's TTRL results demonstrate the limitation of single-objective optimization), evaluation frameworks that measure cross-quadrant chaining rather than independent task performance, and verifier robustness research motivated by the over-execution/under-correction gap. Directions that become less attractive — or at least require re-examination — are those that claim scientific capability based on single-quadrant benchmarks without demonstrating integrated workflow competence. A model achieving 90% on SciCode (code generation) cannot, under the PIM-grounded definition, claim to be an "AI scientist" without also demonstrating Deliberation, Conception, and Perception capabilities — a standard that SGI-Bench now makes empirically enforceable.
Follow-Up Research This Work Enables
Cross-quadrant chaining evaluation with carry-over state. The most significant gap in SGI-Bench's current design — acknowledged in Section 6.4 and discussed in the Limitations section of this analysis — is that the four task categories instantiate PIM quadrants as independent, parallel evaluations rather than sequential, chained ones. No task requires a model to sustain reasoning coherence across the Deliberation → Conception → Action → Perception cycle within a single problem. A natural follow-up would construct a chained evaluation subset within the existing SGI-Bench framework: for a given research direction (e.g., "Tumor Neoantigen Discovery" from the Life Science domain, Table 9), present a model with (1) a literature summary requiring gap identification (Deliberation), then (2) a prompt to generate a methodological proposal addressing that gap while carrying forward the model's own gap analysis (Conception), then (3) a prompt to complete masked code or design a protocol based on its own proposal (Action), then (4) mock experimental results requiring interpretation in light of the proposal (Perception). Evaluation would measure not just per-stage accuracy but cross-stage consistency: does the experiment actually test the hypothesis? Are the interpreted conclusions logically connected to the experimental design? The paper's existing infrastructure — the 75 named research directions, the expert-annotated reference materials, the SGIEvalAgent's customizable metric pipeline — makes this extension tractable. A strong result would show whether models that perform well on independent SGI tasks can maintain coherence when the same capabilities must be deployed sequentially, directly testing the PIM's central claim that SGI requires integrated cognition.
Grounding the Idea Generation evaluation with human expert feasibility judgments. The finding that Feasibility is uniformly the lowest-scoring dimension across all models (Table 6, best result 22.90) is one of the paper's most striking results, but it depends on the Feasibility metric's construct validity — specifically, whether the graph-similarity-to-expert-template component captures genuine infeasibility or penalizes legitimate procedural variation. A critical follow-up would recruit a panel of domain experts (the same PhD-level researchers involved in benchmark construction, or an independent set) to evaluate a representative sample of model-generated ideas for actual feasibility using a standardized rubric independent of template matching. The experiment would compare: (a) SGI-Bench's Feasibility scores, (b) human expert feasibility ratings on a 0–10 scale, and (c) the correlation between the two. This would establish the Feasibility metric's criterion validity — a necessary precondition for interpreting the novelty-feasibility gap as a "realization bottleneck" rather than a metric artifact. If human experts rate model-generated ideas as substantially more feasible than the graph-similarity metric suggests, the bottleneck diagnosis would need revision; if experts confirm low feasibility, the result is strengthened and the paper's call for "planning-aware conception and structured supervision" (Section 6.3) gains empirical urgency.
Multi-objective TTRL with feasibility and rigor rewards. The paper's TTRL experiment demonstrates that test-time reinforcement learning with a novelty-only reward improves novelty from 49.36 to 62.06 (Section 5.1.3), but the paper explicitly notes that "optimizing for novelty in isolation risks ungrounded or implausible ideas" and that "combining novelty with rigor- or feasibility-based rewards is a crucial next step" (Section 6.2). A direct extension would implement a composite TTRL reward function: $R(o) = R_{\text{format}}(o) + \alpha R_{\text{novelty}}(o) + \beta R_{\text{feasibility}}(o) + \gamma R_{\text{effectiveness}}(o)$, where $R_{\text{feasibility}}$ could be operationalized as graph-similarity to a dynamically retrieved implementation template (mirroring SGI-Bench's Feasibility metric), and $R_{\text{effectiveness}}$ could be operationalized as keyword alignment with the problem statement. The experiment would sweep the weights $\alpha, \beta, \gamma$ to identify Pareto-optimal trade-offs between dimensions, and would measure all four Idea Generation dimensions (Effectiveness, Novelty, Detailedness, Feasibility) pre- and post-TTRL. The key hypothesis — that multi-objective TTRL can improve novelty without sacrificing feasibility — would be tested against the null hypothesis that the novelty-feasibility trade-off is inherent and irreducible. A positive result would demonstrate that TTRL can produce genuinely better scientific ideas rather than merely more novel ones; a negative result would suggest that feasibility must be acquired during pretraining or fine-tuning rather than at test time, constraining the scope of test-time learning for scientific applications.
Human baseline establishment across all SGI-Bench tasks. The paper evaluates 30+ LLMs and agents but provides no human performance baseline, making the SGI-Score of 33.83/100 uninterpretable in absolute terms. A critical calibration experiment would recruit domain-expert scientists (PhD students or postdocs in the relevant disciplines) to complete a representative subset of SGI-Bench questions under conditions that parallel the model evaluation: access to web search, time limits, and the same task instructions. The experiment would establish: (a) what SGI-Score human experts achieve, (b) which task categories and disciplines show the largest human-model gaps, and (c) whether the relative difficulty ordering across tasks (Deep Research being hardest by EM, Idea Generation being easiest by average score) holds for humans. This calibration is essential for interpreting all of the paper's quantitative claims — without it, "33.83 represents fragmented, non-integrated cognition" is an assertion about the benchmark's ceiling rather than about model capability relative to a meaningful scientific standard. The paper's existing infrastructure (the SGIEvalAgent's customizable evaluation pipeline, the question selection agent that can sample representative subsets) makes this experiment logistically feasible, though recruiting domain experts across 10 disciplines would require substantial coordination.
Scaling analysis of TTRL across model sizes and families. The paper demonstrates TTRL on a single 8B-parameter model (Qwen3-8B, Section 5.1.2). Whether TTRL's benefits scale with model size — or whether larger models are already near a novelty ceiling that test-time optimization cannot improve — is unknown. A scaling experiment would apply the identical TTRL procedure (retrieval-augmented novelty rewards, GRPO optimization, identical hyperparameters from Table 8) to a range of model sizes within a single family (e.g., Qwen3-8B, Qwen3-32B, Qwen3-235B-A22B) and measure: (a) the pre-TTRL novelty baseline as a function of model size, (b) the absolute novelty gain from TTRL as a function of model size, and (c) the relative gain (percentage improvement) as a function of model size. If the benefit of TTRL diminishes with model size (consistent with a novelty ceiling hypothesis), then test-time learning for ideation is primarily valuable for smaller, more efficient models — a finding with direct implications for deployment economics. If the benefit scales proportionally or super-linearly, then TTRL amplifies the advantage of larger models, reinforcing the case for scale in scientific AI. This experiment would also test whether TTRL transfers across model families (applying the procedure to DeepSeek-R1 or Llama-4-Scout) to establish whether the result is architecture-specific or general.
Practical Applications and Downstream Use Cases
Model selection and capability auditing for scientific AI deployment. Organizations deploying LLMs for scientific workflows — pharmaceutical companies using AI for drug target identification, materials science labs using AI for candidate screening, climate research groups using AI for literature synthesis — face a model selection problem: which of the available models (open and closed) is best suited for their specific scientific pipeline? SGI-Bench's task-category and discipline-level breakdowns (Appendix A.4, Tables 14–19) provide granular, actionable information for this decision. For example, a computational chemistry group focused on molecular property prediction could compare Dry Experiment PassAll@5 scores in Chemistry (Table 17): o4-mini achieves 0.00 while Claude-Sonnet-4.5 achieves 20.00 and GPT-4.1 achieves 20.00 — a substantial difference that justifies model selection even when aggregate SGI-Scores are similar. A neuroscience lab evaluating multimodal experimental reasoning could compare MCA in Neuroscience (Table 19): Gemini-3-Pro achieves 62.50 while GPT-5 achieves 41.67 — a 20-point gap that would directly impact the reliability of automated image-based analysis. The SGIEvalAgent's customizable evaluation pipeline (Section 3) enables organizations to define discipline-specific metrics beyond the predefined ones, such as "molecular validity of generated structures" for drug design or "protocol safety compliance" for wet-lab planning, and benchmark models against these domain-specific criteria. The concrete benefit is avoiding the trap of selecting models based on aggregate benchmarks (MMLU, HumanEval) that do not predict scientific workflow performance — a risk the paper's results make concrete by showing that models with strong general reasoning scores can collapse on specific scientific tasks.
Prioritizing test-time compute investment by task difficulty profile. The paper's finding that some SGI tasks show substantial model variation (Idea Generation averages range from 29.72 to 55.40 across models, Table 6) while others show compressed performance (Deep Research EM clusters at 10–20% across nearly all models, Figures 12–13) has direct implications for resource allocation. For a task where all models perform similarly poorly (Deep Research EM), investing in better model selection yields minimal gains — the bottleneck is likely fundamental capability rather than model choice, and resources are better directed toward tool-integration improvements (Section 5.2's analysis shows that agent tool-use patterns dramatically affect retrieval efficiency) or task-specific fine-tuning. For a task with wide performance dispersion (Idea Generation), model selection matters substantially: choosing GPT-5 (55.40) over Llama-4-Scout (29.72) nearly doubles performance. For tasks where closed-source models substantially outperform open-source models (Idea Generation, Table 6: GPT-5 at 55.40 vs. best open Kimi-k2 at 43.17), organizations with strict data privacy requirements face a concrete trade-off between capability and deployment flexibility that SGI-Bench's metrics can quantify. The paper's per-discipline, per-task breakdowns (Appendix A.4) enable this analysis at finer granularity than the aggregate SGI-Score, allowing organizations to tailor their model selection and compute allocation to their specific scientific domain and workflow stage.
Calibrating expectations for AI-assisted scientific ideation tools. The paper's Idea Generation results establish a clear performance envelope for current LLMs as scientific brainstorming assistants: models can generate ideas that match or exceed human-written references in novelty (GPT-5 Novelty: 76.08, o3: 73.74), but these ideas are consistently less feasible (best Feasibility: 22.90) and only moderately effective (best Effectiveness: 51.36). For organizations building or deploying AI ideation tools — whether for research proposal generation, experimental design suggestion, or hypothesis exploration — these numbers provide a concrete calibration of what to expect and where to invest in human oversight. An ideation tool based on current LLMs can productively serve as a novelty engine (generating diverse candidate approaches that human researchers can filter and refine) but cannot reliably serve as a proposal generator (producing implementation-ready research plans without substantial human revision). The failure modes the paper identifies — underspecified implementation steps, missing resource assumptions, incompatible module interfaces — suggest specific checklist items for human reviewers of AI-generated ideas, and the quadrant-level breakdowns indicate that AI assistance is most immediately valuable in the Conception quadrant (where models are strongest) and least valuable in the Action quadrant (where models struggle with procedural rigor). This calibration is actionable today: it tells research organizations that investing in AI ideation tools is reasonable for the brainstorming phase of scientific workflows but premature for the planning phase, and that human-in-the-loop review should focus specifically on feasibility assessment rather than novelty assessment — an inversion of the traditional assumption that human judgment is most needed for evaluating creativity.
