FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction

ArXiv: 2508.11987

🎯 Pitch

Top LLM agents fail badly at the hardest real-world forecasting tasks, revealing that today's search and reasoning tools cannot substitute for the deep uncertainty management that human experts rely on. FutureX, a live, contamination-free benchmark that updates daily with fresh events, shows that even the best models—led by Grok-4—struggle on open-ended, high-volatility "Super Agent" predictions where answers do not yet exist.

1. Executive Summary

This paper introduces FutureX, a dynamic, live-updating benchmark for evaluating LLM agents on future prediction tasks — forecasting real-world outcomes across domains like politics, finance, sports, and technology before ground-truth answers exist. The benchmark operates a fully automated daily pipeline that curates events from 195 websites (selected from 2,008 candidates) into four difficulty tiers — Basic (few-choice selection), Wide Search (multi-choice exhaustive discrimination), Deep Search (open-ended low-volatility prediction requiring information synthesis), and Super Agent (open-ended high-volatility forecasting under deep uncertainty) — and evaluates 25 models spanning base LLMs, Think&Search agents, open-source deep research frameworks like SmolAgent and AgentOrchestra, and closed-source Deep Research models. Across two weeks of evaluation, Grok-4 achieves the highest overall performance, with search-augmented reasoning models generally outperforming base LLMs, though even top models struggle severely on Super Agent tier events — establishing that current agents can amplify existing capability through tool use and retrieval, but cannot compensate for the fundamental reasoning and uncertainty-aware synthesis demands of the most volatile, open-ended forecasting tasks where human experts maintain a clear advantage.

2. Context and Motivation

The Core Problem: We Cannot Evaluate Whether LLM Agents Can Think Like Human Analysts

The paper addresses a fundamental gap in how we evaluate LLM agents. Existing benchmarks overwhelmingly test agents on static, well-defined problems whose solutions are already known — answering factual questions, navigating simulated websites, or fixing bugs in code repositories. But a critical form of intelligence remains entirely unevaluated: the ability to synthesize dynamic, real-world information, reason under uncertainty, and make predictions about events whose outcomes have not yet occurred. This is precisely what human experts do in fields like financial analysis, political risk assessment, and technology forecasting. The paper frames this as a missing capability in the evaluation landscape:

"While these benchmarks offer valuable insights into agent capabilities, they largely address static, well-defined problems whose solutions are already known. Further, they fail to address a critical gap: the ability to synthesize dynamic, real-world information, process it, and perform complex analysis and reasoning—the very skills possessed by human experts across different domains."

This gap is not merely a matter of academic completeness. It has direct practical consequences: if we cannot measure whether agents can perform expert-level forecasting, we cannot deploy them in high-stakes domains where such capabilities matter most — investment decisions, policy planning, supply chain management, and competitive strategy.

Why Future Prediction Is the Right Task — and Why It Hasn't Been Done

The paper argues that future prediction serves as an ideal testbed for advanced agent capabilities because it inherently bundles together the skills that define expert-level intelligence:

Information gathering: Agents must locate and retrieve relevant, up-to-date evidence from diverse, live sources across the open web — not just retrieve facts from a static corpus.
Data synthesis: Raw information is fragmented, contradictory, and noisy. Agents must integrate signals across multiple domains and weigh their reliability.
Reasoning under uncertainty: The correct answer is not deducible from any single source. Agents must reason probabilistically about causal factors, historical analogs, and emergent trends — the same cognitive demands as professional forecasting.
Dynamic adaptation: The information landscape shifts continuously. Yesterday's key signal may be irrelevant tomorrow, requiring agents to re-weight evidence in real time.

The paper further argues that future prediction is "probably the only methodologically sound way" to evaluate these capabilities. The reasoning is subtle but important:

Temporal leakage is inevitable in retrospective evaluation. If we test on historical data (taking a past event and asking the model to "predict" it as if from an earlier vantage point), the model's training data almost certainly contains information about what actually happened — including search results and analyses written after the fact. This means the model can exploit knowing-the-answer contamination rather than demonstrating genuine forecasting ability.
Retrieval contamination compounds the problem. Even if the model's parameters encode no direct memory of the outcome, internet search results for past events are "inevitably biased by knowledge of events that happened afterward." A search for "2020 election polling October" returns pages that were written or updated after the actual election outcome became known, creating an information environment that did not exist at the time the prediction would have been made.
The only escape is prospective evaluation. Testing on events that have not yet resolved — where no human knows the correct answer — eliminates both forms of contamination by construction.

This logic leads to a strong methodological claim: "the only methodologically sound way to evaluate future prediction capability is to do so prospectively, in a live, forward-looking pipeline." This is a non-trivial philosophical position about evaluation methodology, and it imposes severe engineering demands that the paper must meet.

Where Prior Approaches Fall Short

The paper identifies four categories of prior work, each failing to address the gap in a distinct way.

Static agent benchmarks (WebArena, SWE-bench, GAIA, BrowseComp). These represent the current state of the art in agent evaluation, and the paper acknowledges their contributions: WebArena and mind2web provide high-fidelity web interaction environments, SWE-bench tests code generation against real test suites, GAIA challenges agents with conceptually simple but multi-step real-world questions, and BrowseComp pushes information retrieval to its limits with deliberately entangled queries. However, all share a common limitation: they test agents on tasks with pre-existing answers in controlled or simulated settings. They do not involve genuinely unknown futures, and they do not require the kind of open-ended information synthesis and uncertainty management that professional forecasting demands. The paper is explicit:

"Collectively, these benchmarks offer valuable insights into various aspects of agent performance—from web navigation and tool use to coding—but they primarily operate in controlled environments with predefined task scopes and information. Additionally, these benchmarks do not integrate complex reasoning and information-gathering capabilities, both of which are essential for our proposed FutureX."

Live benchmarks (LiveBench, LiveCodeBench, Arena-Hard). These are closer in spirit to FutureX — they update questions automatically to prevent data contamination — but they still test on tasks with known answers. LiveBench periodically refreshes its question set, but the questions are still drawn from domains where ground truth exists at the time of testing. The paper positions FutureX as an extension of this trend but adds a crucial twist: "where even the events themselves are live." In FutureX, not only are the questions new, but the answers don't yet exist anywhere in the world.

Prior future prediction benchmarks (ForecastQA, Autocast, ForecastBench, FutureBench). This is the most directly relevant prior work, and the paper identifies specific weaknesses in each:

ForecastQA and Autocast contain thousands of events, but their questions are drawn from historical data, making them vulnerable to temporal leakage and retrieval contamination. They also evaluate only base LLMs without search capabilities, which the paper argues is "impractical for real-world future prediction."
ForecastBench (the most recent and ambitious prior effort) takes a step forward by focusing exclusively on future outcomes and being updated monthly with live events. However, the paper identifies three limitations: (1) it "predominantly evaluates vanilla LLMs" without search augmentation or agent frameworks, (2) it "relies on prediction market events, dominated by multiple-choice questions," which "limits both the diversity of events included and, critically, the assessment of an agent's ability to perform open-ended, real-world information gathering," and (3) many prediction market questions are binary choice, making even random guessing achieve 50% accuracy — too easy to be discriminating.
FutureBench is the closest predecessor: it evaluates a single open-source agent on live future events drawn from Polymarket and news websites, updated weekly. But it is extremely small (approximately 30 events), limited in domain diversity, and evaluates only one agent framework. The paper positions FutureX as a dramatic scaling-up of this approach across every dimension — number of events (approximately 500 per week vs. 42 total), number of source websites (195 vs. "several"), number of models evaluated (25 vs. a handful), and diversity of difficulty levels.

The Deep Research Agent Connection

The paper includes a somewhat unusual element in its motivation: an extended discussion of why deep research agents are particularly well-suited for future prediction tasks. This is not merely about having more capable models to evaluate — it is about evaluating a specific class of agent architecture that the authors believe represents the right design pattern for the forecasting domain.

Deep research agents (the paper cites frameworks like SmolAgent from Huggingface and AgentOrchestra from Skywork AI, as well as commercial offerings from Google and ByteDance) are characterized by three properties:

Multi-turn, adaptive planning: Rather than executing a fixed retrieval pipeline, they dynamically adjust their search strategy based on intermediate findings — exploring promising leads, abandoning dead ends, and synthesizing evidence incrementally.
Hierarchical delegation: They decompose complex research questions into sub-tasks handled by specialized sub-agents or tool calls, enabling systematic coverage of diverse information sources.
Iterative tool use: They are designed to make multiple sequential tool calls (web searches, code execution, data processing), with each subsequent action informed by prior results.

The paper argues that these properties map directly onto the demands of future prediction: "Their intrinsic ability to gather, synthesize, and reason about dynamic, real-world information makes them particularly well-suited for tasks involving future predictions, trend identification, and outcome modeling." By including both open-source and closed-source deep research agents in the evaluation, FutureX serves a dual purpose: it tests the frontier of agent capability, and it stress-tests whether the deep research architecture genuinely delivers on its promise for the forecasting domain.

The Engineering and Methodological Challenge

The paper is unusually candid about why this kind of benchmark has not been built before: it is extraordinarily difficult to implement. The challenges are both conceptual and practical:

Conceptual challenge: How do you create questions for events that haven't happened? Unlike static QA benchmarks where questions can be authored once, future prediction requires a continuous pipeline that generates new, high-quality prediction targets daily. The questions must be specific enough to have objectively verifiable answers (within a defined timeframe), challenging enough to discriminate between model capabilities, and diverse enough across domains to test generalization.

Practical challenge: How do you collect answers for events that haven't resolved yet? The benchmark must wait — potentially days or weeks — for ground truth to become available, then automatically retrieve it from the web. This introduces a fundamental evaluation delay that static benchmarks simply don't face. The system must track which events have resolved, crawl websites at the right time, extract answers reliably, and handle cases where answers are unclear or unavailable — all without human intervention in the steady state.

Scale challenge: How do you make this a daily operation? The paper's design requires running up to 25 models daily, each making predictions on 70-100 new events, with a 30-minute timeout per question — a substantial computational orchestration problem. Any failure (model timeout, API outage, crawling error) must not block the entire pipeline.

Quality challenge: How do you ensure events are challenging and reliable? The paper notes that simply scraping prediction market websites yields events that are too easy (binary choices with 50% random baseline), too narrow (limited domains), and sometimes inappropriate (harmful or subjective events). The solution requires a multi-stage curation pipeline involving template generation, variable randomization, LLM-based filtering, and human review — a hybrid approach that the paper frames as necessary for quality while still being scalable.

These engineering challenges are not incidental to the research contribution; they are central to it. The paper positions FutureX's pipeline architecture and the design decisions it embodies as contributions in their own right — demonstrating that live, large-scale, automated evaluation of forecasting capability is achievable, which was not obvious beforehand. The comparison table (Table 1) makes this explicit by evaluating prior benchmarks across dimensions including "Live Update," "Source" type and diversity, "LLM Agents" support, and evaluation "Frequency" — with FutureX checking every box at the highest level.

How This Paper Positions Itself

The paper articulates its contribution through four design principles that serve as both a mission statement and a benchmark specification. Each principle identifies a gap in prior work and defines how FutureX fills it:

Eliminating Data Contamination (contamination-impossible by design). The paper frames this as a fundamental challenge in LLM evaluation — static benchmarks are increasingly compromised as models train on web-scale data that may include test questions. FutureX's response is to make the test answers literally nonexistent at the time of evaluation, transforming the problem from "how do we detect contamination" to "contamination is logically impossible." The paper claims this creates "an absolutely fair evaluation environment" that "compels agents to rely on their capabilities for information gathering, dynamic analysis, and reasoning, rather than on memorization."
Simulating Real-World Challenges and Evaluating Core Intelligence. The paper positions future prediction as a "touchstone for higher-level intelligence" because it requires "high-quality analysis and decision-making under conditions of incomplete information and uncertainty, just as human experts do in many domains." This is a deliberate contrast with benchmarks that test narrow skills (coding, web navigation) in isolation — FutureX demands holistic integration of search, synthesis, reasoning, and judgment.
Large-Scale and Cross-Domain Comprehensive Coverage. The paper claims FutureX is "the largest and most diverse live benchmark for future prediction" — a claim it supports through the numbers (195 websites from a pool of 2,008, spanning 11 domains, generating approximately 500 events per week). The scale serves a methodological purpose: broad coverage prevents overfitting to particular domain patterns and enables the analysis of domain-specific strengths and weaknesses that would be invisible in a narrower benchmark.
Dynamic and Automated Evaluation Process. The paper emphasizes that for a benchmark targeting dynamic information processing, the benchmark itself must be "live" — static evaluation pipelines are "inefficient and unable to keep up with the rapid changes in the real world." The fully automated daily cycle (event curation → agent prediction → answer acquisition → scoring) is both a technical achievement and a methodological requirement for scalability.

These principles collectively position FutureX not as an incremental improvement over prior benchmarks, but as a category-defining contribution that establishes future prediction as a legitimate evaluation paradigm. The paper is essentially arguing that the field has been missing an entire dimension of agent capability assessment, and that FutureX provides the infrastructure to fill that gap — a claim that is ambitious but grounded in the specific methodological critique of prior work laid out throughout the section.

3. Technical Approach

3.1 Reader Orientation

This paper presents FutureX, a fully automated, daily-updating evaluation pipeline that continuously generates future-prediction questions from 195 live websites, runs up to 25 different LLM agents to make predictions before the answers exist, waits for ground-truth outcomes to materialize in the real world, automatically retrieves those outcomes, and scores every agent's prior forecasts — all without manual intervention in the steady state. The problem it solves is that no existing benchmark can evaluate an LLM agent's ability to perform the kind of forward-looking, uncertainty-aware reasoning that human experts do in fields like finance, politics, and technology forecasting, because static benchmarks test memorization of known answers, retrospective benchmarks suffer from temporal contamination, and prior live forecasting benchmarks are too small, too narrow, and too easy. The "shape" of the solution is a closed-loop temporal pipeline with four daily stages — event database maintenance, question curation from templates with randomized variables, parallel agent prediction execution, and delayed answer acquisition via web crawling — orchestrated around a fundamental design constraint: the correct answers do not exist anywhere in the world at prediction time, making data contamination logically impossible by construction.

3.2 Big-Picture Architecture (Diagram in Words)

The system has four major components operating on a daily cycle, with information flowing through them sequentially but with temporal offsets:

Event Database (persistent seed store): A curated collection of 195 high-quality websites, organized by type (prediction markets, news, entertainment rankings, government data, real-time financial platforms), each associated with parameterized event templates that specify how to generate future-dated questions from that site's content. This database is maintained through an initial LLM-assisted curation pipeline followed by ongoing daily updates to remove dead sources and add new ones. Its responsibility is to serve as the stable, quality-controlled foundation from which daily question sets are generated.
Future Event Daily Curation (question generation engine): Each day, this component transforms the event database into a concrete set of 70-100 prediction questions with specific resolution dates. For prediction market websites, it crawls existing future events; for all other websites, it instantiates templates with randomized variables, then filters out harmful, subjective, and trivially easy events. Its responsibility is to produce a diverse, challenging, and temporally valid question set where no ground-truth answer yet exists.
Agent Daily Prediction (parallel evaluation harness): On each event's designated start date — typically several days before its resolution date — this component executes up to 25 different LLM agents in parallel, each given 30 minutes per question to research and submit a prediction. It captures and stores every prediction before the resolution date passes. Its responsibility is to run a fair, time-synchronized evaluation where all models see the same information environment at prediction time.
Answer Daily Acquisition and Scoring (delayed ground-truth pipeline): After each event's resolution date passes, this component crawls the source website at scheduled intervals (14:00, 16:00, 18:00, 20:00 Beijing time), extracts the ground-truth outcome using an LLM-based parser, matches it against previously stored predictions, and computes scores using event-type-specific metrics. Its responsibility is to close the evaluation loop with objective, automatically-retrieved ground truth.

The temporal architecture is what makes the system unusual: stages 2 and 3 run continuously on current events, while stage 4 operates on a one-week delay, processing events whose prediction window closed in the prior week. This asynchronous loop — predict now, score later — is the core mechanism that eliminates data contamination.

3.3 Roadmap for the Deep Dive

First, the Event Database Construction, because this is the foundation that determines what kinds of questions can be asked. I will explain the website collection and curation process, the five website types, and how templates are designed to be parameterized for daily variation.
Second, the Future Event Daily Curation pipeline, because this is where the "live" nature of the benchmark is created. I will walk through event manipulation (template instantiation, variable randomization, prediction market crawling) and event filtering (harmful detection, subjectivity removal, binary downsampling), showing how approximately 500 candidate events per week become the 70-100 selected daily.
Third, the Agent Daily Prediction harness, because this is where models interface with the benchmark. I will detail the four model categories evaluated (base LLMs, Think&Search agents, open-source deep research frameworks, closed-source deep research), the 30-minute timeout constraint, and the orchestration challenges of running 25 models daily.
Fourth, the Answer Daily Acquisition and Scoring pipeline, because this closes the loop. I will explain date filtering, scheduled website crawling, LLM-based answer extraction, failure handling, and — critically — the four distinct evaluation metrics for the four event types (single-choice, multi-choice, open-ended ranking, open-ended numerical) along with the mathematical justification for each.
Fifth, the difficulty tiering system, because this is the conceptual framework that organizes all events and enables the benchmark to discriminate between models at different capability levels. I will explain how event type and volatility combine to produce four difficulty tiers (Basic, Wide Search, Deep Search, Super Agent), what skills each tier assesses, and how tier weights (10%/20%/30%/40%) are used in the overall score.
Sixth, the evaluation architecture decisions, covering the one-week prediction window design choice, the handling of missing predictions through Monte Carlo simulation, and the evaluation delay mechanism that prevents overfitting. These are infrastructure decisions that affect the benchmark's fairness and reliability.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark construction and systems paper whose core idea is that future prediction can serve as a contamination-impossible evaluation paradigm for LLM agents if — and only if — the benchmark operates as a fully automated live pipeline that generates questions, collects predictions, and retrieves answers on a continuous daily cycle.

Event Database Construction (The Foundation)

The event database is the persistent store of high-quality information sources from which all daily questions are generated. Its construction involves a two-phase pipeline: automated website collection and curation, followed by human review — a hybrid approach the paper argues is necessary because fully automated curation misses nuanced quality signals.

Website Collection. The pipeline begins with the AIME agent, which gathers URLs from across the web targeting domains relevant to future prediction: politics, economics, finance, technology, and sports. This initial crawl produces 2,008 candidate websites. The selection is deliberately broad at this stage — the goal is to cast a wide net and filter downstream rather than risk missing valuable sources through premature pruning.

LLM-Based Initial Curation. The 2,008 candidate URLs undergo an automated quality check using a combination of Seed1.5-Thinking and DeepSeek-R1, two reasoning-capable LLMs. This stage performs three operations: (1) deduplication — removing near-identical websites or pages that cover the same content from different URLs; (2) suitability assessment — evaluating whether each website's content structure supports future-prediction question generation (e.g., does it publish rankings that update regularly? does it release scheduled economic data?); and (3) update frequency evaluation — identifying websites that publish new content on at least a weekly basis, which is a hard requirement since the benchmark operates on daily and weekly cycles. This automated pass reduces the candidate set from 2,008 to 636 websites.

Human Review. The remaining 636 URLs undergo manual inspection by human reviewers who focus on two criteria: (1) source reliability — is the website a credible, stable source of information, or does it publish speculative, unverifiable, or low-quality content? and (2) update consistency — does the website actually maintain a regular update schedule, not just claim to? The paper emphasizes that this human step targets "reliable sources—particularly ranking lists and websites with high update frequency." The output of this phase is the final event database of 195 high-quality websites.

Five Website Categories. The 195 curated websites fall into five functional categories, each supporting different types of future events:

Prediction market websites (e.g., gjopen, Polymarket): These already contain user-generated future-event questions with defined resolution criteria and binary or multiple-choice answer formats. The paper can directly crawl these events rather than generating them from templates. However, as will be discussed in the curation section, these events have quality problems that require downstream filtering.
News websites (e.g., Reuters, ESPN): These provide continuously updated reporting on current events across domains. They serve as sources for questions about upcoming developments — earnings reports, policy changes, sports outcomes, technology launches — where the event itself is scheduled but the outcome is unknown.
Entertainment ranking websites (e.g., Billboard, IMDb): These publish regularly updated rankings (music charts, movie ratings, box office figures) that follow predictable update schedules. They are ideal for template-based question generation because the ranking structure provides a natural prediction target: "What will be ranked #1 on the Billboard Hot 100 on [future date]?"
Government websites (e.g., U.S. Bureau of Economic Analysis, China Agricultural and Rural Information Site): These release scheduled economic indicators and statistical reports (GDP, unemployment, agricultural price indices) on known publication calendars. The temporal regularity makes them reliable sources for open-ended numerical prediction questions.
Real-time data platforms (e.g., Shenzhen Stock Exchange, Sina Finance): These provide live or near-live financial market data — stock prices, indices, exchange rates. They support high-frequency prediction questions where the forecast target is a specific numeric value at a specific future time.

The paper notes that it is "actively exploring more types" including corporate websites (company financial reports), e-commerce platforms, and research/educational platforms, indicating that the 195-site database is designed to be extensible.

Ongoing Database Maintenance. The event database is not static. Each day, the system removes events whose outcomes have become unavailable (e.g., websites that stopped publishing, pages that were taken down) and adds new events using the existing high-quality websites as "seeds" — meaning new event templates can be derived from previously vetted sites without re-running the full curation pipeline. The paper argues this daily maintenance "preserves the dynamic nature of FutureX and prevents potential 'cheating' by explicitly designing similar questions to train the agent, ensuring that the agent's true generalization ability is accurately reflected."

Future Event Daily Curation (Question Generation Engine)

This is the core intellectual machinery that transforms static website templates into a daily stream of novel, challenging prediction questions. The pipeline operates in two major phases: event manipulation (creating concrete questions from templates or crawling existing questions) and event filtering (removing questions that are too easy, harmful, or subjective). The paper provides a detailed numeric trace through this pipeline for the week of July 24-31, which I will follow to illustrate the process at scale.

Phase 1: Event Manipulation

The manipulation strategy differs between prediction market websites (where events already exist) and all other website types (where events must be generated from templates).

For prediction market websites: The pipeline performs a daily crawl to extract existing future-event questions. These are typically binary (yes/no) or multiple-choice in format — for example, "Will Ethereum go up or down on August 20?" or "Who will win the 2025 Tour de France King of the Mountains?" For the example week shown in Figure 3, this crawl yielded 487 events. However, the paper identifies a problem: these events are relatively easy because the answer space is constrained to a small set of options. To increase difficulty, the pipeline applies an LLM-based choice expansion step: the Seed1.5-Thinking model introduces additional random (incorrect) choices into each multiple-choice event. For instance, a question about Tour de France winners might receive plausible-sounding but incorrect rider names as distractors. The paper justifies this: "if LLM agents were to query each choice individually, it would significantly reduce efficiency" — meaning the expanded choice set forces agents to discriminate between options rather than simply verifying each candidate answer through targeted search.

For all other website types (news, entertainment rankings, government, real-time data), events are generated through a two-step process of template generation and randomization:

Template Generation. For each website, the pipeline creates one or more parameterized "event templates" — structured question skeletons that accept variables (such as a target date, a ranking position, or a stock ticker) as inputs. The construction process is semi-automated:

An LLM (the paper does not specify which model for this step, but the curation pipeline generally uses Seed1.5-Thinking and DeepSeek-R1) analyzes the content structure of each webpage and proposes candidate question templates.
For each candidate template, the system specifies the input variables needed for answer scraping — for example, a ranking-based template needs to know which date and which list to scrape.
The pipeline runs a test: it instantiates the template with a past date, scrapes the answer from the website using the automated crawling infrastructure, and has the LLM verify whether the answer was successfully retrieved. This verifies that the template is answerable.
A human reviewer examines the results and selects the most appropriate templates. Once established, a template "remains fixed within our pipeline, eliminating the need for recreation in subsequent iterations."

The paper provides two concrete template examples to illustrate the pattern:

Example 1 — Dongchedi (China's largest car review website):

Variables: rank, date, target
Template: "Which car will be ranked {rank} on the {target} board on {date} at Dongchedi?"
What this enables: By varying {target}, the pipeline can generate questions about different ranking boards (SUV Popularity, MPV Sales, Sedan Rankings); by varying {rank}, it can ask about different positions (1st place vs. 3rd place); by varying {date}, it can generate fresh questions every day.

Example 2 — Google Finance:

Variables: stock, date
Template: "What will be the highest point of {stock} on {date}?"
What this enables: By varying {stock}, the pipeline can ask about different companies (Apple, NVIDIA, Tesla); by varying {date}, it generates daily prediction targets. The paper notes this template asks for the highest point rather than closing price, which requires the agent to understand intraday price dynamics — a more granular prediction task.

Variable Randomization. With templates established, the pipeline generates concrete daily and weekly events by randomly sampling values for the template variables. The purpose, as the paper states, is "to prevent asking the same future event every day." The randomization strategy varies by website category:

For ranking websites, the system varies the target rank position (e.g., predicting 1st vs. 3rd place) and the specific ranking board or sub-list.
For government websites, it varies the requested statistic or metric (e.g., GDP growth vs. unemployment rate vs. inflation index).
For real-time data platforms, it varies the market, index, or stock ticker.
The temporal variable (date) is set to a future date — typically within a one-week window, consistent with the benchmark's evaluation delay design.

The paper provides concrete question examples to make the randomization concrete:

From the Dongchedi template:

Q1: "Which car will be ranked 1st on the SUV Popularity Ranking board on September 1st at Dongchedi?"
Q2: "Which car will be ranked 3rd on the MPV Sales Ranking board on September 15th at Dongchedi?"

From the Google Finance template:

Q1: "What will be the highest point of APPLE on September 1st?"
Q2: "What will be the highest point of NVIDIA on September 7th?"

For the example week (July 24-31), the template generation step produced 61 daily templates and 71 weekly templates. Variable randomization expanded these into 498 candidate daily and weekly events. Combined with the 487 prediction market events (which had already undergone choice expansion), the total candidate pool at this stage is 985 events.

Phase 2: Event Filtering

The raw candidate pool of 985 events contains significant quality issues that would compromise the benchmark's difficulty and reliability. The filtering phase applies three operations in sequence:

Drop Harmful and Subjective Events. The first filter removes two categories:

Harmful events involving "discrimination, hate speech, or other harmful factors." The concern is not just ethical — such events "can introduce bias or propagate misleading information, undermining the integrity of the predictions."
Subjective events that "rely on individual opinions or subjective judgment." The problem with subjective events is that "responses introduce significant variability" — there is often no objectively verifiable correct answer, making automated evaluation impossible.

The filtering mechanism uses an ensemble of three LLMs as judges: Seed1.5-Thinking, DeepSeek-R1, and Gemini-2.5-flash. Using multiple models is a design choice to improve reliability — any single model might have blind spots in identifying harmful or subjective content. The paper provides examples of dropped subjective events: "Will we win 100k tomorrow at the mara hackathon?" and "I finish Park's 'Our Nation's Path' by EOM July?" These are clearly personal or poorly-defined questions that resist objective verification.

In the example week, this filter removed 294 events (both harmful and subjective combined), reducing the prediction market events from 487 to 193 — a 60% reduction that the paper frames as evidence of how much low-quality content exists in raw prediction market data.

Downsample Yes-or-No Events. Binary (yes/no) events present a specific difficulty problem: "Events with binary choices (such as yes or no, or the outcome of a single match) are relatively easy to predict, with even random guessing achieving an accuracy of 50%." The paper cannot introduce additional random choices into yes/no questions (unlike multiple-choice questions) because there is no natural way to add plausible-sounding wrong answers to a binary distinction. Therefore, the only option is to heavily downsample these events. For the example week, the 193 remaining prediction market events were reduced to 36.

The paper is explicit that this aggressive downsampling is a deliberate design choice that distinguishes FutureX from prior benchmarks: "This ensures the high quality of FutureX and stands in sharp contrast to previous benchmarks [ForecastBench, FutureBench], where prediction market events dominate." The implication is that previous benchmarks, by accepting raw prediction market data without this filtering, inadvertently created evaluation sets where a substantial fraction of questions could be answered by random guessing — inflating apparent model performance and reducing discriminative power.

Uniform Sampling from Templates. To maintain diversity across the other website types and prevent any single template from dominating the daily question set, the final filter randomly selects one question per template per website for inclusion. This means that if a website has three templates (e.g., three different ranking boards on Dongchedi), only one question from that website appears in a given day's set. The result is 61 daily events (from daily templates) and 71 weekly events (from weekly templates), which, when the weekly events are distributed across 7 days (71/7 $\approx$ 10 per day), produces roughly 71 daily and weekly events combined. The paper reports a total of 498 open-ended events per week (61 $\times$ 7 + 71 = 498).

Final Output. After all filtering, a typical day's question set contains approximately 70-100 events, split between approximately 36 multiple-choice events (from prediction markets) and the remaining open-ended events generated from templates. Every event is associated with a resolution date — the future date on which the ground-truth answer will become available — which is used in the Answer Daily Acquisition phase. The resolution dates are typically 1-7 days in the future, consistent with the one-week prediction window.

Agent Daily Prediction (Evaluation Harness)

This component executes the actual model evaluation. Its primary engineering challenge is the daily execution constraint: because FutureX is a live benchmark with events that change every day, all models must be tested on the same day the questions are generated. The paper emphasizes this time-sensitivity:

"some events may have already yielded results, making the testing process highly time-sensitive. This stands in stark contrast to static benchmark tasks, where the questions remain fixed and can be tested at any time."

If a model misses a day's testing window, it cannot retroactively make predictions — by the time the next testing cycle runs, information available on the internet will have changed (the event may have partially resolved, new relevant information may have appeared), making any later prediction incomparable to those made by models that ran on the correct date.

Models Under Evaluation. The paper evaluates 25 models across 4 categories, selected to represent a gradient of agent capability:

Category 1: Base LLMs (8 models). These are language models without any tool-use augmentation — no web search, no code execution, no multi-step agent frameworks. They represent the "pure knowledge" baseline for how well a model's internal parameters, trained on static data, can forecast future events. The specific models are: DeepSeek-V3, DeepSeek-R1 (a reasoning model), Gemini-2.5-pro, GPT-4o-mini, GPT-4.1, Qwen3-235B, Qwen3-32B, and DouBao-Seed1.6-Thinking (the paper's internal reasoning model). An important methodological note: the authors tested GPT-o1, GPT-o3, and GPT-o4-mini as base LLMs but excluded them because "they frequently refused to make predictions." This refusal behavior — presumably due to safety training that makes these models hesitant to make definitive claims about uncertain futures — is itself an interesting finding about how current alignment techniques interact with forecasting tasks.

Category 2: Agentic LLM with Thinking and Searching (7 models). These are commercial closed-source models that natively integrate reasoning and web search capabilities — the user submits a query and the model autonomously decides when and how to search the internet, then synthesizes retrieved information with its internal knowledge. All seven models are commercial offerings: GPT-o4-mini (Think&Search), GPT-4o (Think&Search), Hunyuan (Think&Search), DeepSeek (Think&Search), Qwen3-235B (Think&Search), Grok-4 (Think&Search), and Doubao (Think&Search). The paper notes that Gemini-2.5-pro (Think&Search) evaluation is "on the way" but not yet completed. This category tests whether integrated search-and-reasoning provides measurable advantages over base LLMs for future prediction, and the results (discussed in later sections of the paper) suggest it does significantly.

Category 3: Open-source Deep Research Agents (8 model-framework combinations). These are publicly available multi-agent frameworks that orchestrate complex research workflows: they generate a research plan, delegate subtasks to specialized agents, make multiple iterative tool calls, and synthesize findings. The paper evaluates two frameworks:

SmolAgent from Huggingface: evaluated with 6 different backbone LLMs — Gemini-2.5-pro, GPT-4.1, GPT-4o-mini, Qwen3-235B, Qwen3-32B, and DeepSeek-V3. The paper explicitly notes that reasoning models (DouBao-Seed1.6-Thinking, GPT-o3, GPT-o4-mini, DeepSeek-R1) are excluded from SmolAgent evaluation because they "incur significantly longer runtimes" — a practical constraint that reveals a tension between model capability and evaluation feasibility.
AgentOrchestra from Skywork AI: evaluated with only 2 backbone LLMs — Gemini-2.5-pro and GPT-4.1. The limitation to two models is due to "the complexity of this agent framework and limited compatibility." Furthermore, AgentOrchestra is so computationally intensive that the paper evaluates it only on Level 1 and Level 2 events (the easier tiers), and it is excluded from the overall leaderboard for this reason.

Category 4: Closed-source Deep Research Models (2 models). These are proprietary commercial research agents: Doubao Deep Research and Gemini Deep Research (running Gemini-2.5-flash as the backend for efficiency; the paper notes that Gemini-2.5-pro Deep Research will be integrated later). The paper explicitly acknowledges coverage gaps due to API access limitations: "Due to policy and API stability issues, we are unable to test the GPT Deep Research and Claude models." This is a practical limitation of evaluating against commercial systems — not all frontier models are accessible for large-scale automated benchmarking.

Automated Testing Pipeline Architecture. The execution infrastructure has several design requirements that the paper makes explicit:

High-performance multi-process framework: All models must be triggered on a daily schedule. The paper implements parallel execution to handle 25 models simultaneously rather than sequentially, which would be infeasible given the 30-minute per-question timeout.
Strict error handling: If any single model or question fails (due to API timeout, refusal to answer, or crash), the failure must not block the rest of the pipeline. The paper implements "strict error handling to prevent failures from blocking the pipeline" but does not specify the exact mechanism (e.g., try-catch per model, graceful degradation to missing prediction).
30-minute maximum per question: Each model is given at most 30 minutes to complete its prediction for a single question. This timeout serves multiple purposes: it prevents any single slow model from delaying the entire daily cycle, it enables comparison of efficiency (the paper later analyzes how many searches different models perform within this time budget), and it reflects a realistic deployment constraint where predictions must be timely.
70-100 events per day: After event filtering, the daily question set contains 70-100 events. Each of the 25 models must process all of them (since alignment is not enforced — missing predictions are handled statistically rather than by dropping events). At 30 minutes per question, this implies a theoretical maximum of 6 hours if all models ran serially on 12 questions. With parallelization, the paper does not specify the exact wall-clock time but implies it completes within the daily cycle.

The pipeline stores every model's prediction for every event in a persistent database indexed by event ID, model ID, and prediction date. These stored predictions are matched against ground-truth answers in the Answer Daily Acquisition phase, typically one week later.

Answer Daily Acquisition and Scoring (Closing the Loop)

This component solves the unique evaluation challenge of a live forecasting benchmark: answers do not exist at prediction time, so they must be retrieved later — automatically, at scale, and with high reliability. The paper reports that achieving this was one of the most engineering-intensive aspects of building FutureX, requiring careful source selection, scheduled crawling, and iterative failure handling.

Date Filtering. The first daily step is to identify which events are eligible for answer retrieval. The pipeline selects events whose resolution date matches the current day. To handle cases where websites have publishing delays (e.g., an economic indicator scheduled for release on July 20 might not appear on the website until July 21), the system also includes unresolved events from previous days. The paper notes that all timestamps are standardized to Beijing time (UTC+8) to ensure consistent scheduling, since the source websites span global time zones.

Scheduled Website Crawling. The core challenge is that different websites update at different times of day, and a single crawl-at-noon might miss afternoon publications. To maximize retrieval success, the pipeline performs scheduled crawls at 14:00, 16:00, 18:00, and 20:00 each day — four attempts spread across the afternoon and evening Beijing time. Each crawl extracts the core content from the target webpage. The paper does not specify the crawling infrastructure (headless browser vs. HTTP requests, JavaScript rendering strategy), but the multi-attempt design suggests that the system treats website availability as probabilistic and compensates through redundancy.

LLM-Based Answer Extraction. Once the core content is retrieved, the Seed1.5-Thinking model extracts the specific answer value. The extraction prompt includes three inputs to improve accuracy: (1) the original question text (so the model knows what it's looking for), (2) the resolution date (to disambiguate when multiple dates' worth of data appear on the same page), and (3) the raw crawled content. The paper notes that this design was refined through iterative debugging: "various types of failure cases arise" during initial deployment, requiring manual review and customized prompt tuning.

Failure Handling and Iteration. The paper categorizes failures into two types and describes systematic responses:

Crawling errors — typically caused by anti-bot measures (CAPTCHAs, IP blocking, dynamic content loading that the crawler cannot handle). When a website proves consistently uncrawlable, the paper's response is to "update our event database to exclude websites that are no longer accessible or reliably crawlable." This means the event database naturally shrinks toward sources that are amenable to automated retrieval — potentially introducing a survivorship bias toward more crawlable (and possibly more structured, less dynamic) websites.
Extraction errors — cases where the LLM extracts the wrong value or fails to find an answer in the crawled content. The response is manual review and prompt engineering: "we manually review the cases and design customized prompts to improve answer extraction accuracy."

With these iterative improvements, the paper reports that in the stable online version, the answer acquisition success rate exceeds 97%. This high reliability is what makes the fully automated pipeline feasible — if the answer rate were substantially lower, the benchmark would lose too many events to be a reliable evaluation instrument.

Evaluation Metrics (Event-Type-Specific Scoring)

FutureX's four event types require four different scoring functions, each designed to be appropriate for the answer structure and prediction difficulty of that event type. The paper provides explicit mathematical formulations and justifications for each.

Single-Choice Events. For questions with exactly one correct answer from a set of options:

$\text{score}(Y, \hat{Y}) = \mathbb{I}(Y = \hat{Y})$

where $\mathbb{I}(\cdot)$ is the indicator function (1 if the condition is true, 0 otherwise), $Y$ is the ground-truth correct option, and $\hat{Y}$ is the model's predicted option.

What it computes: exact-match accuracy — the model gets full credit (1) for the correct answer and zero credit for any wrong answer.

Why this form: single-choice questions have a discrete, well-defined correct answer — either the prediction matches or it does not. Partial credit makes no sense because the answer space is categorical. This is the standard metric for multiple-choice evaluation and needs no further justification. Note that for yes/no binary events (which are a subset of single-choice), the random baseline is 50%, which is why the paper aggressively downsamples these events.

Multi-Choice Events. For questions where multiple options can be correct simultaneously (e.g., "which teams will qualify for the playoffs?" with a correct answer set of {Team A, Team B, Team C}):

$\text{score}(\mathcal{Y}, \hat{\mathcal{Y}}) = \text{F1-Score}(\mathcal{Y}, \hat{\mathcal{Y}})$

where $\mathcal{Y}$ is the set of ground-truth correct options and $\hat{\mathcal{Y}}$ is the set of model-predicted options.

What it computes: the harmonic mean of precision and recall — precision measures what fraction of predicted options are correct (penalizing over-prediction), recall measures what fraction of correct options were predicted (penalizing under-prediction), and the F1-score balances both. The paper states the operational consequence: "leaving out any true answer cuts the item's score in half, whereas selecting even one wrong option reduces the score to zero." This is because selecting a wrong option sets precision to less than 1.0, and since precision enters the F1 numerator, the score drops sharply — though not necessarily to exactly zero unless precision is zero. The precise behavior depends on the specific precision and recall values.

Why this form: the F1-score is the standard metric for multi-label classification where both false positives (predicting an option that isn't correct) and false negatives (missing an option that is correct) matter. Using simple accuracy ( $\mathcal{Y} = \hat{\mathcal{Y}}$ ) would be too harsh — a model that gets 3 of 4 correct options but misses one would receive zero credit, which fails to differentiate between nearly-correct and completely-wrong predictions. Using Jaccard similarity (intersection over union) would also be possible but would weight precision and recall equally without the harmonic mean's property of being dominated by the smaller value, which the paper seems to want for penalizing either type of error heavily.

Open-Ended Ranking Events. For questions requiring prediction of an ordered top- $k$ list (e.g., "predict the top 5 movies by box office on [date]"):

$\text{score}(\{y_1, \dots, y_k\}, \{\hat{y}_1, \dots, \hat{y}_k\}) = \begin{cases} 1, & \text{if } y_i = \hat{y}_i \text{ for } i = 1, \dots, k \\\ 0.8 \times \frac{|\{y_1, \dots, y_k\} \cap \{\hat{y}_1, \dots, \hat{y}_k\}|}{k}, & \text{otherwise,} \end{cases}$

where $\{y_1, \dots, y_k\}$ is the ground-truth ordered list, $\{\hat{y}_1, \dots, \hat{y}_k\}$ is the predicted ordered list (not necessarily in the correct order), $|\cdot|$ denotes set cardinality, $\cap$ denotes set intersection, and $k$ is the length of the list.

What it computes: if the prediction matches the ground truth perfectly in both content and order, the score is 1.0. If the sets match (same items) but the order is wrong, the score is 0.8 (the overlap fraction multiplied by the partial-credit factor). If the sets partially overlap — e.g., the model predicted 3 of 5 correct items — the score is $0.8 \times (3/5) = 0.48$ . The 0.8 multiplier caps the maximum partial-credit score; even if a model gets all the right items but in the wrong order, it receives 0.8 rather than 1.0.

Why this form: the paper needed a metric that rewards getting the right items (content accuracy) while also rewarding getting the right order (ranking accuracy). The two-case structure provides full credit for perfect ranking but downgrades to a content-only score when the order is imperfect. The 0.8 factor is a design choice reflecting that correctly identifying items is important (80% credit) but correctly ordering them adds additional value (20% credit). Alternative designs considered would be rank-aware metrics like Normalized Discounted Cumulative Gain (NDCG) or Kendall's tau, but these would require position-specific weights that might be harder to justify across heterogeneous ranking domains. The set-overlap approach is simpler and more interpretable, though it sacrifices fine-grained position sensitivity (a prediction that swaps positions 1 and 2 is penalized identically to one that swaps positions 1 and 5, since the set overlap is the same).

Open-Ended Numerical Prediction Events. For questions requiring prediction of a specific numeric value (e.g., a stock price, an economic index value):

$\text{score}(Y, \hat{Y}) = \max \left( 0, 1 - \left( \frac{Y - \hat{Y}}{\sigma(Y)} \right)^2 \right)$

where $Y$ is the ground-truth numeric outcome, $\hat{Y}$ is the model's predicted value, and $\sigma(Y)$ is the standard deviation of the outcome over the past 7 days, computed from historical data of the same metric.

What it computes: the prediction error $Y - \hat{Y}$ is normalized by the recent volatility $\sigma(Y)$ to produce a standardized error. This ratio is squared and subtracted from 1. Predictions within one standard deviation ( $|Y - \hat{Y}| \leq \sigma(Y)$ ) receive positive scores — for example, an error of exactly $\sigma(Y)$ yields $1 - 1^2 = 0$ , while an error of $0.5\sigma(Y)$ yields $1 - 0.25 = 0.75$ . Predictions beyond one standard deviation receive a score of zero (due to the $\max(0, \cdot)$ clamp), regardless of how far off they are (a prediction 10 standard deviations away still gets zero, not a large negative number).

Why this form: the core design insight is that prediction difficulty should be measured relative to the metric's inherent unpredictability — forecasting the exact value of a highly volatile stock price is fundamentally harder than forecasting a stable government bond yield, and the scoring should reflect this. The recent 7-day standard deviation serves as a local volatility estimate: if the metric has been fluctuating wildly recently, a larger absolute error is tolerated. The quadratic penalty $(Y - \hat{Y})^2$ means that small errors within the tolerance band receive substantial partial credit, but errors beyond $\sigma(Y)$ are treated as complete misses. The max-with-zero clamp ensures that pathological predictions (wildly wrong guesses) don't produce large negative scores that could dominate the average — they are simply treated as failures (score 0). The one-standard-deviation threshold is a common statistical convention (roughly 68% of normally-distributed values fall within $\pm 1\sigma$ ), though the paper does not assume normality and uses it as a practical scaling factor. An alternative like Mean Absolute Percentage Error (MAPE) would penalize all errors linearly without a "reasonable range" concept, making highly volatile metrics dominate the scoring. The paper does not discuss why 7 days was chosen over other lookback windows (14 days, 30 days) — this is presumably an empirical choice balancing responsiveness to changing volatility against estimation stability.

Overall Score Aggregation. The final overall score for each model combines results across the four difficulty tiers using fixed weights:

$\text{Overall Score} = 0.10 \times \text{Score}_{\text{Level 1}} + 0.20 \times \text{Score}_{\text{Level 2}} + 0.30 \times \text{Score}_{\text{Level 3}} + 0.40 \times \text{Score}_{\text{Level 4}}$

The increasing weights (10%, 20%, 30%, 40%) reflect the paper's judgment that harder tiers should contribute more to the overall score. The justification, as discussed in Section 4.2 of the paper, is that Level 1 and Level 2 events are "not sufficiently challenging to distinguish between models of varying capabilities" — most models score reasonably well on them, so they contribute little discriminative power. Level 3 and especially Level 4 events are where models differentiate, so they receive more weight. The specific weights (10/20/30/40) appear to be design choices rather than optimized values; the paper does not report sensitivity analysis to alternative weightings.

Difficulty Tiering System (Conceptual Organization)

The difficulty tiering system classifies every FutureX event into one of four levels based on two factors: event type (single-choice, multi-choice, open-ended) and expected volatility (low vs. high for open-ended events). The tiers are designed to assess progressively more demanding agent capabilities, and the paper explicitly maps each tier to specific assessed skills in planning, reasoning, and searching (Table 3).

Level 1: Basic Tier. Single-choice events with fewer than 4 options. The defining characteristic is that the answer space is small and predefined — the agent does not need to search broadly or reason deeply; it needs to identify the most probable option among a few candidates. The assessed skills are "Weak" across planning, reasoning, and searching because the task can be solved through straightforward knowledge retrieval or simple elimination. The paper notes that after downsampling yes/no events, Level 1 events constitute the smallest fraction of the benchmark.

Level 2: Wide Search Tier. Multi-choice events with several correct answers. The defining characteristic is the requirement for exhaustive yet precise discrimination: the agent must identify all correct options and only correct options, with no tolerance for false positives. The assessed skills are "Weak" for planning (the search strategy is straightforward), "Medium" for reasoning (must discriminate between similar options), and "Medium" for searching (must verify each candidate).

Level 3: Deep Search Tier. Open-ended events with low volatility — the underlying facts are relatively stable over time. The defining characteristic is that no options are provided; the agent must propose its own answer by searching for and synthesizing information. Because volatility is low, exhaustive information collection should converge on the correct response — the task tests whether the agent can navigate multiple sources, integrate evidence, and produce a coherent prediction. The assessed skills are "Medium" for planning (must decide what to search for), "Medium" for reasoning (must synthesize evidence), and "Strong" for searching (must perform multi-step, interactive information retrieval including clicking, scrolling, and filtering on real websites).

Level 4: Super Agent Tier. Open-ended events with high volatility — the target values fluctuate significantly and unpredictably. The defining characteristic is that even exhaustive search cannot guarantee a correct answer because the underlying process is inherently uncertain. The agent must "cast a wide net for information and reason probabilistically under shifting signals and deep uncertainty." The assessed skills are "Strong" across all dimensions: planning (complex multi-step strategy), reasoning (probabilistic synthesis under uncertainty), and searching (wide-scope information gathering across diverse sources). The paper frames this tier as "taxing even for human experts—let alone machines—because the scenarios are complex, ambiguous, and resist simple fact retrieval."

Volatility Determination for Open-Ended Events. The paper does not provide a detailed methodology for computing volatility, but the concept is described as measuring "how much the target outcome is expected to fluctuate over time" based on historical data. Open-ended events are tagged as either "Low Volatility" or "High Volatility," which determines whether they fall into Level 3 or Level 4. The paper explicitly states that events with extremely low volatility — such as all-time box office rankings that remain virtually unchanged — are excluded from the benchmark entirely because they "offer little predictive challenge."

Tier Weighting Rationale. The overall score weights (10/20/30/40) reflect a deliberate de-emphasis of easier tiers and emphasis on harder ones. The paper's experimental results (Section 4.2) validate this choice: they find that "Level 1 and Level 2 events are not sufficiently challenging to distinguish between models of varying capabilities" and that base LLMs without search tools can already achieve high accuracy on these tiers. The heavier weight on Levels 3 and 4 ensures that the overall leaderboard ranking primarily reflects performance on tasks that genuinely discriminate between model capabilities.

Evaluation Architecture Decisions (Infrastructure Choices)

Several infrastructure-level design decisions affect the benchmark's fairness, reliability, and resistance to overfitting. The paper discusses these explicitly.

One-Week Prediction Window. The benchmark asks models to predict events with resolution dates up to one week in the future, not just next-day events. The paper frames this as a tradeoff:

"A longer delay allows the benchmark to include more events each day (e.g., many events may resolve within 100 days), but our evaluation pipeline faces more pressure... Conversely, a shorter delay improves timeliness (e.g., predicting only next-day events), but significantly reduces the number and diversity of events available, while also increasing the randomness of daily evaluations."

The one-week window is presented as the equilibrium point that provides "sufficient event coverage and manageable evaluation latency." The paper also argues this design "better reflects real-world use cases, where agents must make judgments about a broader horizon of future events, not just what will happen tomorrow." This is a methodological claim: forecasting skill at a one-week horizon is more representative of professional analyst capabilities than next-day prediction, which might be dominated by short-term trend extrapolation.

Handling Missing Predictions. Because the pipeline runs 25 models daily, each making predictions on 70-100 events, missing predictions are inevitable — API failures, model timeouts, and refusals to predict certain events all contribute. The paper explicitly declines to align events across models by dropping questions that any model missed, arguing that "full alignment would substantially reduce the total number of events, which is undesired." Instead, they use a Monte Carlo simulation to quantify how much missing predictions affect score reliability.

The simulation works as follows: given approximately 500 events per week, they first sample 500 events (with both results and predictions) and calculate the "true" average score $s_i$ . They then randomly drop $\kappa\%$ of these events (simulating missing predictions), compute the "pseudo" average score $\hat{s}_i$ from the remaining events, and repeat this process 20,000 times for each missing rate $\kappa$ from 1% to 20%. The standard deviation of the pseudo scores $\text{Std}(\hat{S})$ is plotted against the missing rate (Figure 7). The paper reports that "the standard deviation remains relatively small" and that "as FutureX progresses and the test size grows, the standard deviation is expected to decrease at a rate proportional to the inverse square root of the sample size." Based on this analysis, they conclude that "given the complexity of the auto-evaluation pipeline and the relatively minor impact of missing predictions, we prioritize increasing the test sample size over strict alignment."

This is a pragmatic choice: statistical noise from non-overlapping event sets is smaller than the noise that would be introduced by drastically reducing the sample size through alignment. The paper quantifies the relative increase in standard deviation in Figure 16 (referenced but not reproduced in the main text).

Evaluation Delay and Overfitting Prevention. Because answers do not exist at prediction time, the evaluation has an inherent delay — predictions made on a Monday for events resolving throughout the week can only be scored after the last resolution date passes, roughly one week later. The paper notes a beneficial side effect of this delay: "it helps prevent overfitting to our live benchmark through frequent submissions. Since performance scores are not available immediately after predictions, model developers cannot directly optimize against recent feedback." This means that unlike static benchmarks where a developer can iterate rapidly by testing against known answers, FutureX imposes a minimum one-week feedback cycle, making it "more robust and reliable as an evaluation framework" against gaming through repeated submission.

Daily Update Cadence. The entire pipeline operates on a 24-hour cycle: new questions are generated each day, all models are tested on that day's set, and the answer acquisition phase checks for newly resolved events. The event database receives daily updates to remove dead sources and add new events. This daily rhythm is what makes FutureX "live" in the sense that it continuously tracks the evolving real-world information environment, unlike benchmarks with monthly or one-time update cycles.

4. Key Insights and Innovations

Innovation 1: Contamination-Impossible-by-Design as a New Evaluation Paradigm

The dominant approach to addressing data contamination in LLM evaluation has been detection and mitigation: researchers develop post-hoc tests to identify whether specific benchmark questions appeared in training data, use canary strings to track memorization, or construct benchmarks from newly-released data sources that post-date model training. These approaches are inherently reactive — they assume contamination has already occurred and attempt to measure or patch around it. FutureX proposes something categorically different: an evaluation where contamination is logically impossible, not just statistically unlikely.

The mechanism is the temporal structure of the task itself. By asking models to predict events whose outcomes have not yet occurred, FutureX guarantees that the correct answer literally does not exist in any training corpus, anywhere in the world, at the moment of prediction. This is not a stronger version of the detection paradigm — it is an entirely different strategy that moves from "how do we know if the model memorized this?" to "memorization of the answer is physically impossible." The paper articulates this as "being 'contamination-impossible by design,' which creates an absolutely fair evaluation environment" — a phrase that captures the shift from the probabilistic to the logical.

This innovation has significance beyond the specific domain of future prediction. It demonstrates that the structure of an evaluation task can be engineered to provide contamination guarantees through its temporal relationship to ground truth, rather than through dataset construction tricks (using only recently published sources) or post-hoc detection. This opens a design space for benchmarks in other domains where ground truth emerges over time — real-time question answering, breaking news summarization, live event commentary — that could adopt the same "predict now, score later" architecture. The concept of evaluation delay as a contamination prevention mechanism rather than an inconvenience is a genuinely new idea in benchmark design.

The contrast with prior work makes the novelty clear. LiveBench and LiveCodeBench, which the paper cites as predecessors, refresh their question sets to avoid memorization, but their questions still have answers that exist in the world at test time — a model with sufficiently aggressive web crawling during training could potentially encounter them. ForecastBench, the closest prior work on future prediction, also tests on future events, but its reliance on prediction market questions (which are binary or multiple-choice) means that random guessing achieves non-trivial accuracy, limiting the benchmark's ability to distinguish between models that are genuinely forecasting and those that are guessing. FutureX's innovation is the combination of logical impossibility of contamination with sufficient task complexity that guessing is not a viable strategy — enabled by the open-ended question types that dominate the benchmark.

The evidence for this innovation is structural rather than empirical: it is a property of the benchmark's design, not a result that needs experimental validation. The paper supports the claim by walking through the logical argument (answers don't exist → can't be in training data → can't be memorized) and by demonstrating through the benchmark's two-week operation that the pipeline can actually sustain daily question generation at scale with 97% answer retrieval — proving that the "impossible-by-design" architecture is not just theoretically sound but operationally feasible.

Innovation 2: Difficulty Tiering as a Diagnostic Framework for Agent Capability, Not Just Task Categorization

Most benchmarks partition their test sets by domain (MMLU has 57 subjects) or by superficial difficulty labels (GAIA has three levels). These partitions serve organizational purposes — they tell you which subjects a model is good at — but they don't constitute a diagnostic framework that systematically maps task properties to assessed cognitive skills. FutureX's four-tier system (Basic, Wide Search, Deep Search, Super Agent) is qualitatively different because each tier is explicitly defined by which agent capabilities are required to succeed — planning depth, reasoning complexity, and search sophistication — rather than by subject matter or an author-assigned difficulty rating.

The innovation here is conceptual, not mechanical. The paper doesn't just say "Level 4 is harder than Level 3." It specifies that the jump from Level 3 to Level 4 represents a transition from tasks where exhaustive information gathering can in principle converge on the correct answer (low-volatility, stable underlying facts) to tasks where no amount of search can eliminate uncertainty because the ground truth itself is inherently volatile. This distinction — between solvable-through-search and requiring-probabilistic-reasoning — is a genuinely diagnostic axis that isolates a specific cognitive capability (reasoning under deep uncertainty) from mere search proficiency. A model that performs well on Level 3 but collapses on Level 4 is demonstrating a specific and interpretable failure mode: it can retrieve and synthesize information, but it cannot reason probabilistically when that information is inherently noisy and contradictory.

This diagnostic value is empirically validated by the paper's results. Base LLMs achieve high accuracy on Level 1 and Level 2 events (validating that these tiers test knowledge retrieval rather than search sophistication), while search-augmented models pull ahead on Level 3 (validating that this tier tests the ability to gather and synthesize information that isn't in pre-training data). Most tellingly, even the best models — including Grok-4 and commercial Deep Research systems — struggle severely on Level 4 (Figure 8d), and the human baseline shows a dramatic advantage precisely at this tier. This pattern confirms that the difficulty tiering is not merely a relative ordering but a diagnostic instrument that reveals where in the capability stack current models break down.

The contrast with prior forecasting benchmarks sharpens the innovation. ForecastBench, the most comparable prior effort, collects events from prediction markets and databases without any tiering system — all questions contribute equally to the aggregate score. This means a model could achieve a high rank by excelling at binary-choice questions (which represent a large fraction of prediction market events) while being incapable of open-ended forecasting. FutureX's tiering system prevents this by design: the overall score weights (10%/20%/30%/40%) increasingly emphasize the harder tiers where binary-choice questions are rare (after aggressive downsampling) and open-ended questions dominate. The tiering thus serves dual purposes — diagnosis of specific capability gaps and construction of an aggregate metric that reflects what the benchmark actually cares about measuring.

This innovation is incremental in its mechanism (binning by event type and volatility is straightforward) but fundamental in its conceptual contribution: it provides a vocabulary and framework for discussing agent forecasting capability that the field previously lacked. The terms "Wide Search," "Deep Search," and "Super Agent" are not just labels — they name specific capability thresholds that future research can target, and the mapping in Table 3 (Assessed Agent's Skills: Planning, Reasoning, Searching) provides a rubric for evaluating whether new models have crossed those thresholds.

Innovation 3: The Discovery That Refusal Behavior Is a First-Order Barrier to Forecasting Evaluation

One of the paper's most striking empirical findings is not about model performance but about model participation: GPT-o1, GPT-o3, and GPT-o4-mini — some of the most capable reasoning models available — "frequently refused to make predictions" when evaluated as base LLMs, and were consequently excluded from the results. This is not a minor implementation detail. It is a finding with significant implications for the alignment community and for anyone attempting to deploy frontier models in forecasting contexts.

The significance lies in what this refusal behavior reveals about current alignment strategies. These models were presumably trained to be cautious about making definitive claims, to express uncertainty, and to avoid generating harmful or misleading predictions — all reasonable safety objectives. But in a forecasting context, the task is to make a definitive claim under uncertainty, and the appropriate safety behavior is to provide a calibrated probabilistic prediction with supporting reasoning, not to refuse to engage. The refusal pattern suggests that current alignment techniques have produced models that cannot distinguish between "I should not make harmful claims" and "I should not make uncertain predictions about verifiable future events" — a failure of nuance that renders them unusable for an entire class of legitimate and valuable applications.

This finding connects to broader concerns about over-refusal in aligned models, but with a specific twist: the refusal is not triggered by obviously harmful content (the paper's pipeline already filters harmful and subjective events), but by the inherent uncertainty of the prediction task. The models appear to treat "I might be wrong" as equivalent to "I should not answer," which is a failure mode that prior work on refusal behavior (focused primarily on toxicity, stereotypes, and dangerous knowledge) has not systematically documented.

The paper does not deeply analyze this refusal behavior — it simply reports it as a constraint that shaped model selection — but the finding itself is an innovation because it identifies a capability-safety tension specific to forecasting: models that are safe enough to deploy may be too cautious to forecast, and models that are willing to forecast may require safety compromises. This tension has not been articulated in prior work on either alignment or forecasting benchmarks. It suggests that future work on alignment for LLM agents must explicitly consider forecasting as a legitimate use case and develop safety training that permits calibrated probabilistic prediction while still preventing genuinely harmful content.

The evidence is qualitative but striking: three frontier models, all from the same model family (GPT-o-series), were excluded from a major evaluation because they wouldn't perform the task. The paper does not provide refusal rates or analyze what fraction of questions triggered refusal versus what fraction of models exhibited refusal, but the fact that exclusion was necessary for three of the eight base LLM candidates is itself the key empirical signal.

Innovation 4: A Decomposition of Agent Capability into Search Quantity, Search Quality, and Planning Sophistication as Independent Factors

Section 4.5 of the paper, which presents focused case studies on prediction timing, planning analysis, and search behavior, introduces a conceptual decomposition of agent capability that goes beyond the standard "model X outperforms model Y" reporting. The decomposition identifies three partially independent drivers of forecasting performance:

Search quantity (how many web queries the agent makes — operationalized as average search count per question, measured in Figure 14)
Search quality (how effectively the agent discriminates between reliable and unreliable sources — operationalized as the Source Reliability score in the planning analysis, Table 5)
Planning sophistication (how comprehensive, specific, and actionable the agent's research strategy is — operationalized as the Comprehensiveness and Plan Actionability scores in Table 5)

The innovation is not the mere observation that these factors matter — that much is obvious — but the empirical demonstration that they are partially decoupled and that failure in any one dimension can limit overall performance even when the others are strong. This is visible in the case study results:

Hunyuan demonstrates strong search quantity (Figure 14: high search counts) and strong past-prediction performance (Figure 12: dark bars), but its future-prediction performance lags significantly — suggesting that search quality and planning sophistication, not search quantity, are its bottleneck.
SmolAgent with Gemini-2.5-pro shows substantial performance improvement over SmolAgent with other backbones (Figure 12), and the planning analysis (Table 5) reveals that Gemini-2.5-pro generates plans with high scores on all three dimensions (Comprehensiveness: 10, Source Reliability: 10, Plan Actionability: 10), while the same agent framework with GPT-4o-mini generates plans with low scores (5, 2, 4) — suggesting that the agent framework's search infrastructure is adequate when paired with a capable planner.
DeepSeek-v3 in the SmolAgent framework receives high planning comprehensiveness (9) and source reliability (8) scores but a Plan Actionability score of 0 because it "reaches a conclusion within the planning phase itself" rather than executing the planned search — demonstrating that even good plans are useless if the model doesn't follow through with actual tool use.

This decomposition matters because it provides a diagnostic toolkit for future research. Rather than treating "forecasting performance" as a monolithic capability to be improved, researchers can now ask: is my model's weakness in generating enough search queries? In discriminating source quality? In formulating specific, actionable research plans? The regression analysis in Figure 13 provides quantitative support: the number of tool calls, source reliability, and plan comprehensiveness all show significant positive coefficients, while main agent think length (a proxy for redundant dialogue) shows a negative coefficient. The R² of 0.518 indicates that these factors explain roughly half the variance in SmolAgent's performance — substantial but not complete, leaving room for other factors not captured by this decomposition.

The comparison with prior work clarifies the innovation. Most agent benchmarks report aggregate scores and perhaps per-domain breakdowns, but do not instrument the agent's internal planning and retrieval process to identify where breakdowns occur. GAIA reports success rates, SWE-bench reports patch correctness, but neither tells you whether the model's failure was due to poor search strategy, poor source selection, or poor synthesis of retrieved information. FutureX's planning and search analyses (Tables 5, Figure 13, Figure 14) represent a move toward process-level diagnostics — understanding not just whether an agent succeeded, but what pattern of internal behavior led to success or failure. This is methodologically significant because it transforms the benchmark from a pure scoring instrument into a research tool that can guide model improvement.

The innovation is incremental in its execution (the planning analysis uses a simple LLM-as-judge scoring with three dimensions) but fundamental in its orientation: it treats the agent's internal decision-making as an object of study, not a black box to be evaluated only by its outputs. For the forecasting domain specifically, where process quality (did you look at the right sources? did you consider relevant factors?) is arguably as important as outcome accuracy, this process-level perspective is particularly valuable.

5. Experimental Analysis

Evaluation Methodology

Dataset. FutureX operates as a live benchmark, not a static dataset. The experiments cover a two-week evaluation window from July 20 to August 3, 2025, during which the automated pipeline generated and evaluated predictions on 1,272 events across 11 domains (politics, sports, crypto, culture, finance, business, technology, weather, health, space, and others). Events are sourced from 195 high-quality websites curated from an initial pool of 2,008 candidates, spanning five website types: prediction markets, news outlets, entertainment rankings, government data portals, and real-time financial platforms. The event distribution across the four difficulty tiers is shown in Figure 6: after aggressive downsampling of binary yes/no events, the remaining three levels are relatively balanced.
Base models. The evaluation covers 25 models across 4 categories: (1) 8 base LLMs without tool use — DeepSeek-V3, DeepSeek-R1, Gemini-2.5-pro, GPT-4o-mini, GPT-4.1, Qwen3-235B, Qwen3-32B, and DouBao-Seed1.6-Thinking; (2) 7 closed-source Think&Search agents — GPT-o4-mini, GPT-4o, Hunyuan, DeepSeek, Qwen3-235B, Grok-4, and Doubao (all with integrated reasoning and web search); (3) 8 open-source deep research agent configurations — SmolAgent with 6 backbone LLMs (Gemini-2.5-pro, GPT-4.1, GPT-4o-mini, Qwen3-235B, Qwen3-32B, DeepSeek-V3) and AgentOrchestra with 2 backbones (Gemini-2.5-pro, GPT-4.1); (4) 2 closed-source Deep Research models — Doubao Deep Research and Gemini Deep Research (running Gemini-2.5-flash). GPT-o1, GPT-o3, and GPT-o4-mini as base LLMs were excluded because they "frequently refused to make predictions." AgentOrchestra was evaluated only on Level 1 and Level 2 events due to computational cost and is excluded from overall scores.
Metrics. The benchmark uses four event-type-specific scoring functions, described in Section 3.4.3: (1) single-choice events use exact-match accuracy score = I(Y = Ŷ); (2) multi-choice events use the F1-score between predicted and ground-truth option sets, where "leaving out any true answer cuts the item's score in half, whereas selecting even one wrong option reduces the score to zero"; (3) open-ended ranking events use a two-case metric — 1.0 for perfect order-and-content match, and 0.8 × (|intersection| / k) for partial credit when the order is wrong but items overlap; (4) open-ended numerical events use score = max(0, 1 − ((Y − Ŷ) / σ(Y))²), where σ(Y) is the 7-day historical standard deviation, meaning predictions within one standard deviation receive positive partial credit and those beyond receive zero. The overall score aggregates across the four difficulty tiers with weights of 10%, 20%, 30%, and 40% respectively, placing progressively more emphasis on harder tiers.
Baselines. The paper does not designate a single "baseline" method; rather, the four model categories serve as natural performance strata: base LLMs represent the "no tools" lower bound, Think&Search agents represent integrated retrieval-augmented reasoning, and deep research agents (both open-source and closed-source) represent the current frontier of autonomous multi-step research. The human baseline consists of 40 industry experts (current or former employees from Big Four accounting firms, top consulting firms, and nine leading investment banks) who independently answered 300 randomly sampled questions using the same evaluation metrics.
Generation budget / compute accounting. Each model is given a maximum of 30 minutes per question to produce a prediction. The paper does not measure compute in FLOPs or tokens; instead, it reports search count as a proxy for retrieval effort (Figure 14: average number of web queries per question for models where this is measurable). The paper explicitly notes that models like GPT-o4-mini and GPT-4o do not expose search counts, so they are excluded from that particular analysis. The 30-minute timeout serves as the primary resource constraint, and the paper reports that Grok-4 achieves its top performance while "completing searches in less than 5 minutes, compared to approximately 30 minutes for Gemini deep research" — suggesting substantial efficiency variation across models.
Statistical protocol. The paper handles missing predictions (due to API failures, refusals, or timeouts) by declining to align events across models — each model's score is computed on the events it actually answered. To quantify the impact of this choice, a Monte Carlo simulation (Section 3.4.2) samples 500 events with replacement, computes "true" scores, randomly drops κ% of events for κ ranging from 1% to 20%, recalculates "pseudo" scores, and repeats 20,000 times to estimate the standard deviation introduced by non-overlapping event sets. The paper reports that "the standard deviation remains relatively small" (Figure 7) and that it decreases proportional to the inverse square root of sample size as more data accumulates. For the factor analysis (Section 4.4), linear regression is performed with statistical significance indicated at p < 0.005. The two-week evaluation window (1,272 events) is acknowledged as limited; the benchmark is designed to accumulate more data over time.

Main Quantitative Results

Overall Performance (Figure 1 and Section 4.1)

The headline result is the overall leaderboard shown in Figure 1, which aggregates weighted scores across all four difficulty tiers for the evaluation period of July 20 to August 3. Grok-4 (Think&Search) achieves the highest overall score, followed by Gemini-2.5-flash Deep Research, GPT-o4-mini (Think&Search), and Doubao-Seed1.6-Thinking (base LLM, labeled as Seed1.6 in the figure). The paper does not report exact numeric overall scores in the text — the results are presented visually in Figure 1 as a bar chart — but the ordering is clearly established.

A cross-category pattern is evident: "Generally, reasoning models equipped with search capabilities outperform the rest, underscoring the importance of advanced search and reasoning in FutureX." Specifically, Think&Search models and closed-source Deep Research models occupy the top positions, while base LLMs and SmolAgent-based configurations cluster lower. The paper flags one open-source agent finding: "SmolAgent-DR underperforms compared to LLM (Think&Search), likely reflecting differences in their search API capabilities." AgentOrchestra is excluded from the overall leaderboard entirely because it was evaluated only on Levels 1 and 2 due to computational intensity.

The human baseline, computed from 40 industry experts on 300 randomly sampled questions, is shown as red dotted lines in the per-tier breakdowns (likely Figures 8 or 17 — the paper references Figure 17 for detailed tier results). Human experts "significantly outperform LLM agents on Level 1, Level 3, and Level 4 events," with the notable exception of Level 2 multi-choice events, where "some models actually surpass human performance; this may be because these multi-choice questions involve so many options that people often cannot exhaustively compare every possibility." The paper cautions that "because the human annotations and model tests did not use exactly the same question set, these comparisons should be viewed as rough indicators."

Results Across Difficulty Tiers (Section 4.2, Figures 8 and 17)

The paper presents a detailed breakdown of performance across the four difficulty tiers, unveiling five key findings:

Finding 1: Difficulty tiers reflect genuine task complexity. A "clear, consistent decline in performance across the 4 defined difficulty tiers" is observed: models achieve high accuracy on Levels 1 and 2, but "their performance drops significantly on Level 3 and, in particular, on Level 4." This monotonic performance decline validates the tier construction. The paper further notes that even within the same domain, comparing Level 1&2 versus Level 3&4 events (Figures 9 and 10) shows "model performance declines substantially," confirming that difficulty is driven by event structure rather than domain alone.

Finding 2: Base LLMs perform well on Level 1 and Level 2. On these easier tiers — single-choice and multi-choice events with predefined options — base LLMs without any tool use "consistently achieve high accuracy." Notably, DouBao-Seed1.6-Thinking "outperforms several agents equipped with web search tools, including the two Deep Research agents" on these levels. The paper proposes two explanations: "These events may primarily rely on basic factual recall or straightforward reasoning, which base LLMs are already capable of handling without external tools," and "achieving around 60% accuracy on these events does not demand extensive searching." This finding directly motivates the tier weighting scheme: "Level 1 and Level 2 events are not sufficiently challenging to distinguish between models of varying capabilities," hence their 10% and 20% weights respectively. The paper uses this to validate its downsampling decision: "They also underscore the advantage of FutureX over prior benchmarks that predominantly feature Level 1 events derived from prediction markets."

Finding 3: Search and tool use become increasingly important for harder events. On Level 3 events (open-ended, low volatility), "models that incorporate external tools such as web search, calculators, or code execution tend to perform significantly better than those that rely solely on static knowledge." The paper argues this is because "many of the harder events, especially open-ended ones, involve dynamic or real-world developments where up-to-date information is essential" — knowledge that "may not exist in their training data." This is most visible in domains like "current affairs, emerging technologies, or ongoing geopolitical situations, where factual accuracy depends on retrieving the most recent context." The finding confirms that FutureX successfully isolates search-dependent capability as a distinct evaluation axis.

Finding 4: DouBao-Seed1.6-Thinking excels on knowledge retrieval, Grok-4 excels on difficult events. Among base LLMs, DouBao-Seed1.6-Thinking performs best on Level 1 and Level 2 events, "even outperform[ing] agents equipped with search tools as well as deep research models." The paper interprets this as evidence that "when provided with answer options, DouBao-Seed1.6-Thinking is highly effective at retrieving and applying its internal knowledge to make accurate predictions about future events." On the hardest events (Levels 3 and 4), Grok-4 "stands out" — it "surpasses even premium models such as Gemini Deep Research in both accuracy and efficiency" while "operating with fewer searching and faster inference speeds." GPT-o4-mini also achieves "top-tier results" on these tiers, demonstrating "an impressive balance between reasoning strength and runtime efficiency."

Finding 5: LLM agents still lag behind humans. The human baseline of 40 industry experts significantly outperforms LLM agents on three of the four difficulty levels (Levels 1, 3, and 4). The exception is Level 2, where some models surpass human performance — attributed to the exhaustive comparison burden of multi-choice questions overwhelming human working memory. The paper frames this as evidence that "despite their promise, LLMs still have considerable ground to cover before matching human expertise."

Results Across Domains (Section 4.3, Figures 9 and 10)

Domain-specific radar charts are presented separately for Level 1&2 events (Figure 9) and Level 3&4 events (Figure 10). Three patterns emerge:

Different models have different domain strengths. GPT-family models — GPT-4.1 (base), GPT-4.1 (SmolAgent), and GPT-o4-mini (Think&Search) — demonstrate "superior performance in Crypto and Technology." DouBao-Seed1.6-Thinking "excels in Finance & Economy and Business & Companies." DeepSeek-V3 (SmolAgent) "performs exceptionally well in Politics, even outperforming closed-source deep-research agents and Think&Search LLMs." This domain-specific variation suggests that model rankings are not uniform — different deployment contexts may favor different model choices.

Search-enhanced reasoning most benefits information-driven domains. For domains like Culture & Media and Technology, "performance increases notably as we move from basic to more advanced reasoning frameworks." The paper attributes this to these domains benefiting "directly from timely information access and contextual reasoning grounded in real-world updates." This is consistent with the difficulty-tier finding that search matters more when answers aren't in pre-training data.

Tool use amplifies performance differences. On Level 1 and Level 2 events, "the performance differences among the base LLMs are not very large, but as tools are added, the gaps in the radar charts widen." The paper speculates this "likely reflect[s] each model's choice of search tools and reasoning style" — suggesting that tool-augmented evaluation reveals capability differences that are masked when only static knowledge is tested.

Advanced search is insufficient in complex domains. On Level 3 and Level 4 events (Figure 10), "even models equipped with strong search capabilities struggle to perform well." The paper identifies a potential limitation: "These open-ended tasks often demand deeper abstraction, multi-hop reasoning, and a synthesis of heterogeneous information," capabilities that go beyond what current search-augmented architectures provide. This result directly supports the claim that Super Agent tier events test a capability frontier that current models have not yet reached.

Factor Analysis (Section 4.4, Figure 11)

To systematically quantify which factors drive performance, the paper runs a linear regression where the target variable is each model's score on each event, and predictors include the choice of LLM, event domain, and difficulty tier. The regression achieves an R² of 0.418 — the predictors explain about 42% of the variance in model scores.

Difficulty level is the strongest factor. The estimated coefficients show that "difficulty level has a significant impact on model performance" — consistent with all prior analyses. The paper notes this "validates our overall scoring scheme, in which we assign 10% and 20% weights to Level 1 and Level 2 events, respectively, to place greater emphasis on more challenging cases."

Domain matters independently. "Substantial variation in the coefficients of different domains" is observed, confirming that domain-specific challenges are real and not merely artifacts of difficulty tiering. This "highlights the importance of domain-specific challenges and the need for tailored evaluation."

Top models align with the overall leaderboard. The four highest-performing models identified in the regression — Grok-4, GPT-o4-mini, Gemini Deep Research, and Seed1.6 (DouBao) — are "exactly the same as those in the overall score ranking, confirming the consistency and robustness of our benchmark." This is an internal consistency check: if the factor analysis had identified different top models, it would suggest that the overall weighting scheme was distorting the ranking.

Focused Case Studies (Section 4.5)

Past Prediction vs. Future Prediction (Section 4.5.1, Figure 12)

To isolate search capability from forecasting ability, the paper introduces a past-prediction task: for 30 randomly selected Level 1 and Level 2 events, models are asked to retrieve the outcome one week after its resolution date (when the answer is publicly available online). The same models also made future predictions on the same events before resolution. Comparing dark bars (past) and light bars (future) in Figure 12 reveals:

Grok-4 leads in search capability, followed by GPT models and Hunyuan. In past prediction, "Grok-4 significantly outperforms all other methods, underscoring its robust and timely information retrieval." GPT-o4-mini and GPT-4o also deliver "strong past-prediction performance." This establishes a search-quality baseline — Grok-4 can find answers when they exist.
Hunyuan shows a large search-reasoning gap. Despite "impressive past-prediction results," "the large gap between its future- and past-prediction scores suggests weaker reasoning ability — meaning that despite effective search, its overall performance suffers." This is a concrete illustration of the search-versus-reasoning decomposition: Hunyuan can find information but cannot effectively use it for forecasting.
SmolAgent with Gemini-2.5-pro achieves significant gains, unlike with other backbones. Within the open-source SmolAgent framework, integrating Gemini-2.5-pro "yields a marked improvement in past-prediction performance — comparable to most commercial models (except Grok-4)." Crucially, this demonstrates that "SmolAgent's relatively modest overall performance cannot be blamed solely on the quality of its search API." When paired with other backbones, SmolAgent "shows little to no improvement — and in some cases, even a performance decline — suggesting that the underlying search capabilities of those models play a significant role in the agent's effectiveness."

Planning Analysis of SmolAgent (Section 4.5.2, Table 5 and Figure 13)

The paper conducts a detailed analysis of SmolAgent's internal planning behavior by examining the agent's memory — which contains "a plan to solve the problem, a detailed log of tool usage, and the outcome of each tool call in every iteration." Since closed-source model internals are inaccessible, this analysis is restricted to SmolAgent configurations.

Planning quality evaluation methodology. For each study plan generated by a SmolAgent instance, Gemini-2.5-pro (acting as an evaluator, with model identities anonymized to prevent bias) assigns scores from 0 to 10 on three dimensions: (1) Comprehensiveness — how thoroughly the plan covers necessary components and relevant information; (2) Source Reliability — how credible and valid the referenced sources are; (3) Plan Actionability — how practical and executable the proposed steps are. Table 5 provides scored examples for each model-configuration.

GPT-4.1 and Gemini-2.5-pro achieve the highest planning scores. Table 5 shows GPT-4.1 receives scores of 9 (Comprehensiveness), 8 (Source Reliability), and 9 (Plan Actionability), while Gemini-2.5-pro receives 10, 10, and 10 respectively. The paper notes this "is consistent with their superior overall performance demonstrated in Figure 2, which suggests a strong relationship between an agent's planning capability and its future-prediction performance."

Concrete differences in planning quality by dimension. The paper draws contrasts between strong and weak planners:

Comprehensiveness: GPT-4.1's plan addresses "a wide range of specific and sophisticated risk factors" such as "upcoming catalysts or risk factors affecting ETH from now until July 2025 (e.g., expected upgrades, ETF approvals/rejections, major regulations, known hack/theft risks, etc.)." In contrast, Qwen3-32B's plan covers only "basic elements and lack[s] the necessary depth or specific guidance" — a generic statement about "any relevant news, events, or macroeconomic indicators."
Source Reliability: Gemini-2.5-pro's plan specifies sources like "Bloomberg, CoinDesk, Messari, Goldman Sachs, JPMorgan, etc." — "unmatched in its list of specific, top-tier financial and crypto-native sources." Qwen3-32B's plan, by contrast, suggests "crypto-focused forums like Reddit or Twitter/X" — "lower[ing] reliability by mixing authoritative sources with unvetted social media for factual research."
Plan Actionability: A critical finding emerges with DeepSeek-v3, which scores 0 on Plan Actionability because it "reaches a conclusion within the planning phase itself" — providing a final answer in the plan rather than executing the planned search. The paper interprets: "This behavior suggests that these models may not always rely on search tools to solve problems, instead leveraging their internal knowledge base to form a final plan." This is a concrete failure mode where a model "knows" the answer from pre-training data and short-circuits the research process.

Regression analysis of planning factors (Figure 13). A linear regression restricted to SmolAgent's performance on Level 1 and Level 2 events (since it performs poorly on Levels 3 and 4) examines the impact of total tool calls, search text length, overall context length, and the three planning scores. The R² is 0.518 — slightly higher than the overall factor analysis.

Key findings:

"Number of tool calls, source reliability, and plan comprehensiveness exert the strongest positive effects on the overall score: more frequent tool calling, higher trustworthiness of referenced information, and more thorough answer content all drive substantially higher user ratings."
"Main agent think length carries the most negative effect: longer accumulated dialogue history introduces noise and redundancy, which hurts the performance." This is a practically actionable insight — verbose reasoning chains in agent memory may degrade rather than improve performance.

Search Analysis (Section 4.5.3, Figure 14)

Figure 14 plots the average number of web queries per question for commercial Think&Search models and Deep Research models. GPT-o4-mini and GPT-4o are excluded because their search counts are not measurable.

Grok-4 issues the largest number of searches among Think&Search models. This finding has two implications: "first, Grok-4's high query volume corresponds with its superior performance on the FutureX; second, it achieves this result with remarkably low latency — completing searches in less than 5 minutes, compared to approximately 30 minutes for Gemini deep research." Efficiency and query volume appear to be complementary rather than antagonistic — Grok-4 searches more and faster.

Deep Research models conduct substantially more searches than Think&Search models. The two deep research models perform "substantially more searches than the rest of the Think&Search cohort, highlighting their aggressive retrieval strategies." The paper does not report exact search counts but orders them visually in Figure 14.

Low-search models underperform. Qwen3-235B and Hunyuan "perform significantly fewer web queries, which may help explain their relatively lower performance on the same tasks." This is suggestive but not conclusive — as the Hunyuan past-prediction analysis showed, search count alone is not sufficient without reasoning quality.

Ablation Studies and Robustness Checks

The paper does not contain traditional ablation studies in the sense of removing components from a model or pipeline and measuring degradation. This is because FutureX is a benchmark, not a method — there is no proposed model to ablate. However, several analyses serve as implicit robustness checks on the benchmark's design choices:

Difficulty tier validity check: The monotonic performance decline across the four tiers (Finding 1, Section 4.2) serves as validation that the event type and volatility-based tiering system captures genuine task complexity. If models performed equally well on Levels 3 and 4, or if the ordering were inconsistent across models, the tiering would be called into question. The paper reports a "clear, consistent decline" across all model categories, which supports the tier construction. However, the paper does not report whether this monotonicity holds for every individual model or only in aggregate — an important nuance.

Tier weight sensitivity: The overall score uses fixed 10%/20%/30%/40% weights for Levels 1-4. The paper does not report sensitivity analysis showing how the leaderboard ranking would change under alternative weightings (e.g., equal weights, or even more extreme emphasis on Level 4). This is a genuine limitation: the top-model ordering (Grok-4 > Gemini DR > GPT-o4-mini > Seed1.6) might be sensitive to the specific weights chosen. The factor analysis (Figure 11) partially addresses this by showing that difficulty level is significant, but it does not directly test weight sensitivity.

Oracle vs. predicted difficulty: Unlike some benchmarks where difficulty labels are assigned by human annotators and might not correspond to actual model difficulty, FutureX's difficulty tiers are derived from structural properties of the events (choice structure and outcome volatility) — objective properties that exist independently of any model's performance. This structural approach is inherently more robust than oracle (performance-based) difficulty binning, though it relies on the assumption that event type and volatility are sufficient proxies for cognitive difficulty. The paper does not validate this assumption beyond showing that performance declines across tiers — a correlation that is consistent with but does not prove the causal claim that event structure determines difficulty.

Missing prediction robustness: The Monte Carlo simulation (Section 3.4.2, Figure 7) is effectively a robustness check on the decision not to align events across models. By showing that the standard deviation introduced by missing rates up to 20% is "relatively small" and decreasing with sample size, the paper justifies its pragmatic choice. However, the simulation assumes missing events are randomly distributed, which may not hold in practice — if certain models systematically refuse to predict specific types of events (as the GPT-o-series refusal behavior suggests), the missingness is not random and could introduce bias that the simulation does not capture.

Human baseline as external validity check: The inclusion of 40 industry experts on 300 questions serves as a quasi-ablation: if models could match or exceed human performance on all tiers, that would suggest the benchmark is too easy. The finding that humans significantly outperform on Levels 1, 3, and 4 validates that the benchmark is appropriately challenging. The Level 2 exception (models surpassing humans) is interesting but the paper provides a plausible mechanistic explanation (exhaustive comparison burden).

Past-vs-future comparison as search-reasoning decomposition: The past-prediction experiment (Section 4.5.1) serves as an informal ablation of the search component: by asking models to retrieve a known answer rather than predict an unknown one, the task is reduced to pure search capability. The gap between past and future performance for models like Hunyuan isolates the reasoning contribution. This is methodologically clever but limited to 30 events and only Levels 1-2 — it does not address whether the same decomposition would hold for more difficult events.

Planning quality as a process-level validation: The evaluation of SmolAgent planning (Table 5) serves as a form of construct validity check: if the benchmark genuinely measures forecasting capability, then process-level measures of planning quality should correlate with outcome-level scores. The paper reports that GPT-4.1 and Gemini-2.5-pro achieve the highest planning scores and the best SmolAgent performance, which is consistent with this assumption. The regression (Figure 13) provides quantitative support with R² = 0.518.

Critical Assessment

The experimental results demonstrate several things convincingly, but there are important gaps between what the experiments show and what the paper's framing claims.

What is convincingly demonstrated:

FutureX operates as a functioning live benchmark. The paper reports results from a two-week window with 1,272 events, 25 models, and a reported 97% answer acquisition success rate. This establishes that the automated pipeline — the core engineering contribution — actually works at scale.
Search-augmented models outperform base LLMs on open-ended prediction tasks. The tier-stratified results (Figures 8, 17) show this pattern consistently: base LLMs are competitive on Levels 1-2 but fall behind Think&Search and Deep Research models on Levels 3-4. This supports the claim that the benchmark measures something beyond static knowledge retrieval.
Current models struggle substantially on the hardest tier. The Super Agent tier results (Figure 8d) show low or near-zero performance for most models, with the human baseline substantially outperforming. This establishes a clear capability ceiling that validates the benchmark's difficulty.
The difficulty tiers produce monotonic performance ordering. Performance consistently decreases from Level 1 to Level 4 across all model categories, suggesting the tier construction is meaningful.
Domain-specific strengths exist and are measurable. The per-domain radar charts (Figures 9, 10) reveal non-uniform performance patterns — GPT models excel at Crypto and Technology, DouBao-Seed1.6 excels at Finance, DeepSeek-V3 excels at Politics — suggesting the benchmark can detect genuine capability variations.

What is not adequately demonstrated:

The claim that FutureX is "the largest and most diverse live benchmark for future prediction" (Section 1) is supported by the comparison table (Table 1), but the comparison is somewhat asymmetric. FutureX's daily pipeline generates approximately 500 candidate events per week, of which 70-100 are selected daily — an impressive operational throughput. However, the two-week evaluation window contains 1,272 events, which is smaller than ForecastQA (10,392 events) and Autocast (6,707 events), both static benchmarks. The paper's claim to "largest" is qualified by "live" — it is comparing against other live-updating benchmarks like ForecastBench (6,402 events accumulated over months) and FutureBench (42 events total). This is a fair comparison on the "live" dimension, but the absolute number of evaluated events is modest relative to static benchmarks, and the two-week evaluation window is short. The paper acknowledges this implicitly by stating that the results "are updated weekly" and that "as FutureX progresses and the test size grows, the standard deviation is expected to decrease." The current results should be viewed as a snapshot from an ongoing benchmark rather than a definitive ranking.

The claim that FutureX is "contamination-impossible by design" (Section 3.1) is structurally true — answers don't exist at prediction time — but the paper does not address whether the questions themselves could be contaminated. The question templates and source websites are fixed; a model trained on data from these websites could potentially learn patterns (e.g., "Dongchedi rankings tend to follow certain patterns in August") or memorize the templates themselves. The paper argues that variable randomization prevents asking the same question daily, but it does not discuss whether models could learn domain-specific regularities that partially shortcut the prediction task. This is probably a minor concern for future events (since the exact outcome is genuinely unknown), but it means the contamination guarantee is specific to the answer, not the task structure.

The comparison to human experts is suggestive but methodologically limited. The paper reports that human experts "significantly outperform LLM agents on Level 1, Level 3, and Level 4 events," but cautions that "because the human annotations and model tests did not use exactly the same question set, these comparisons should be viewed as rough indicators." This is a significant caveat: the 300 randomly sampled questions answered by humans are a subset of the total events, and it's unclear whether the difficulty distribution of this subset matches the full benchmark. Without alignment between human and model evaluation sets, the comparison is qualitative rather than quantitative. Additionally, the human baseline is from "40 industry experts," but their expertise may not be uniformly relevant across all 11 domains — a financial analyst may have no special insight into sports or entertainment predictions. The paper does not report domain-stratified human performance, which would be more informative than the aggregate comparison.

The model selection has significant gaps. GPT-o1, GPT-o3, and GPT-o4-mini as base LLMs were excluded because they "frequently refused to make predictions." Claude models are excluded entirely due to "policy and API stability issues." GPT Deep Research is excluded for similar reasons. These are not random exclusions — they systematically remove some of the most capable frontier models from the evaluation, which could affect the leaderboard ranking if those models were included. The Grok-4 result is notable and interesting, but we cannot know whether it would remain the top performer if all frontier models were evaluated. The paper is transparent about these exclusions, but the framing of "25 models across 4 categories" may overstate the comprehensiveness of the evaluation.

The planning analysis (Section 4.5.2) is restricted to SmolAgent and only Levels 1-2. The paper acknowledges that SmolAgent performs poorly on Levels 3-4, which means the planning analysis is conducted on the difficulty tiers where planning matters least (since base LLMs already perform well on Levels 1-2 without sophisticated planning). The finding that planning quality correlates with performance on easy events is less informative than an equivalent analysis on hard events would be. The paper does not discuss this limitation.

The search analysis (Section 4.5.3) cannot measure query counts for GPT-o4-mini and GPT-4o, which are among the top-performing models. The correlation between search count and performance is therefore observed on a restricted and potentially biased subset of models — specifically, Grok-4 (top performer, high search count) and several lower-performing models (Qwen3-235B, Hunyuan, low search count). The paper cannot rule out that there exist models with low search counts and high performance (efficient search) or high search counts and low performance (ineffective search), which would complicate the narrative.

Missing experiments that would strengthen the paper:

Longer evaluation windows. Two weeks is a snapshot. The paper's claim that FutureX is a sustainable live benchmark would be strengthened by reporting results over months, showing that the pipeline remains operational, that rankings are stable, and that the benchmark resists overfitting over time.
Test-retest reliability. Do model rankings remain consistent across different weeks? If Grok-4 is the top performer one week but falls to fifth the next, the benchmark may be too noisy to support reliable comparisons. The paper does not report any measure of temporal stability.
Ablation of the 30-minute timeout. Does performance improve with longer time budgets? The paper notes that Gemini Deep Research uses the full 30 minutes while Grok-4 finishes in under 5 minutes — this suggests Grok-4 is under-utilizing its budget, and a longer timeout might advantage models that conduct deeper research. Without a timeout sensitivity analysis, we cannot know whether the 30-minute constraint distorts the ranking.
Calibration analysis. Future prediction is inherently probabilistic, yet the evaluation metrics are all accuracy-based. A model that consistently predicts 60% confidence and is correct 60% of the time is better calibrated than a model that predicts 90% confidence and is correct 60% of the time, but both would receive the same score on single-choice events. The paper does not assess whether models are calibrated — a significant omission for a benchmark targeting "reasoning under uncertainty."
Comparison of event types holding difficulty constant. The paper shows that performance declines across difficulty tiers, but does not isolate whether specific event types (ranking vs. numerical vs. multi-choice) are harder than others independent of their tier assignment. Such an analysis would validate or challenge the tier construction.

Where the claims hold conditionally:

The claim that search-augmented models outperform base LLMs holds only on open-ended events (Levels 3-4). On Levels 1-2, base LLMs are competitive or superior (DouBao-Seed1.6 outperforms Deep Research agents). The paper acknowledges this ("Finding 2"), but the overall framing sometimes elides this condition.
The claim that the difficulty tiers "accurately reflect the complexity of the events" holds in aggregate but the paper does not report whether it holds for every model individually. A model with strong search but weak reasoning might find Level 3 (search-heavy, low-volatility) easier than Level 2 (reasoning-heavy, exhaustive discrimination), which would challenge the monotonicity assumption.
The claim of "the largest and most diverse live benchmark" holds relative to other live benchmarks (ForecastBench, FutureBench) but the absolute scale (1,272 events over two weeks, ~500 events/week) is modest compared to the scale of static benchmarks and will need to grow substantially for the benchmark to support fine-grained model comparisons with narrow confidence intervals.

6. Limitations and Trade-offs

The Difficulty Estimation and Template Construction Pipeline Relies on Substantial Human Effort That Limits Scalability

The assumption or constraint. The paper frames FutureX as a "fully automated" benchmark, but the initial construction of the event database and question templates requires significant human intervention that is not amortized away over time — it is a recurring cost whenever new websites or domains are added. Section 3.2.1 explicitly describes a manual review stage where the 636 LLM-filtered websites are "manually reviewed, focusing on identifying reliable sources—particularly ranking lists and websites with high update frequency," reducing the set to 195. Similarly, the template generation process in Section 3.2.2 involves human verification: "a human reviewer then verifies the results and selects the most appropriate question templates." The paper also acknowledges that when answer extraction fails, "we manually review the cases and design customized prompts to improve answer extraction accuracy" (Section 3.2.4). This human effort is not measured, costed, or factored into any claim about the benchmark's scalability.

The consequence. The benchmark's domain coverage is gated by human labor. The paper reports 11 domains from 195 websites; expanding to new domains (e.g., the "corporate websites, e-commerce websites, and research and educational platforms" mentioned as future work in Section 3.2.1) would require repeating the manual curation process. This means FutureX cannot scale to arbitrary breadth without proportional human investment — unlike fully automated benchmarks that grow through programmatic data collection. The quality of the benchmark also depends on the thoroughness of this human review, introducing variability that is difficult to replicate or audit. A second team attempting to reproduce FutureX on a different set of source websites might produce an easier or harder benchmark depending on their curation choices, undermining the standardization that benchmarks typically provide.

What evidence exists in the paper. The paper is transparent about the human involvement but does not quantify it. There is no report of person-hours required for the initial curation, the template creation, or the ongoing failure-case review. The answer acquisition pipeline reports 97% success — but this number was achieved after iterative manual prompt refinement and website exclusion, meaning the benchmark's reliability depends on undocumented human tuning that new adopters would need to replicate. Section 3.2.1 states that the event database "is updated daily to remove events with unavailable outcomes and continuously add new events using current high-quality websites as 'seeds'" — but whether this daily updating process also requires human oversight is not specified.

Mitigation status. The paper does not present this as a limitation to be mitigated; it describes the human review as a quality-control mechanism rather than a cost. No estimates of labor requirements are provided, and no proposals for automating the curation or template-generation steps are discussed. The paper frames FutureX as a "semi-automated pipeline," which is accurate, but the headline claim of being "fully automated" (used in the abstract and Section 3.2) is in tension with the multiple human-in-the-loop steps described in the construction phase.

The Two-Week Evaluation Window Is Too Short to Establish Reliable Model Rankings or Temporal Stability

The assumption or constraint. All quantitative results in the paper are drawn from a two-week evaluation period (July 20 to August 3, 2025) comprising 1,272 events. The paper acknowledges this implicitly: "Note that the pipeline runs daily, so these results are updated weekly" (Section 4 opening) and "as FutureX progresses and the test size grows, the standard deviation is expected to decrease" (Section 3.4.2). The leaderboard in Figure 1, the domain-specific analyses in Figures 9 and 10, the difficulty-tier breakdowns in Figure 8, and the factor analysis in Figure 11 are all computed from this single snapshot. There is no test-retest analysis showing whether model rankings are stable across consecutive weeks or whether the observed performance differences are statistically significant given the sample size.

The consequence. The paper's central empirical claims — that Grok-4 achieves the highest overall performance, that DouBao-Seed1.6-Thinking excels on knowledge retrieval, that search-augmented models outperform base LLMs on harder tiers — rest on a sample of 1,272 events that may not be representative of the benchmark's long-term behavior. Event distributions could shift week-to-week (the paper notes fluctuations: "the number of events each week varies due to fluctuations in prediction market events," Figure 3 caption). A model that benefits from a particular domain mix in one week might underperform in another. More critically, with 25 models and overlapping confidence intervals (visible in the bar charts of Figures 8 and 17, though exact confidence intervals are not reported), many pairwise comparisons may not be statistically significant. The human baseline is drawn from 300 questions that do not perfectly align with the model evaluation set, further complicating model-human comparisons.

What evidence exists in the paper. The Monte Carlo simulation in Section 3.4.2 quantifies the standard deviation introduced by missing predictions (roughly 500 events per week, with up to 20% missingness), and Figure 7 shows standard deviation increasing with missing rate. However, this simulation addresses score stability given a fixed underlying distribution, not distributional stability across weeks. It tells us that scores estimated from 500 events have a certain variance, but it does not tell us whether the true underlying model performance varies from week to week as the event mix changes. The paper does not report confidence intervals on model scores, does not perform pairwise significance tests between models, and does not report ranking correlations between consecutive weeks (which would require a second week of data not included in the paper).

Mitigation status. The paper positions FutureX as an ongoing benchmark and implies that reliability will improve with accumulated data. Section 3.4.2 notes that "as FutureX progresses and the test size grows, the standard deviation is expected to decrease at a rate proportional to the inverse square root of the sample size." This is correct statistically — larger samples reduce estimation variance — but it addresses only sampling error, not distribution shift. The paper does not commit to releasing temporal stability analyses or confidence intervals in future updates, nor does it discuss what sample size would be needed for reliable ranking (e.g., with what confidence can we say Grok-4 outperforms GPT-o4-mini by some margin?). The current two-week snapshot should be interpreted as a proof-of-concept demonstration that the pipeline produces sensible results, not as a definitive performance ranking.

Model Selection Has Systematic Gaps That May Distort the Leaderboard, Particularly at the Frontier

The assumption or constraint. The paper evaluates 25 models across four categories, but several of the most capable frontier models are excluded for reasons that are not random. Section 4 states that GPT-o1, GPT-o3, and GPT-o4-mini as base LLMs were excluded because "they frequently refused to make predictions." Section 3.2.3 notes: "Due to policy and API stability issues, we are unable to test the GPT Deep Research and Claude models." Gemini-2.5-pro (Think&Search) evaluation is described as "on the way" but not yet completed. AgentOrchestra is evaluated only on Levels 1 and 2 due to computational cost and excluded from the overall leaderboard. SmolAgent configurations exclude reasoning-model backbones (DouBao-Seed1.6-Thinking, DeepSeek-R1) because they "incur significantly longer runtimes."

The consequence. The leaderboard ranking — with Grok-4 at the top, followed by Gemini-2.5-flash Deep Research and GPT-o4-mini (Think&Search) — is established on an incomplete field. We cannot know whether the top positions would hold if the missing frontier models were included. The exclusion of GPT Deep Research and Claude is particularly consequential: if either of these models would have outperformed Grok-4, the paper's headline finding (that xAI's model leads) would be an artifact of missing competitors rather than a genuine capability signal. The exclusion of reasoning-model backbones from SmolAgent means the open-source agent results represent a lower bound — we cannot assess whether SmolAgent with DeepSeek-R1 would approach commercial performance. The omission of GPT-o-series models as base LLMs (due to refusal) is itself an important finding about safety-forecasting tensions, but it also means the base LLM category is missing what may be its strongest entrants, artificially widening the apparent gap between base LLMs and search-augmented models.

What evidence exists in the paper. The paper is transparent about every exclusion, listing them explicitly in Sections 3.2.3 and 4. No attempt is made to estimate how the missing models would perform — no partial results, no small-scale pilot with Claude or GPT Deep Research on a subset of questions, no analysis of whether GPT-o-series refusal rates vary by difficulty tier or domain. The planning analysis (Section 4.5.2) is restricted to SmolAgent "as we cannot access the internal memory of closed-source models," which is a reasonable constraint but limits the generalizability of the process-level findings.

Mitigation status. The paper acknowledges these gaps as future work: "We are actively working to integrate Gemini-2.5-pro (Think&Search), which will be added soon" (Section 4 footnote). For Claude and GPT Deep Research, the constraint is external (API policy and stability) and the paper does not propose a workaround. The refusal behavior of GPT-o-series models is mentioned but not analyzed as a phenomenon in its own right — the paper treats it as an obstacle to evaluation rather than a finding about model behavior. No recommendations are made for how future models should balance safety training with forecasting capability, despite this being one of the paper's implicit discoveries.

The Human Baseline Comparison Is Methodologically Insufficient to Support Claims About Human-Level Performance

The assumption or constraint. The paper reports that "humans significantly outperform LLM agents on Level 1, Level 3, and Level 4 events" based on a study with 40 industry experts answering 300 randomly sampled questions. However, Section 4.2 explicitly cautions: "because the human annotations and model tests did not use exactly the same question set, these comparisons should be viewed as rough indicators. Actual performance gaps may vary depending on question difficulty distribution and annotator backgrounds."

The consequence. The non-overlapping question sets mean that any observed human-model gap confounds genuine capability differences with differences in the sample of questions each group faced. If the 300 randomly sampled questions happened to be harder (or easier) than the full 1,272-event model evaluation set, the comparison would be systematically biased. Furthermore, the 40 experts are described as "current or former employees from the Big Four accounting firms, top consulting firms, and nine leading investment banks" — this is a specific professional demographic whose expertise is concentrated in finance and business domains. An expert in M&A at Goldman Sachs may have no comparative advantage over a random guess on a question about Billboard music rankings or Tour de France outcomes. The paper does not report domain-stratified human performance, so we cannot assess whether human superiority holds uniformly across all 11 domains or is driven by a subset where the experts have relevant knowledge.

Additionally, human experts in the study could use web search and external tools (the paper does not specify whether they were permitted to), but the time pressure and incentive structure likely differed substantially from the model evaluation — models had 30 minutes per question with no fatigue effects, while human experts answered 300 questions with presumably varying motivation and attention. The human baseline therefore represents a particular configuration of ability, tools, and motivation that may not generalize to what "human experts" would achieve under different conditions.

What evidence exists in the paper. The human results are presented as red dotted lines in the per-tier figures (Figure 17, referenced in Finding 5 of Section 4.2). No exact numeric scores are reported in the text. The paper does not provide: the distribution of expert backgrounds relative to question domains, the inter-annotator agreement or variance across experts, the time experts spent per question, whether experts had access to web search, or whether expert performance varied systematically with domain (e.g., did finance experts outperform on finance questions?). The Level 2 exception — where "some models actually surpass human performance" — is attributed to the exhaustive comparison burden of multi-choice questions overwhelming human working memory, which is a plausible explanation but is offered post-hoc without experimental manipulation.

Mitigation status. The paper acknowledges the limitations of the comparison ("these comparisons should be viewed as rough indicators") but does not commit to improving it. Section 4.2 states: "In future work, we plan to expand our question bank and include a more diverse pool of experts to improve the reliability and representativeness of these comparisons." This mitigates neither the overlapping-question-set problem nor the domain-expertise mismatch. A methodologically rigorous human baseline would require: (1) the same question set for humans and models, (2) domain-matched experts (or generalists with comparable information access), (3) comparable time and tool constraints, and (4) reporting of per-domain and per-tier scores with variance estimates. The current human study is best interpreted as a qualitative existence proof that the benchmark is not trivially solvable, not as a quantitative measure of the human-machine capability gap.

The Benchmark Does Not Assess Calibration or Probabilistic Reasoning Quality, Measuring Only Prediction Accuracy

The assumption or constraint. All four evaluation metrics in Section 3.4.3 are accuracy-based: single-choice events use exact-match (0/1), multi-choice events use F1-score, ranking events use set-overlap partial credit, and numerical events use a quadratic penalty within one standard deviation but zero credit beyond. None of these metrics reward a model for expressing appropriate uncertainty — a model that says "I am 60% confident that Option A will win" is scored identically to a model that says "I am 100% confident that Option A will win" when Option A is correct, and identically to a model that says "I am 60% confident" when Option A is incorrect. The numerical prediction metric partially captures a related concept (predictions within one standard deviation receive partial credit, reflecting that exact prediction is unreasonable), but this is a function of outcome volatility, not of the model's own expressed uncertainty.

The consequence. The benchmark cannot distinguish between a well-calibrated forecaster and an overconfident guesser — two models with identical accuracy but vastly different epistemic humility receive the same score. This is particularly problematic for a benchmark that claims to evaluate "reasoning under uncertainty" (Section 3.1, Table 3) and "decision-making under conditions of incomplete information and uncertainty" (Section 3.1). If a model confidently predicts an incorrect answer, the benchmark treats this identically to a model that acknowledges low confidence in an incorrect answer. For Super Agent tier events — which are explicitly described as requiring "nuanced, uncertainty-aware forecasting" (Section 3.3) — the absence of calibration evaluation means the benchmark cannot assess the very capability it claims to target.

This limitation also creates an incentive misalignment for model developers optimizing against FutureX: a model that always expresses 100% confidence will maximize its expected accuracy-based score, even though such a model would be dangerous to deploy in real forecasting applications where users need to know when a prediction is unreliable. The Deep Research agent frameworks tested (SmolAgent, AgentOrchestra) are capable of producing confidence estimates in their internal reasoning, but this output is never evaluated.

What evidence exists in the paper. The evaluation metrics section (3.4.3) defines four scoring functions, none of which take model-expressed confidence as an input. The scoring functions receive only the predicted answer Ŷ (or $\hat{\mathcal{Y}}$ for sets, or $\{\hat{y}_1, \dots, \hat{y}_k\}$ for rankings) and the ground truth. There is no mechanism for models to submit probabilistic predictions (e.g., "30% chance of Option A, 70% chance of Option B"), no Brier score or log-loss metric, and no reliability diagram or expected calibration error analysis. The human baseline is also evaluated using the same accuracy metrics, meaning we cannot compare human and model calibration — only their hit rates. The planning analysis (Section 4.5.2) evaluates qualities like comprehensiveness and source reliability but does not assess whether the agent's final prediction includes appropriate uncertainty quantification.

Mitigation status. The paper does not acknowledge this as a limitation. The scoring design choices are presented as natural consequences of the event types rather than as conscious decisions to exclude calibration. The numerical prediction metric's use of σ(Y) (7-day outcome volatility) is the closest the benchmark comes to incorporating uncertainty into scoring, but this measures outcome volatility, not forecast uncertainty — a stock might be highly volatile (rewarding wide prediction ranges) while a model might be overconfident within that range. Incorporating calibration evaluation would require either: (1) allowing models to submit probabilistic forecasts (e.g., full distributions for numerical questions, probability vectors for choice questions), (2) implementing proper scoring rules (Brier score, continuous ranked probability score), and (3) reporting calibration diagnostics alongside accuracy. This is a non-trivial extension that would substantially change the benchmark's interface and evaluation philosophy, but it would align the evaluation with the paper's stated goal of assessing reasoning under uncertainty.

7. Implications and Future Directions

How This Work Changes the Landscape

FutureX does not introduce a new model, training method, or algorithmic innovation. Instead, it introduces a new evaluation paradigm — live, prospective forecasting as a benchmark for agent capability — and in doing so, it reshapes how the field should think about several interconnected problems.

A contamination-impossible evaluation standard shifts the burden of proof for benchmark integrity. The dominant approach to data contamination in LLM evaluation has been reactive: researchers develop post-hoc tests to detect memorization (canary strings, training data overlap analysis) or construct benchmarks from newly released sources that postdate model training. These approaches are inherently probabilistic — they reduce the likelihood of contamination but cannot guarantee its absence. FutureX demonstrates a categorically different strategy: structure the task such that correct answers do not exist anywhere in the world at evaluation time, making contamination logically impossible rather than statistically unlikely. This is not a stronger version of existing approaches — it is a different axis entirely. The paper's demonstration that a daily pipeline can sustain this "predict now, score later" architecture at scale (195 websites, approximately 500 events per week, 97% answer retrieval) proves that the approach is operationally feasible, not just theoretically appealing.

This shift has implications beyond forecasting. Any domain where ground truth emerges over time — breaking news summarization, live event commentary, real-time question answering about unfolding situations — could adopt the same temporal architecture. The concept of evaluation delay as a contamination prevention mechanism rather than an inconvenience is a genuinely new design primitive for benchmark construction. It makes the reactive contamination-detection paradigm look increasingly fragile: why spend effort detecting memorization when you can design tasks where memorization is physically impossible?

The difficulty tiering system introduces a diagnostic vocabulary that was missing from agent evaluation. Most benchmarks partition their test sets by topic (MMLU's 57 subjects) or by author-assigned difficulty (GAIA's three levels). These partitions answer the question "what is the model good at?" but not the question "what specific capability does the model lack?" FutureX's four-tier system — Basic, Wide Search, Deep Search, Super Agent — maps each tier to specific assessed skills (planning depth, reasoning complexity, search sophistication) in a rubric that enables capability-gap diagnosis. The critical conceptual distinction is between Level 3 (Deep Search: open-ended but low-volatility, where exhaustive information gathering can in principle converge on the correct answer) and Level 4 (Super Agent: high-volatility, where no amount of search can eliminate fundamental uncertainty). This distinction isolates reasoning under deep uncertainty from comprehensive information retrieval — two capabilities that prior benchmarks conflate.

The empirical results validate that this diagnostic framework works: base LLMs perform well on Levels 1-2 (knowledge retrieval), search-augmented models pull ahead on Level 3 (retrieval + synthesis), and even the strongest models collapse on Level 4 (uncertainty-aware reasoning), where humans maintain a clear advantage. This pattern transforms FutureX from a leaderboard into a capability profiler — a researcher can look at a model's performance across the four tiers and identify precisely which cognitive bottleneck is limiting its forecasting ability. No prior forecasting benchmark (ForecastBench, FutureBench) provides this diagnostic resolution because they lack a principled difficulty taxonomy grounded in task properties rather than author judgment.

The paper reconciles a tension between "agents need tools" and "base models already know a lot." The finding that base LLMs — particularly DouBao-Seed1.6-Thinking — match or exceed search-equipped agents on Level 1 and Level 2 events (Section 4.2, Finding 2) might seem to undermine the paper's own premise that search is essential for forecasting. But the result is more subtle and more important: it shows that the value of tools is difficulty-conditional. On easy tasks with predefined options, a model's internal knowledge is sufficient and tool use adds overhead without benefit. On hard open-ended tasks, internal knowledge is insufficient and tools become essential. This parallels the difficulty-conditioned findings from the test-time compute scaling literature (Snell et al., 2024, analyzed in the reference example), where search helps on medium problems but over-optimizes on easy ones. The convergence of these patterns across different domains (math reasoning vs. real-world forecasting) suggests a general principle: augmentation strategies are not uniformly beneficial; their value depends on whether the base model's unaided capability is sufficient for the task. This reframes the debate about tool-augmented agents from "should we use tools?" to "when should we use tools?" — a more productive question that FutureX's difficulty tiering enables researchers to answer empirically.

The identification of refusal behavior as a forecasting-specific capability barrier. The paper's report that GPT-o1, GPT-o3, and GPT-o4-mini "frequently refused to make predictions" (Section 4) is easy to overlook as a methodological nuisance, but it is actually one of the paper's most significant contributions to the alignment conversation. These models represent the frontier of reasoning capability from a major lab, and their refusal to engage with forecasting tasks — which the paper's pipeline has already filtered for harmful and subjective content — reveals a capability-safety tension specific to prediction under uncertainty. Current alignment techniques appear to have produced models that cannot distinguish between "I should not make harmful claims" and "I should not make uncertain predictions about verifiable future events." The consequence is that some of the most capable reasoning models are structurally excluded from an entire class of legitimate, high-value applications (financial analysis, policy planning, risk assessment).

This finding has not been documented in prior work on model refusal, which has focused primarily on toxicity, stereotypes, dangerous knowledge, and illegal content. The forecasting case is different: the harm is not in the content of the prediction but in the act of being wrong, and alignment techniques that penalize confident statements about uncertain topics may inadvertently produce models that refuse to forecast at all. This creates a concrete research agenda: how do we train models that are willing to make calibrated probabilistic predictions while still refusing genuinely harmful content? The paper does not solve this problem, but it provides the first clear empirical evidence that the problem exists at scale.

The paper makes open-source agent frameworks directly comparable to commercial systems on a shared, objective task. Most agent benchmarks evaluate either commercial APIs (as black boxes) or open-source frameworks on different tasks, making direct comparison difficult. FutureX evaluates SmolAgent, AgentOrchestra, commercial Think&Search APIs, and closed-source Deep Research systems on the identical question set under identical time constraints (30 minutes per question) and identical scoring rules. This enables findings like "SmolAgent with Gemini-2.5-pro achieves planning scores comparable to commercial models" (Table 5) and "SmolAgent's relatively modest overall performance cannot be blamed solely on the quality of its search API" (Section 4.5.1) — claims that require a shared evaluation framework to be meaningful. The benchmark thus serves as a Rosetta Stone for translating between open-source agent research and commercial capability claims, enabling the open-source community to track its progress against proprietary systems on a level playing field.

Follow-Up Research This Work Enables

Calibration-aware forecasting evaluation. FutureX's current metrics are entirely accuracy-based — a model receives the same score whether it expresses 60% or 100% confidence in a correct (or incorrect) prediction. This is the single largest gap between the benchmark's stated goal (evaluating "reasoning under uncertainty") and its actual measurement. A natural extension would modify the evaluation interface to accept probabilistic predictions — probability vectors for choice questions, predictive distributions for numerical questions — and implement proper scoring rules (Brier score for binary/multi-class, continuous ranked probability score for numerical distributions). The key experiment would be: does the leaderboard ranking change when calibration is rewarded? A model that achieves high accuracy through overconfident guessing might rank first under accuracy metrics but fall behind a well-calibrated model under proper scoring rules. The decomposition would reveal whether current frontier models achieve their accuracy through genuine forecasting skill or through lucky overconfidence — a distinction with direct practical consequences for deployment in high-stakes settings. The paper already has the infrastructure (daily events with ground truth, multiple model outputs) to compute calibration diagnostics retrospectively without changing the prediction interface; the gap is analytical, not infrastructural.

Dynamic difficulty-adaptive tool-use policies. The finding that tool use is most valuable on Level 3-4 events but unnecessary (or even counterproductive) on Level 1-2 events (Section 4.2, Findings 2-3) suggests an unexplored optimization: can agents dynamically decide whether to search based on an initial assessment of question difficulty, rather than always searching or never searching? A concrete experiment would take the SmolAgent framework (which exposes its internal planning process) and modify it to include a two-stage decision: first, classify the question's likely difficulty tier using only the question text and a lightweight classifier (or few-shot prompt), then allocate search budget accordingly — minimal search for likely-Level-1 questions, aggressive multi-hop retrieval for likely-Level-3/4 questions. The baseline would be SmolAgent's current behavior (always searching) and the base LLM's behavior (never searching). The hypothesis, grounded in the paper's difficulty-tier results, is that a difficulty-adaptive policy would match base LLM performance on easy questions (saving compute), match search-augmented performance on hard questions, and potentially outperform both by avoiding the noise and overthinking that the planning analysis identified as harmful (Figure 13: longer think length carries a negative coefficient). This would extend the paper's diagnostic framework into an operational strategy.

Source reliability as a trainable component of agent architecture. The planning analysis (Table 5) reveals that source reliability judgments vary dramatically across models — Gemini-2.5-pro's plans cite "Bloomberg, CoinDesk, Messari, Goldman Sachs, JPMorgan" while Qwen3-32B's plans cite "Reddit or Twitter/X." The regression analysis (Figure 13) shows that source reliability scores have a strong positive coefficient on overall performance. Yet current deep research agents treat source selection as an emergent property of the underlying LLM's planning, not as a separately trained or optimized component. A concrete follow-up would train a lightweight source-credibility classifier on the FutureX event database — using the 195 vetted high-quality websites as positive examples and the websites filtered out during curation (the 2,008 → 636 → 195 pipeline) as negative examples — and integrate it as a tool that the agent can query during planning. The experiment would compare SmolAgent with and without this credibility tool on Levels 3-4, with the prediction that source quality improves most on domains where the base LLM lacks domain-specific knowledge about which sources are authoritative (e.g., a general-purpose LLM may not know that CoinMarketCap is a reliable crypto data source but some random crypto blog is not).

Cross-temporal stability analysis of model rankings. The paper reports results from a two-week window (1,272 events) and acknowledges that "as FutureX progresses and the test size grows, the standard deviation is expected to decrease" (Section 3.4.2). But statistical power is not the only concern: event distributions may shift week-to-week (the paper notes fluctuations in prediction market event volume, Figure 3 caption), and model performance may vary with those shifts. A methodologically important follow-up would compute the rank correlation of the leaderboard across consecutive weeks after accumulating several months of data. If Grok-4 ranks first in week 1 but fifth in week 3, the benchmark is too noisy to support reliable comparisons regardless of sample size. If rankings are stable (rank correlation > 0.8 across weeks), the benchmark can be trusted as a reliable evaluation instrument. This analysis would also reveal whether certain models are more sensitive to event-distribution shifts than others — a finding that would itself be informative about model robustness. The paper has the infrastructure to perform this analysis as data accumulates; the gap is one of reporting rather than capability.

Adversarial robustness of deep research agents to fake web content. The out-of-benchmark case study on fake websites (Section 5.2) demonstrates that GPT-o3 Deep Research, DouBao Deep Research, and Qwen3-235B Deep Research were "consistently misled" by fabricated websites across all five tested scenarios, while Gemini-2.5-Pro Deep Research remained unaffected — even refusing to cite the fake site when explicitly prompted. This is a striking result from a small pilot (five scenarios, four models) that demands systematic follow-up. A concrete research program would: (1) scale the fake-website generation pipeline to produce 50-100 adversarial websites across all 11 FutureX domains using the iterative LLM-assisted refinement process described in the paper; (2) evaluate all 25 FutureX models on their susceptibility, measuring both whether the fake content appears in the agent's final prediction and whether the agent cites the fake source in its reasoning; (3) correlate susceptibility with other measured properties (search count, source reliability planning score, planning comprehensiveness) to identify what architectural or behavioral features confer robustness. The finding that Gemini was resistant — possibly due to "backend statistics or detailed domain-level credibility assessments" — suggests that search-engine-level signals (domain age, PageRank, SSL certificate validity) may provide a robustness layer that agent-level source evaluation cannot replicate. This is both a safety contribution (quantifying an underexplored attack surface) and a capability contribution (identifying what makes an agent robust).

Domain-specific fine-tuning for forecasting using FutureX as a training signal. The domain-specific results (Figures 9, 10) show that model strengths are non-uniform: GPT-family models excel at Crypto and Technology, DouBao-Seed1.6 excels at Finance and Business, DeepSeek-V3 excels at Politics. This suggests that no single model is Pareto-optimal across all domains, and that domain-specialized agents might outperform generalists. A concrete experiment would use FutureX's daily event stream as a training signal: for each domain, collect the agent's predictions, the retrieved ground truth (after the resolution delay), and the planning traces (where accessible, as in SmolAgent). Fine-tune a base LLM on the (question, retrieved-context, correct-answer) tuples, effectively distilling the search-and-reasoning process into model weights for that domain. The evaluation would compare the domain-fine-tuned model against the generalist model on held-out FutureX events in the same domain, testing whether specialization provides gains beyond what generalist search can achieve. The key metric is whether the fine-tuned model can match search-augmented performance without search in its specialized domain — effectively amortizing the cost of search into pretraining, which would have direct implications for latency-sensitive forecasting deployments.

Practical Applications and Downstream Use Cases

Financial analysis and earnings forecasting. The out-of-benchmark study comparing LLMs against Wall Street analysts on S&P 500 Q2 2025 earnings predictions (Section 5.1) provides the most direct evidence of practical viability: leading Think&Search models achieved win rates of 37.5% (revenue) and 32.3% (EPS) against professional sell-side analyst consensus estimates, with Gemini-2.5-pro achieving the lowest MAPE for revenue and GPT-o3 achieving the lowest MAPE for EPS. These are not numbers that justify replacing human analysts — no model exceeded a 50% win rate — but they demonstrate that LLM agents are entering the range where they can serve as a supplementary signal in analyst workflows: flagging cases where the model's prediction diverges substantially from consensus, providing a second opinion that might catch analyst herd behavior, or prioritizing which companies deserve deeper human analysis. The practical deployment model is not "replace the analyst" but "give the analyst an AI forecasting assistant that occasionally spots things they missed." The 30-minute per-question runtime constraint used in FutureX is actually generous for this use case — earnings forecasts are made quarterly, not daily, so deeper research per company is feasible.

Risk assessment and early warning systems. FutureX's domain coverage (politics, economics, health, weather, space) maps directly onto the information needs of organizations that monitor emerging risks: insurance companies assessing climate-related exposure, supply chain managers tracking geopolitical instability, public health agencies monitoring outbreak indicators. The benchmark's daily update cadence and multi-domain scope demonstrate that a single automated pipeline can continuously monitor diverse risk indicators and produce structured forecasts — a capability that currently requires teams of human analysts scanning different information sources. The finding that search-augmented models substantially outperform base LLMs on open-ended prediction tasks (Levels 3-4) suggests that the key infrastructure investment for such systems should be in retrieval and synthesis capabilities, not just in larger base models. The 97% answer acquisition success rate demonstrates that automated ground-truth retrieval is reliable enough for production use — a risk monitoring system could automatically verify its past predictions and surface systematic errors without human auditing.

Competitive intelligence and market monitoring. The template-based question generation system (Section 3.2.2) is directly transferable to corporate competitive intelligence workflows. A company tracking competitor product launches, pricing changes, or market share shifts could adapt FutureX's architecture: define templates for each competitor's known data sources (app store rankings, job postings, patent filings, earnings call transcripts), randomize variables to generate daily monitoring questions, run a research agent to gather current evidence, and produce structured forecasts with automatic verification when ground truth becomes available. The paper's finding that model performance varies substantially by domain (GPT models excel at Technology, DouBao-Seed1.6 excels at Finance) means that an effective deployment would likely use domain-routed model selection — different backbone models for different monitoring targets — rather than a single generalist agent. The benchmark's infrastructure (daily questions, automated answer retrieval) provides a template that could be replicated with proprietary data sources.

Agent capability auditing and procurement decisions. For organizations deciding which commercial LLM API or agent framework to adopt for forecasting-related tasks, FutureX provides an independently maintained, contamination-free evaluation that maps directly onto the intended use case. Unlike static benchmarks (which may be gamed or saturated) or vendor-provided benchmarks (which have obvious conflicts of interest), FutureX's live, prospective design means that today's leaderboard reflects actual forecasting capability against events whose outcomes were unknown at prediction time — exactly the condition under which a deployed forecasting system would operate. The domain-specific radar charts (Figures 9, 10) enable domain-informed procurement: an organization focused on financial forecasting can weight Finance & Economy performance more heavily, while a sports analytics company can focus on the Sports domain results. The finding that model rankings are not uniform across domains (Section 4.3: "different models have different strengths") means that a single "best model" recommendation is inappropriate — FutureX enables domain-matched selection that generic benchmarks cannot support.