STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence

ArXiv: 2510.24693

🎯 Pitch

Existing audio benchmarks largely test text-recoverable semantics, but STAR-Bench reveals a stunning collapse: caption-only answering drops model accuracy by 31.5% on temporal and 35.2% on spatial tasks. The benchmark redefines audio intelligence as reasoning over dynamics in time and 3D space, exposing that even the best closed-source models fail 84% of the time due to fine-grained perceptual bottlenecks rather than reasoning gaps.

---

1. Executive Summary

This paper introduces STAR-Bench, a benchmark for evaluating what it formalizes as audio 4D intelligence—reasoning over sound source dynamics in time (1D) and three-dimensional space (3D)—across 19 large audio-language and omni-language models. The benchmark couples a Foundational Acoustic Perception setting (procedurally synthesized pure tones and physics-simulated binaural audio probing six attributes under absolute range and relative discrimination sensitivity) with a Holistic Spatio-Temporal Reasoning setting (segment reordering for continuous processes and discrete event scripts, and spatial tasks spanning static localization, multi-source relations, and dynamic trajectory tracking). Evaluating 19 models reveals that existing benchmarks largely test text-recoverable semantics, while STAR-Bench induces far larger drops under caption-only answering (−31.5% temporal, −35.2% spatial), and exposes a capability hierarchy where closed-source models like Gemini 2.5 Pro are bottlenecked by fine-grained perception (84% of temporal errors) while open-source models lag across perception, knowledge, and reasoning, establishing that test-time audio reasoning remains deeply challenged only when tasks require integrating linguistically hard-to-describe acoustic cues with physical world knowledge.

2. Context and Motivation

The Core Problem: Audio Benchmarks Test Semantics, Not Perception

The fundamental problem STAR-Bench addresses is that existing audio benchmarks evaluate what language models can read about audio, not what they can hear. This distinction is critical because it means the field has been measuring—and celebrating—progress on a proxy task rather than on genuine auditory intelligence.

The paper provides direct quantitative evidence for this claim in Figure 1 (left) and Section 1. The authors take two recent, representative benchmarks—MMAU (test-mini) and MMAR—and conduct a simple but revealing experiment. They use Gemini 2.5 Pro to generate detailed textual captions of the audio samples from these benchmarks. Then they evaluate the same Gemini 2.5 Pro on the benchmark questions, but using only the captions instead of the raw audio. The result: accuracy drops by only 5.9% on MMAU and 9.0% on MMAR compared to when the model processes the actual audio. As the paper states:

"existing benchmarks predominantly focus on coarse-grained semantic content, which is audio information that can be distilled into textual descriptions with minimal loss."

This is a damning finding. It means that current benchmarks are effectively testing a model's ability to answer questions about text descriptions of audio—which a powerful language model can do quite well—rather than its ability to perceive and reason about the acoustic signal itself. The implication is that much of the field's reported progress in "audio understanding" may be progress in language understanding of audio captions, not in auditory perception.

The paper formalizes this gap through a clear contrast. Existing tasks—automatic speech recognition, sound event classification, simple question answering about "what sound is this?"—all map cleanly onto text. A human can write "a dog barking" or "a car honking" and an LLM can answer questions about it. But consider what humans do effortlessly that text cannot capture: judging how full a container is from the changing pitch of pouring water; inferring whether a vehicle is approaching or receding from subtle Doppler shifts; determining the spatial layout of multiple simultaneous sound sources in a room. These abilities depend on linguistically hard-to-describe acoustic cues—fine-grained patterns in frequency, intensity, phase, and temporal dynamics that resist simple verbal encoding.

The right panel of Figure 1 illustrates the paper's diagnostic framework: solving tasks on STAR-Bench requires synthesizing three core abilities—fine-grained perception, world knowledge (physics and common sense), and multi-step reasoning—and a failure in any one pillar leads to an incorrect answer. This tripartite decomposition is important because it allows the paper to diagnose where specific models fail, not just that they fail.

Why This Matters: Real-World Impact and Theoretical Significance

Real-world impact. The paper situates audio 4D intelligence within concrete application domains that depend on perceptual understanding beyond text. In embodied AI and robotics, agents must integrate fine-grained auditory cues to interact naturally with their surroundings—using sound to infer an object's trajectory, monitoring subtle machine operations for anomalies, or navigating based on auditory landmarks. These are not niche scenarios; they are fundamental to operating in unstructured physical environments where visual information may be occluded, ambiguous, or unavailable.

The safety dimension is equally important. The paper gives the example of a vehicle approaching from behind: a human can estimate its trajectory and distance from auditory cues alone, even without seeing it. An AI system that can do the same—or an embodied agent that knows when a sound source is moving toward it versus away—acquires a crucial survival capability. These competencies are not captured by asking a model to transcribe speech or identify that "a siren is playing."

Theoretical significance: formalizing audio 4D intelligence. Beyond practical impact, the paper makes a conceptual contribution by explicitly defining a new evaluation paradigm. Audio 4D intelligence is:

"the ability to perform deep reasoning over the dynamics of sound sources in time (1D) and three-dimensional space (3D), grounded in an understanding of the physical world."

This definition is significant because it moves the goalposts for audio AI from linguistic comprehension to physical understanding. The "grounded in an understanding of the physical world" clause is not decorative—it means that to succeed, a model must apply principles like the Doppler effect (frequency shifts with relative motion), the inverse-square law (intensity falls as 1/r² for point sources), fluid dynamics (turbulence and resonance patterns during pouring), thermodynamics (irreversible state changes in boiling or frying), and causal reasoning (one event triggering another). These are all properties of the physical world that manifest in sound but cannot be recovered from text captions.

This framing also draws a direct parallel to how the vision community has long distinguished between object recognition (classifying "what is in this image") and scene understanding (inferring 3D structure, spatial relationships, and dynamics). The audio community, the paper argues, lacks an analogous benchmark for the latter, and STAR-Bench aims to fill that gap.

Where Prior Approaches Fall Short

The paper identifies specific limitations in prior audio benchmarks across several dimensions, summarized in Table 1. The critique is systematic rather than dismissive—the authors acknowledge that existing benchmarks serve their purpose but argue they address a fundamentally different (and easier) problem.

Benchmarks that test semantics, not perception. AIR-Bench (Yang et al., 2024) and AudioBench (Wang et al., 2024) focus on traditional tasks: automatic speech recognition, spoken question answering, and audio captioning. These are valuable but, as the caption-only experiment shows, they test whether a model can produce or comprehend text about audio, not whether it can perform fine-grained auditory reasoning. The paper does not claim these benchmarks are useless—rather, they measure a different, lower-level capability that the field has largely solved.

Benchmarks that scratch the surface but don't go deep. The paper gives a more detailed critique of MMAU (Sakshi et al., 2025), MMAU-Pro (Kumar et al., 2025), and MMAR (Ma et al., 2025), which are the most recent and ambitious audio reasoning benchmarks. While these benchmarks do include temporal and spatial questions, the paper argues their coverage is "limited in both scale and depth." Specifically:

• Temporal tasks in prior benchmarks mainly involve identifying the timing or ordering of events—for example, "when does the dog bark?" or "which event comes first?" These are perceptual-layer tasks that can be solved by local timing cues. STAR-Bench's temporal reasoning, by contrast, requires understanding physical principles or causal dynamics across segments—inferring how a process evolves over time or how one event implies another. A model cannot simply detect that sound A precedes sound B; it must understand why A must precede B based on physics, function, or causality.

• Spatial tasks in prior benchmarks are often restricted to single-source localization, and the paper makes a pointed observation: "many items do not necessitate meaningful use of stereo cues (e.g., simple arriving vs. departing judgments)." In other words, a model might correctly answer "is the car getting closer or farther?" using only monaural loudness cues, without exploiting binaural information at all. STAR-Bench's spatial tasks, by contrast, include multi-source spatial relations (comparing positions of simultaneous sources), dynamic trajectory tracking (integrating spatial and temporal information), and explicit stereo-cue dependencies.

The caption-only benchmark auditing. The paper's most incisive critique of prior benchmarks is the caption-only experiment described above. By showing that removing the audio and substituting text captions barely degrades performance, the paper demonstrates that these benchmarks are effectively testing a model's language capabilities rather than its audio capabilities. This is not just a theoretical concern—it means that a pure text model with good reading comprehension could potentially score well on these benchmarks without any audio processing capability at all, making them invalid as measures of audio intelligence.

The channel-averaging problem. A subtler but equally important critique concerns the technical preprocessing pipeline used by most current audio-language models. The paper demonstrates (Figure 3) that the common practice of averaging multi-channel audio into mono—which is standard in models ranging from Qwen-2.5-Omni to GPT-4o-audio-preview—destroys spatial information. The experiment is elegant: construct 20 pseudo-stereo signals by assigning a sound to the left channel and its additive inverse to the right channel. A human listener easily identifies the sound (the left and right ear signals cancel in the air, not in the ear), but all tested models output random guesses because channel averaging produces silence. The paper states:

"The result confirms their lack of explicit support for genuine stereo audio processing."

This is a hardware/architecture limitation masquerading as a capability gap. No amount of training data or model scale will teach a model to extract interaural time differences (ITD) and interaural level differences (ILD)—the primary binaural cues for spatial hearing—if those differences are destroyed before the model ever sees them.

How This Paper Positions Itself

STAR-Bench positions itself not as an incremental improvement to existing benchmarks but as a paradigm shift in what audio benchmarks measure. The key differentiators, articulated in Table 1 and throughout Section 2, are:

1. Deep reasoning, not shallow perception. The benchmark explicitly requires physical and causal reasoning over temporal and spatial dimensions, going beyond event ordering and localization to demand understanding of why events occur in a particular sequence or how spatial relationships manifest in acoustic signals.

2. Linguistically hard-to-describe cues. The paper's core empirical claim is that STAR-Bench's tasks depend on acoustic information that cannot be recovered from text captions. This is validated by the caption-only degradation numbers (−31.5% temporal, −35.2% spatial, versus −5.9% and −9.0% on prior benchmarks), which the paper interprets as evidence that its benchmark genuinely measures audio perception rather than language comprehension.

3. Hierarchical task design. Rather than a flat collection of questions, STAR-Bench is organized into a two-level hierarchy: Foundational Acoustic Perception (quantitative, controlled, probing basic perceptual limits) and Holistic Spatio-Temporal Reasoning (complex, real-world, requiring synthesis of perception, knowledge, and reasoning). This structure is deliberate—the foundational level identifies what a model can perceive, while the holistic level tests whether it can use that perceptual information for complex reasoning. This allows diagnostic insights (e.g., "model X fails temporal reasoning because it can't perceive fine pitch differences") rather than just aggregate scores.

4. Multi-audio reasoning. Many of STAR-Bench's tasks require comparing, integrating, or grounding information across multiple audio clips—a capability that the paper notes is largely absent from prior benchmarks and that the ablation study (Figure 9) shows is a major weakness in open-source models.

5. Rigorous human validation. The data curation pipeline (four stages: taxonomy construction, AI-assisted filtering, human annotation with inter-annotator consensus, and final validation through human performance) is designed to ensure that every task is unambiguous, well-posed, and reliably solvable by human experts. The final stage—where items are retained only if at least two-thirds of domain experts independently solve them correctly—is particularly important because it establishes a human performance ceiling that models can be measured against, and guarantees that poor model performance reflects genuine capability deficits rather than flawed benchmark items.

6. Two complementary input modes for spatial audio. Recognizing that most current models cannot process true stereo, STAR-Bench evaluates spatial tasks under both native input (the model's default preprocessing, which typically channel-averages) and channel-wise input (left and right channels presented as separate audio streams with explicit textual instructions). This dual-mode design serves as a built-in ablation: it distinguishes between models that fail spatial tasks because their preprocessing destroys spatial information (native input failures) versus models that lack the reasoning capability even when binaural information is preserved (channel-wise input failures). As Figure 18 demonstrates, different models fail for different reasons, and the dual-mode design enables this diagnosis.

In summary, the paper positions STAR-Bench as filling a specific and significant gap: the absence of a benchmark that measures whether models can perform physically grounded reasoning over fine-grained, linguistically hard-to-describe acoustic cues in time and 3D space—capabilities that are essential for embodied AI and that current benchmarks, by testing largely text-recoverable semantics, systematically fail to evaluate.

3. Technical Approach

3.1 Reader Orientation (Approachable Technical Breakdown)

STAR-Bench is a hierarchically structured evaluation benchmark—not a model or a system—that tests how well AI models can perceive and reason about fine-grained acoustic properties and dynamic spatio-temporal relationships in audio. It solves the problem of diagnostic blindness in prior audio benchmarks: by coupling tightly controlled perceptual tests (synthetic, parameterized sounds) with complex real-world reasoning tasks (reordering shuffled audio segments, tracking sound movement, comparing spatial positions), the benchmark isolates whether a model fails because it literally cannot hear subtle differences in pitch or loudness, or because it cannot apply physical reasoning to sounds it can perceive. The "shape" of the solution is a two-level diagnostic pyramid: a foundation layer that quantitatively profiles six core perceptual abilities under absolute and relative regimes, and a holistic layer that demands models synthesize fine-grained perception, world knowledge, and multi-step reasoning to solve temporally and spatially complex tasks built from real-world audio curated through rigorous human validation.

3.2 Big-Picture Architecture (Diagram in Words)

The STAR-Bench benchmark has five major components, connected through a curation-and-evaluation pipeline rather than a runtime inference system:

1. Taxonomy Construction Framework — a human-expert and LLM-assisted process that defines the hierarchical task space (foundational perception attributes, temporal reasoning subcategories, spatial reasoning subcategories) and specifies the logical principles each task type probes.

2. Audio Synthesis Engine (Foundational Tasks) — a procedural generation pipeline that produces fully parameterized pure tones (for pitch, loudness, duration) and physics-simulated binaural recordings (for azimuth, elevation, distance) using the Pyroomacoustics room acoustics simulator, providing ground-truth control over every acoustic parameter.

3. Real-World Audio Curation Pipeline (Holistic Tasks) — a four-stage process (taxonomy-driven sourcing from audio libraries, AI-assisted automated filtering, human annotation with inter-annotator cross-validation, and final validation through human performance evaluation) that converts raw audio from datasets like Clotho, FSD50K, and STARSS23 into benchmark questions with verified correctness and unambiguous answer keys.

4. Question Generation Framework — standardized multiple-choice question templates for each task type, with controlled difficulty levels (e.g., 4–6 discrimination sensitivity levels per attribute with precise $\Delta$ values, option randomization via CircularEval for robust evaluation).

5. Evaluation Protocol — a dual-metric scoring system (Average Accuracy across runs, All-Correct Rate for reliability) with two complementary audio input modes (native preprocessing, channel-wise presentation) and perturbation-based robustness testing (cyclic option rotation for foundational/spatial tasks, segment order shuffling for temporal tasks).

Information flows as follows: the taxonomy defines what to test → the synthesis engine or curation pipeline produces audio-question-answer triplets → the evaluation protocol standardizes how models are queried and scored → the dual-metric output (AA, ACR) provides both aggregate and reliability diagnostics per task category.

3.3 Roadmap for the Deep Dive

• First, the formal definition of audio 4D intelligence and the motivation for the two-level benchmark structure (Foundation + Holistic), because this establishes the conceptual framework that every subsequent design decision serves.
• Second, the foundational acoustic perception tasks—the six attributes, the two sub-regimes (absolute range and relative discrimination sensitivity), and the procedural synthesis methodology—because these provide the quantitative baseline of "what can the model physically hear?" that is prerequisite for understanding holistic task failures.
• Third, the temporal reasoning task design—the segment reordering setting, the categorization into continuous processes and discrete event sequences, and the logical principles governing each subcategory—because temporal reasoning is the first dimension of 4D intelligence and its design illustrates the benchmark's philosophy of testing deep understanding rather than surface patterns.
• Fourth, the spatial reasoning task design—the three subcategories (static localization, multi-source relations, dynamic trajectory tracking), the critical channel-averaging problem, and the dual native/channel-wise input strategy—because spatial reasoning is the second dimension of 4D intelligence and the channel-wise design is a key methodological innovation that enables diagnostic insight.
• Fifth, the data curation pipeline—the four stages from taxonomy construction through human performance validation—because this is what ensures benchmark quality, and understanding the rigor is essential for assessing the validity of results.
• Sixth, the evaluation protocol—the robust evaluation methodology including CircularEval, dual metrics (AA and ACR), and the human baseline establishment—because this determines how scores are computed and what they mean.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a benchmark design and empirical evaluation paper whose core idea is that measuring audio 4D intelligence requires tasks that (a) depend on linguistically hard-to-describe acoustic cues, (b) demand synthesis of fine-grained perception with physical world knowledge, and (c) isolate perceptual failures from reasoning failures through a hierarchical, diagnostic task structure.

---

What Are We Testing? The Formal Definition of Audio 4D Intelligence

The paper defines its target construct operationally: audio 4D intelligence is the capability to perform deep reasoning over sound source dynamics in time (1D) and three-dimensional space (3D), grounded in an understanding of the physical world. This definition is not merely descriptive—it imposes specific requirements on benchmark design.

First, the "deep reasoning" clause excludes simple detection or classification tasks. A model that identifies "a car engine is present" demonstrates 0-dimensional audio understanding (what is the sound source?). A model that identifies "the car engine starts, then revs, then idles" demonstrates 1-dimensional temporal understanding (what is the sequence of events?). But audio 4D intelligence requires: can the model infer that the car is approaching from the left, passing by, and receding to the right based on Doppler shifts (frequency) and interaural level differences (loudness)? This requires integrating perceptual cues (pitch change over time, loudness change in each ear) with physical knowledge (the Doppler equation, the inverse-square law for sound intensity decay with distance).

Second, the "grounded in an understanding of the physical world" clause means that tasks cannot be solved by pattern matching against training data. The paper deliberately constructs tasks where the correct answer depends on applying physical principles—fluid dynamics for pouring sounds, thermodynamics for cooking sounds, acoustics for spatial localization—that transcend any particular sound event instance. A model trained on thousands of "car passing" recordings might learn correlational patterns without understanding the underlying physics, but the benchmark's task diversity (five temporal subcategories, three spatial subcategories, six perceptual attributes with multiple difficulty levels) and its deliberate departure from conventional audio QA formats are designed to probe whether models have acquired genuine physical understanding or merely dataset-specific heuristics.

Third, the hierarchical two-level structure (Foundation → Holistic) is motivated by diagnostic necessity. If a model fails a holistic temporal reasoning task—say, reordering segments of water being poured into a glass—the failure could stem from (a) inability to perceive the subtle pitch change as the container fills (perception failure), (b) lack of knowledge that water pouring into an empty container produces a lower-pitched sound than pouring into a nearly-full one (knowledge gap), or (c) inability to chain the perceptual observations into the correct sequential inference (reasoning failure). The foundational perception tasks are designed to disambiguate these causes: by testing the model's basic pitch discrimination sensitivity independently, an evaluator can determine whether the holistic failure is a perception problem or a knowledge/reasoning problem. This diagnostic architecture is the paper's key methodological contribution—it transforms benchmark scores from opaque aggregates into interpretable capability profiles.

---

Foundational Acoustic Perception: Six Attributes, Two Regimes

The foundational perception level is motivated by an explicit analogy that appears in Section 3.1:

"Just as a solid understanding of grammar is required for writing a complex narrative, a model must be able to accurately perceive these core attributes before it can reason about the dynamic, spatial relationships of sound sources in the physical world."

The paper operationalizes this through six attributes spanning two domains—three non-spatial (pitch, loudness, duration) and three spatial (azimuth, elevation, distance)—each tested under two evaluation regimes: Absolute Perception Range and Relative Discrimination Sensitivity. Table 3 in the paper provides the exact parameter ranges and example question templates for every attribute-by-regime combination.

Non-Spatial Attributes: Pitch, Loudness, Duration

For pitch and loudness, the paper adapts the design of human audiometry tests—the clinical procedure for measuring hearing thresholds—to create what it calls an "audiogram" for language models. This is a deliberate methodological choice: audiometry is well-validated in audiology for characterizing the limits of auditory perception, and adapting it provides a principled framework for mapping a model's perceptual range.

Absolute Perception Range (Pitch and Loudness): The paper synthesizes pure sine waves with frequencies ranging from 125 Hz to 8000 Hz and loudness levels from −10 dB HL to 110 dB HL. The task format is a two-alternative forced-choice with a "not present" option: the model hears an audio clip divided into two halves, and must identify whether a clear beep occurs in the first half, the second half, or is not present at all. This "not present" option is critical—without it, a model could achieve 50% accuracy by always guessing, and a 50% score would be uninformative. The three-way choice sets the random baseline at 33.3%, making above-chance performance detectable while still being conservative.

The frequency range (125 Hz to 8000 Hz) spans roughly the human vocal range plus the fundamental frequencies of most musical instruments and environmental sounds, but notably excludes very low frequencies (infrasound below 20 Hz) and very high frequencies (above 8000 Hz, where human hearing typically rolls off). The loudness range (−10 dB HL to 110 dB HL) extends from below the threshold of normal hearing to uncomfortably loud levels. The paper does not explicitly justify these exact endpoints, but they appear chosen to span the ecologically relevant range while avoiding extreme values where synthesis artifacts might confound results.

What the audiogram produces: For a given model, a two-dimensional map over (frequency, loudness) where each cell indicates the proportion of correct detections. Greener regions in Figure 8 indicate higher accuracy; the spatial extent of green coverage represents the model's perceptual range.

Relative Discrimination Sensitivity (Pitch, Loudness, Duration): This regime investigates Just Noticeable Difference (JND)-analogous thresholds for each attribute. For each of the three non-spatial attributes, the paper constructs 4–6 difficulty levels with progressively larger stimulus differences $\Delta$, ranging from subtle variations perceptible to humans to significant real-world changes. Level 1 is a control condition: for non-spatial attributes, it presents identical sounds ($\Delta = 0$), and a model scoring above chance here indicates reliance on non-acoustic cues (e.g., position bias, spurious textual patterns) rather than genuine discrimination.

The task format is comparison-based: "Which sound has a higher pitch: the first sound, the second sound, or are they the same?" The inclusion of a "same" option is methodologically important—it prevents the model from defaulting to a forced comparison when it cannot detect any difference, and establishes a true random baseline at 25% (four options: first higher, second higher, same, unable to determine).

The specific difficulty levels for each attribute are:

• Pitch: $\Delta = 0, 50, 100, 200, 400, 1200$ cents, where a cent is 1/100th of a semitone (so 1200 cents = 1 octave). The finest gradated levels (50–200 cents) probe sensitivity to microtonal differences; the larger levels (400–1200 cents) test discrimination of clearly distinct pitches. Human musicians can typically discriminate differences of ~5–10 cents under ideal conditions, and non-musicians ~25–50 cents, so level 2 (50 cents) is at the threshold of non-expert human perception.

• Loudness: $\Delta = 0, 4, 8, 12, 24, 48$ dB. A difference of 1 dB is approximately the just-noticeable loudness change for humans under controlled conditions; 3 dB is a clearly audible but small change; 10 dB is perceived as roughly twice as loud. Thus level 2 (4 dB) probes clearly audible but small differences, while level 5 (24 dB) and level 6 (48 dB) probe discrimination of very pronounced loudness contrasts.

• Duration: $\Delta = 0, 20, 50, 100, 150, 200$ percent (where $\Delta$ represents the proportional increase relative to the shorter sound). For example, a 20% difference means comparing a 1.0-second sound to a 1.2-second sound. Human temporal discrimination follows Weber's law (the JND is proportional to the base duration), with JNDs typically around 5–10% for durations above 200 ms, so the 20% level is above but near the human threshold.

What the sensitivity curves produce: For each attribute and each model, a function mapping $\Delta$ to accuracy (or All-Correct Rate, as shown in Figure 8, second row). The steepness of the curve characterizes discrimination sensitivity: a model with good sensitivity should show near-chance performance at small $\Delta$ and near-perfect performance at large $\Delta$, with a sharp transition between regimes. A flat curve indicates inability to use the acoustic attribute regardless of how large the difference is.

Spatial Attributes: Azimuth, Elevation, Distance

For the three spatial attributes, the paper uses Pyroomacoustics, a physics-based room acoustics simulator (Scheibler et al., 2018), to generate binaural audio with controlled ground-truth source positions. This is the key methodological choice that distinguishes the foundational spatial tasks from prior benchmarks: rather than relying on recorded audio with unknown or approximated source positions, every spatial parameter is precisely specified and reproducible.

The simulation setup as described in Appendix B.2.1 is detailed and full of non-obvious design decisions:

• Room acoustics: Three rectangular rooms are simulated—small (4.0 × 3.5 × 2.8 m), medium (8.0 × 6.0 × 3.5 m), and large (20 × 15 × 8 m)—each with a frequency-independent wall absorption coefficient of 0.25. The rooms span typical domestic to large-hall sizes, providing diverse reverberation characteristics that affect spatial perception. The absorption coefficient 0.25 is moderately reflective (0.0 is perfectly absorptive/anachoic, 1.0 is perfectly reflective), producing realistic but not extreme reverberation.

• Image-source modeling: Reflections are modeled up to order 10, meaning the simulator computes up to 10th-order reflections of the sound source off room surfaces. This is computationally intensive but necessary for accurate binaural rendering, especially for distance perception where the direct-to-reverberant energy ratio is a primary cue.

• Binaural rendering: The simulation uses a measured SOFA HRTF (Head-Related Transfer Function) from the MIT KEMAR dummy head with "normal pinna" configuration, interpolated to order 12 with 1,000 sampling points. The HRTF encodes how the pinna (outer ear), head, and torso filter incoming sound depending on source direction, and it is the basis for elevation and front-back localization (which monaural spectral cues cannot resolve). Using a measured rather than synthetic HRTF is important for ecological validity; the "normal pinna" specification means the artificial ear approximates an average human outer ear shape.

• Early/late energy mix for distance cues: The authors preserve the first 80 ms of each binaural room impulse response (BRIR) and attenuate the late tail by a factor of 0.5. This is not physically realistic—in real rooms, the late reverberant tail decays exponentially, not abruptly at 80 ms—but it is a deliberate choice to emphasize distance-dependent direct-to-reverberant ratio while avoiding excessively long impulse responses that would extend audio durations. The 80 ms cutoff approximately corresponds to the precedence effect window in human hearing, where early reflections are perceptually fused with the direct sound.

• Source signals: Three curated audio clips ("alarm," "applause," "telephones") serve as dry source signals, convolved with the computed BRIRs. Using multiple source types prevents models from memorizing specific sound-event-to-position mappings.

• Global peak normalization is applied across batches to avoid clipping while preserving inter-position level differences, ensuring that relative loudness cues for distance are maintained.

Absolute Perception Range (Spatial): The paper discretizes each spatial attribute into fixed partitions for multiple-choice questioning:

• Azimuth (0°–360°): The full circle is partitioned into four 90° quadrants—Front-Right (0°–90°), Back-Right (90°–180°), Back-Left (180°–270°), Front-Left (270°–360°)—generated by sampling eight specific angles {30°, 60°, 120°, 150°, 210°, 240°, 300°, 330°}. The 30° offset from cardinal directions is intentional: sounds directly at 0°, 90°, 180°, or 270° create ambiguous interaural cues (perfectly symmetrical for front/back, or maximal interaural difference for left/right), so the offset angles produce more typical binaural patterns.

• Elevation (−90° to +90°): Partitioned into three bins—Above ear level, At ear level, Below ear level—sampled from six angles {−75°, −45°, −15°, 15°, 45°, 75°}. Excluding 0° (exactly ear-level) avoids the ambiguity of whether "at ear level" means precisely 0° or approximately 0°.

• Distance (0–10 m): Partitioned into three bins—Near (0–3 m), Medium (3–8 m), Far (>8 m)—with non-uniform sampling that emphasizes near-field distances (6 utterances per meter for 1–7 m, 3 per meter for 8–10 m). The denser near-field sampling is justified because most distance-dependent acoustic cues (direct-to-reverberant ratio, high-frequency attenuation) change most rapidly at close distances and asymptote at far distances.

Relative Discrimination Sensitivity (Spatial): Rather than comparing two sounds within a single audio clip as in the non-spatial case, spatial relative discrimination presents two separate audio clips and asks the model to compare their spatial parameters. This design choice is forced by the constraints of binaural rendering: presenting two sources at different positions simultaneously would create a complex spatial mixture, making it impossible to attribute perceptual differences to specific source positions. By presenting the two spatial configurations sequentially and labeling them "Audio 1" and "Audio 2," the task isolates spatial discrimination from simultaneous sound source segregation.

The difficulty levels for spatial attributes are:

• Azimuth difference: $\Delta = 30°, 60°, 90°, 120°, 150°, 180°$ (6 levels). Human minimum audible angle (MAA) for broadband sounds in the frontal horizontal plane is approximately 1°–2° directly ahead, degrading to ~10° at lateral positions, so even the smallest difference (30°) is clearly above the human threshold. The "consider differences of less than 45° as the same" instruction in the prompt accounts for the fact that the categorical task (same/different) is coarser than continuous angular discrimination.

• Elevation difference: $\Delta = 15°, 90°, 120°, 150°$ (4 levels). The 15° minimum is substantially above human elevation MAA (~4° for broadband sounds), but elevation perception depends heavily on high-frequency spectral cues from the pinna, which may not be well-captured by the HRTF processing or by models' audio encoders.

• Distance difference: $\Delta = 1–2, 4–5, 6–7, 8–9$ meters (4 levels), with a "consider differences of less than 3 meters as the same" instruction. Human distance perception is notoriously poor compared to angular localization—Weber fractions for distance are typically 10–20% for sources beyond 1 meter—so the 3-meter "same" threshold is relatively lenient.

Design Synthesis: Why This Structure?

The foundational perception design embodies several deliberate methodological choices:

1. Pure tones for non-spatial attributes, complex sounds for spatial: Pitch and loudness perception are most cleanly isolated using sinusoidal stimuli, which have a single frequency and controllable intensity without confounding spectral or temporal complexity. But spatial perception—especially elevation and distance—depends on spectral shaping by the HRTF, which requires broadband source signals to be effective. Using pure tones for spatial tasks would produce ambiguous or missing elevation cues, so the paper correctly switches to curated broadband sounds.

2. Absolute range before relative sensitivity: The audiogram-style absolute range tasks establish whether a model can detect or localize a sound at all under ideal conditions. If a model shows zero detection capability at certain frequency-loudness combinations, there is no point in testing its discrimination sensitivity at those combinations—the failure is at the detection level, not the discrimination level. The hierarchical ordering within the foundation level thus mirrors the Foundation → Holistic hierarchy: establish basic capability before testing finer gradations.

3. Human audiometry as template: By adapting the well-established methodology of clinical audiometry, the paper inherits decades of validation in human psychophysics. The audiogram format provides an intuitive visualization (Figure 8, first row), and the JND-based difficulty scaling provides a principled way to set discrimination thresholds rather than arbitrary $\Delta$ values.

4. "Unable to determine" as a universal option: Every question in the foundational tasks includes "Unable to determine" as a response option. This is methodologically important because it prevents models from being forced into incorrect answers when they genuinely cannot perceive the relevant acoustic cue, and it makes the task more ecologically valid—in real-world audio reasoning, recognizing uncertainty is often the correct behavior.

---

Temporal Reasoning: Segment Reordering with Physical Logic

The temporal reasoning tasks are built around a single, novel task format: Audio Segment Reordering. Given an audio event that has strong sequential uniqueness (the order matters), semantic clarity (the event identity is recognizable from sound alone), and logical universality (the ordering follows principles that generalize beyond specific instances), the event is segmented into three clips, the clips are shuffled, and the model must restore the original temporal sequence based solely on acoustic content.

The paper organizes these tasks into two meta-categories—Continuous Processes and Discrete Event Sequences—and five subcategories, each defined by a distinct logical principle that governs the correct ordering. This taxonomy is not arbitrary: it systematically enumerates the types of physical and causal knowledge that temporal audio reasoning requires.

Continuous Processes: Tracking Gradual Acoustic Evolution

Continuous processes involve a single, uninterrupted acoustic event where the sound evolves gradually over time according to physical principles. The model cannot rely on discrete event boundaries—it must track continuous changes in acoustic features and infer the direction of evolution.

Object Spatial Motion — This subcategory reconstructs the spatio-temporal trajectory of moving sound sources. The core acoustic cues are:

• Doppler effect: As a sound source approaches, the perceived frequency increases (compressed wavefronts); as it recedes, the frequency decreases (stretched wavefronts). The paper specifically notes that the Doppler effect produces "frequency shifts indicating relative velocity." For a source moving at constant speed past a stationary listener, the frequency shift is not linear—it changes most rapidly at the point of closest approach—so models must track a non-monotonic frequency trajectory.

• Inverse-square law: Sound intensity from a point source decreases as $1/r^2$ where $r$ is source-listener distance. Combined with the Doppler effect, this produces a characteristic loudness-and-pitch trajectory: for an approaching source, loudness increases while pitch rises; for a receding source, loudness decreases while pitch falls; at the point of closest approach, loudness peaks while pitch rapidly sweeps from above to below the source's rest frequency.

Examples from the paper include passing cars and airplanes—sources with distinctive, recognizable sounds whose trajectories follow predictable spatio-temporal patterns. The model must integrate two perceptual dimensions (pitch change, loudness change) and map them onto a physical interpretation (approach → peak → recession).

In-Situ State Evolution — This subcategory tracks the intrinsic evolution of a stationary object's state, governed by predictable trend patterns in the acoustic signal. Unlike object spatial motion, there is no spatial component—the sound source doesn't move—but its acoustic properties change systematically as its physical state evolves.

The paper enumerates four underlying physical principles that govern these trends:

• Fluid and Pneumatic Dynamics: Turbulence, resonance, and pressure changes produce characteristic acoustic signatures. The paper gives the examples of a toilet flushing and water being poured. In pouring, as a container fills, the air column above the liquid shortens, raising the resonant frequency—the pitch of the pouring sound increases. The model must detect this gradual pitch rise and recognize that higher pitch corresponds to a fuller container, establishing the temporal direction.

• Thermodynamic Processes: Irreversible state changes driven by heat produce monotonic acoustic trends. Examples include water boiling (increasing bubble formation rate, rising pitch of steam sounds) and food frying (transition from wet sizzling to dry crackling as moisture evaporates). The irreversibility is key—once water has boiled away, it doesn't spontaneously re-condense in the pan, so the acoustic trajectory has a clear direction.

• Energy Decay: After a single excitation, the sound energy decays due to resonant damping and friction. Examples include a bell's chime or an explosion's echo—the sound starts loud and bright (high-frequency components decay fastest) and gradually becomes quieter and mellower. The model must detect the amplitude envelope and spectral centroid descent to order segments.

• Biological Rhythms: Evolving physiological or emotional states produce acoustic trajectories, such as breathing becoming more rapid or labored, or vocalizations changing in pitch or intensity. These are the most complex cases because they follow statistical rather than deterministic patterns, but the paper includes them to probe whether models can learn the typical temporal profiles of biological sounds.

Discrete Event Sequences: Understanding Inter-Event Logic

Discrete event sequences involve multiple, distinct acoustic events whose temporal ordering is governed by function, convention, or causality rather than continuous physical evolution.

Tool and Appliance Operation — This subcategory tests whether the model understands the standardized operating procedure for tools and appliances. The examples include a microwave (button press → hum → beep) and a power drill (trigger pull → motor whine → screw engagement sound → release). The correct sequence follows from the tool's designed function: you press buttons before the microwave operates; you engage the drill before it drives the screw. This requires functional knowledge rather than acoustic pattern recognition—the sounds themselves do not contain the ordering information; the model must import knowledge about how tools work.

Daily Scene Scripts — This subcategory applies commonsense and contextual script knowledge to daily activities. Examples include brushing teeth (water running → brushing sound → spitting/rinsing) and drinking water (opening bottle → pouring → swallowing). These follow conventional sequences that any human would recognize from lived experience. The paper notes that these tasks require the model to apply "contextual script knowledge," a term from cognitive psychology (Schank and Abelson, 1977) referring to the stereotyped sequence of actions that constitute a familiar activity.

Event-Triggered Consequences — This subcategory applies causal reasoning: a trigger event produces an automatic and irreversible outcome. Examples include a firework explosion triggering a crowd cheering (social causation) or glass shattering (physical causation). The key property is that the consequence is temporally dependent on the trigger—the cheer always follows the explosion, never precedes it—and the dependency is causal rather than merely statistical. The model must recognize the causal relationship, not just the temporal adjacency.

Why Three Segments?

The paper standardizes on exactly three segments per temporal reasoning task. This is a deliberate design choice that balances several constraints:

• Two segments would create a binary choice (AB vs. BA) that could be solved by chance 50% of the time, requiring many more questions to achieve statistical power.
• Four or more segments would dramatically increase the combinatorial space (4! = 24 possible orderings), making the task potentially unsolvable even for humans and introducing combinatorial complexity that could mask perceptual failures.
• Three segments yields 3! = 6 possible orderings, with a random baseline of 16.7% (1/6), providing reasonable statistical power while remaining cognitively tractable. The six options are enumerated as multiple-choice responses <A> through <F>, with each option specifying a complete ordering (e.g., "clip 1 → clip 2 → clip 3").

The paper also implements a robustness measure for temporal tasks: each question is presented three times with different temporal segment orders. This means the model sees the same three audio clips but presented in different shuffled arrangements on each run. Consistent correct responses across runs indicate robust temporal reasoning; correct answers on only some runs suggest the model is sensitive to presentation order or partially solving the task.

---

Spatial Reasoning: From Localization to Dynamic Tracking

The spatial reasoning tasks are organized into three subcategories of increasing complexity, designed to probe progressively deeper levels of spatial understanding while accommodating the reality that most current models cannot natively process stereo audio.

The Channel-Averaging Problem: A Preprocessing Bottleneck

Before detailing the spatial tasks, the paper identifies a critical technical limitation that fundamentally constrains what current models can achieve. The experiment described in Section 3.2.2 and Figure 3 demonstrates the problem concisely:

"The common practice of these models is to average multi-channel audio into a mono signal, resulting in the loss of substantial spatial information."

The demonstration uses 20 pseudo-stereo signals constructed by assigning a sound event (e.g., bird chirping, dog barking) to the left channel and its additive inverse (the waveform multiplied by −1) to the right channel. For a human listener wearing headphones, this produces a peculiar but clearly perceptible spatial impression—the sound appears to come from inside the head or from an indeterminate location, but the sound event identity remains clearly identifiable because the left and right ear signals arrive at separate ears and do not cancel acoustically.

For a model that averages the two channels to mono before processing, however, the sum is $x(t) + (-x(t)) = 0$—complete silence. The model receives no audio information at all, and its performance collapses to random guessing (25% for four-option sound event classification, which is exactly what all tested models achieved). This is not a capability limitation—it is a preprocessing bug that makes spatial reasoning impossible regardless of model quality.

The implications are severe but often overlooked. Interaural time differences (ITD)—the difference in arrival time of a sound wavefront at the two ears—are the primary cue for horizontal-plane localization at frequencies below ~1500 Hz. Interaural level differences (ILD)—the intensity difference caused by head shadowing—dominate at higher frequencies. Both are destroyed by channel averaging. Any model that averages channels to mono is fundamentally incapable of extracting the binaural cues that human spatial hearing relies on, regardless of how sophisticated its architecture or how much training data it has seen.

The Dual Native/Channel-Wise Evaluation Strategy

To address this problem without requiring all evaluated models to support true stereo processing (which most do not), the paper introduces two complementary evaluation modes:

Native input: The model processes stereo audio using its default preprocessing pipeline, whatever that may be. For most models, this involves channel averaging and thus information destruction. Evaluating under native input measures the model's performance under standard deployment conditions—it answers the question "if someone uses this model off-the-shelf for spatial audio tasks, how well does it do?"

Channel-wise input: The left and right channels are presented as two separate mono audio streams, each with an explicit textual label: "Audio 1 is the left-ear channel and Audio 2 is the right-ear channel." The task prompt includes explicit instructions about the binaural nature of the recording. This mode preserves the binaural information (both channels are available to the model, albeit as separate inputs rather than a synchronized stereo stream), enabling an assessment of whether the model has any spatial reasoning capability when the raw information is accessible.

The channel-wise mode is explicitly framed as an ablation study:

"This configuration serves as an ablation study to examine whether current models have any spatial capability when the binaural information is preserved at the input."

By comparing native and channel-wise performance, an evaluator can distinguish between two failure modes: (a) the model fails because its preprocessing destroys spatial information (large native→channel-wise improvement), versus (b) the model fails because even with preserved binaural information, it cannot perform spatial reasoning (no improvement).

The paper acknowledges that channel-wise input is an imperfect solution:

"most existing models are not trained to handle multi-audio inputs. As a result, they consistently struggle to align channel representations and fail to make reliable use of interaural differences."

A model trained exclusively on mono audio may not have learned that two audio streams can represent left and right ear perspectives, or how to extract ITD and ILD from comparing the two streams. The channel-wise evaluation therefore represents an upper bound on what current models could achieve if their architectures supported binaural input—but not necessarily what they do achieve, since the training data may not have taught them binaural processing.

Three Spatial Reasoning Subcategories

Single-Source Static Localization — This is the most basic spatial task: identify the direction (or elevation, or distance) of a single, stationary sound source. The questions are structured similarly to the foundational spatial perception tasks but use real-world recordings rather than synthesized audio. Example: "Given that 0° is directly in front and the angle increases clockwise, which azimuth range is the sound most likely coming from?" The four options partition the full 360° circle into quadrants.

This subcategory establishes whether the model has basic spatial perception—can it determine where a sound is coming from? It is prerequisite for the more complex spatial reasoning tasks, and failures here indicate that the model cannot extract even coarse spatial information from audio, making more advanced spatial reasoning impossible.

Multi-Source Spatial Relation — This subcategory requires the model to determine the relative spatial relationships among multiple simultaneous sound sources. For example, comparing the positions of two speakers to decide which one is farther to the right, or determining whether two sounds come from the same direction or different directions.

The key advance over single-source localization is that the model must compare and relate spatial percepts rather than merely classify a single source location. This requires:

• Perceiving the spatial attributes of each source individually (requiring the ability to separate simultaneous sounds, known as auditory scene analysis or the "cocktail party problem").
• Retaining the spatial information for each source in working memory.
• Performing a relational comparison (farther vs. closer, left vs. right) that may involve continuous rather than categorical spatial reasoning.

Dynamic Trajectory Tracking — This subcategory introduces moving sound sources, requiring the model to integrate spatial and temporal information to reason about movement trajectories. Example: "What is the movement trajectory of the racing car?" Options include "From left to right," "From right to left," and "Remains unchanged."

This is the most challenging spatial task because it requires:

• Tracking the spatial position of a sound source over time (temporal integration of spatial percepts).
• Inferring the overall movement pattern from moment-to-moment position changes.
• Distinguishing true movement from stationary sources with fluctuating loudness (perceptual constancy).

The trajectory tracking task most directly instantiates the "4D" in audio 4D intelligence: it demands reasoning that spans both time (the trajectory unfolds over seconds) and 3D space (the position changes in azimuth, possibly elevation and distance).

---

Data Curation Pipeline: Four-Stage Quality Assurance

The curation pipeline for holistic reasoning tasks (Section 3.3) is one of the paper's most methodologically rigorous components. It spans four stages and involves multiple rounds of human and AI validation, with explicit quality gates at each transition. The pipeline is described for temporal and spatial tasks separately, though they share the same overall structure.

Stage 1: Taxonomy Construction and Data Sourcing

The taxonomy is built through a "collaborative process involving domain experts and the Gemini 2.5 Pro." The paper does not specify the exact division of labor between humans and the LLM, but the domain experts likely provided the initial conceptual framework and validated the LLM's expansions and refinements.

For temporal reasoning: Candidate data is sourced from two large-scale, real-world audio libraries: Clotho (Drossos et al., 2019), an audio captioning dataset with diverse environmental sounds, and FSD50K (Fonseca et al., 2022), a dataset of 51,197 Freesound audio clips labeled with 200 sound event categories. Both datasets are widely used in the audio understanding community, which means they are likely present in the training data of many evaluated models. The paper explicitly acknowledges this:

"Although the underlying audio data for the temporal tasks (e.g., FSD50K, Clotho) is commonly used for model pre-training, our novel task formulation of temporal reasoning deliberately departs from conventional audio QA formats."

This is an important methodological point: the benchmark's challenge comes from the task formulation, not the novelty of the audio data. Even if a model has heard these exact recordings during training, it was trained on tasks like "classify this sound" or "generate a caption for this audio," not "reorder these three shuffled segments." The novel task probes capabilities that standard training objectives do not directly optimize.

For spatial reasoning: Candidate data comes from STARSS23 (Shimada et al., 2023), an audio-visual dataset of spatial recordings with spatiotemporal annotations, plus audio sourced from the internet. STARSS23 is particularly valuable because it provides ground-truth spatial annotations (azimuth, elevation, distance) for sound events in real environments, enabling the creation of spatial questions with verified answer keys.

Stage 2: AI-Assisted Automated Filtering

The automated filtering is described as an "efficient three-stage funnel" that progressively filters unsuitable samples:

Sub-stage 1 (Basic Properties): Discard samples based on basic properties like duration and energy. For example, very short clips (less than ~1 second) may not contain sufficient temporal structure for reordering; very quiet clips may be dominated by noise; excessively long clips may make the reordering task practically unsolvable due to memory demands. The paper does not specify exact thresholds, but the principle is to remove obviously unsuitable candidates before incurring the cost of LLM-based analysis.

Sub-stage 2 (LLM Metadata Screening): An LLM (specifically DeepSeek-V3, Liu et al., 2024a) performs an initial screening based on textual metadata, without access to the audio itself. The LLM receives the task description and the metadata associated with each candidate (e.g., sound event labels from FSD50K, captions from Clotho) and decides whether the audio appears suitable for the reordering task. It also provides justifications for its decisions. The metadata-only approach is computationally efficient (no audio processing) and leverages the LLM's ability to reason about whether a described sound event has strong sequential uniqueness and logical universality—for example, "water pouring" can be judged as likely suitable based on text description alone, while "traffic noise" may be judged as unlikely suitable because it lacks clear sequential structure.

Sub-stage 3 (Multimodal Model Audio Screening): A powerful multimodal model (Gemini 2.5 Pro) receives the audio, the metadata, the LLM's outputs from sub-stage 2, and the full task criteria, and produces: (i) a binary judgment (applicable / not applicable), (ii) a quality score, (iii) a preliminary classification into continuous vs. discrete process or spatial subcategories, and (iv) a reasoning justification. The prompts used are provided in Appendix B.3.1 (Figures 10 and 11), and they are extensive—they specify the criteria of "strong sequence uniqueness, semantic clarity, and high logical universality" and ask the model to evaluate each candidate against these criteria.

The paper adopts a conservative filtering strategy: only samples explicitly marked as "not applicable" are discarded. Borderline cases are retained for human review. This is a deliberate choice to minimize false negatives—it is better to let humans review some unsuitable samples than to discard potentially valid ones based on an imperfect AI filter.

Stage 3: Human Annotation and Quality Control

Ten undergraduate annotators are recruited and trained to label the data using a professional annotation platform. The annotation process differs slightly between temporal and spatial tasks but follows the same quality control structure.

Training: All annotators receive "detailed written guidelines" and complete a trial annotation of 10 samples, which are meticulously reviewed by domain experts to ensure unified understanding of criteria. This step is crucial for inter-annotator consistency—without it, different annotators might apply different standards for segment boundaries or task compliance.

Temporal task annotation: For each accepted candidate:

1. Verify task compliance (does the audio genuinely exhibit strong sequential uniqueness?).
2. Delineate segment boundaries—identify three natural break points that divide the audio into semantically meaningful units.
3. Write textual descriptions for each sub-clip and for the global audio (captions that capture the acoustic content).
4. Classify the scene into the appropriate subcategory (e.g., object spatial motion, tool operation).
5. Score audio quality.

Spatial task annotation: For each accepted candidate:

1. Select appropriate segments from the recording that exhibit clear spatial characteristics.
2. Classify the task into the appropriate subcategory (single-source static, multi-source relation, dynamic trajectory).
3. Generate a question, the correct answer, and distractor options for the multiple-choice format. The distractors must be plausible but incorrect—random obviously-wrong options would make the task too easy.

Two-round review process:

Round 1: Inter-annotator cross-validation. An initial annotation by Annotator A is fully reviewed by Annotator B, who flags inconsistencies with detailed comments and marks the sample as "failed" if issues are found. Annotators A and B then discuss all flagged issues to reach consensus. The paper identifies the primary sources of ambiguity that emerge during these discussions:

• For temporal tasks: reasonableness of segment boundaries, existence of multiple logically plausible orderings, discrepancies in audio quality scores, adherence to formatting guidelines for captions.
• For spatial tasks: unambiguous alignment of spatial percept with annotated answer, clear necessity of audio spatial cues for question resolution, potential ambiguity in mapping event names to specific sounds, appropriate difficulty and plausibility of distractors.

If consensus cannot be reached, the sample is escalated to an expert panel for final decision. If the experts cannot agree, the sample is discarded—a strict quality gate that prioritizes annotation reliability over sample quantity.

Round 2: Expert spot-check. After passing cross-validation, a random 10% of samples undergo a final quality check by domain experts. Any discovered issues are sent back for revision, creating a feedback loop that improves overall annotation quality.

Stage 4: Final Validation via Human Performance Evaluation

This is the most stringent quality gate in the pipeline. Domain experts—distinct from the annotators—act as examinees and solve the tasks independently. Only items that are correctly solved by at least two-thirds of the experts are retained in the benchmark.

The rationale is explicit:

"Our rigorous protocol ensures that all problems in our benchmark are well-posed and reliably solvable by human experts."

This stage serves multiple methodological functions:

1. Validity check: If domain experts cannot solve a task, it may be ill-posed, ambiguous, or simply too difficult for any current system, making it uninformative for model evaluation.
2. Human baseline establishment: The expert accuracy provides an upper-bound performance estimate—if humans achieve near-perfect accuracy on the retained items, then model performance substantially below that ceiling indicates genuine capability deficits.
3. Distractor quality verification: For multiple-choice questions, human experts consistently selecting the correct answer implies that the distractors, while plausible, are genuinely incorrect—the question has a well-defined right answer.

The paper separately establishes a non-expert human baseline for comparison: 10 university students (explicitly excluding anyone involved in annotation or with domain-specific expertise) evaluate a randomly sampled ~10% subset of each task. The non-expert baseline (reported in Table 2) is notably lower than the expert baseline used for validation, reflecting that the benchmark requires some domain knowledge or trained listening skills.

---

Evaluation Protocol: Robustness Through Perturbation

The evaluation protocol (Section 4 and Appendix C) is designed to address a known problem in LLM benchmarking: sensitivity to superficial input variations that should not affect performance if the model genuinely understands the task.

CircularEval for Foundational and Spatial Tasks

For perception and spatial tasks, the paper adopts the CircularEval method from MM-Bench (Liu et al., 2024b). The core idea: present each question $N$ times, where $N$ is the number of multiple-choice options, with the option order cyclically rotated in each run. For a 4-option question, run 1 presents options as A-B-C-D, run 2 as B-C-D-A, run 3 as C-D-A-B, run 4 as D-A-B-C.

The rationale: LLMs are known to exhibit position bias—they disproportionately select certain option positions (especially the first or last) regardless of content. By cycling option order, CircularEval averages out position biases: a model that always selects the first option will be correct in exactly one of $N$ runs (the run where the correct answer happens to be first), yielding 1/$N$ accuracy, which is the random baseline. Thus, above-chance CircularEval accuracy cannot be achieved by position bias alone.

The rotation is cyclic, meaning each option appears in each position exactly once across all runs. This is a balanced design that eliminates confounds between option identity and position. For models with deterministic outputs, the exact option sequence matters; for stochastic models, the distribution of outputs across runs characterizes response stability.

Three-Run Evaluation for Temporal Tasks

For temporal tasks, a different robustness strategy is used: each question is presented three times with different temporal segment orders (different shufflings of the three clips). The rationale is that a model genuinely solving the reordering task should produce the same correct ordering regardless of how the clips are initially presented—the task is to recover the chronological order, which is invariant to the input order.

If a model's performance varies substantially across runs, it indicates that the model is sensitive to the arbitrary input order—either because it has position biases (e.g., preferring "clip 1 → clip 2 → clip 3" regardless of content) or because it cannot maintain consistent reasoning across different initial presentations.

Dual Metrics: Average Accuracy and All-Correct Rate

The protocol produces two complementary metrics:

Average Accuracy (AA): The mean classification accuracy across all evaluation runs. AA measures the model's expected accuracy on a single presentation of the question. It is the standard metric for most benchmarks and enables direct comparison with random baselines and human performance.

All-Correct Rate (ACR): The proportion of questions answered correctly in every single run. For CircularEval, this means correct across all $N$ cyclic rotations; for temporal tasks, correct across all three shuffles. ACR is a stricter metric that measures output reliability—a model might achieve moderate AA by being correct on some runs and wrong on others, but high ACR requires consistently correct performance.

The paper emphasizes the distinction:

"AA measures expected accuracy on a single presentation, while ACR measures the proportion of questions the model answers correctly regardless of superficial input variations, serving as a stronger indicator of model reliability."

The gap between AA and ACR is diagnostically informative. A large AA-ACR gap (as observed for most models on STAR-Bench, with even Gemini 2.5 Pro showing an average drop of 25.01 percentage points) indicates that the model is sensitive to input perturbations—it can sometimes produce the correct answer but cannot do so reliably. A small AA-ACR gap indicates robust, perturbation-invariant performance.

Human Baseline Establishment

Human performance is evaluated on "a randomly sampled subset of approximately 10% of the data from each task." Ten university students serve as evaluators, with explicit exclusion criteria: anyone involved in data annotation or with domain-specific expertise is excluded. This ensures the human baseline represents general, non-expert performance rather than specialized listening skills.

The human baseline serves as a calibration point: if human non-experts can solve the tasks at high accuracy, then the benchmark is measuring a capability that is within human reach, and model performance below human level represents a genuine capability gap rather than an impossible task. The expert baseline (from Stage 4 of curation) serves a different purpose: it validates that the tasks are well-posed, not that they are easy.

Handling Multi-Audio Input Limitations

Most evaluated models support only single-audio input. For models that do not support multiple separate audio inputs (the paper notes that only Audio Flamingo 3 and its Think variant have this limitation among evaluated models), the paper concatenates the audios with a 2-second silence between them and specifies this in the prompt. For models that do support multiple audio inputs, the audios are fed sequentially with textual indices linking each audio to its label (e.g., "clip 1").

The 2-second silence gap is a pragmatic but potentially problematic design choice. In natural audio, a 2-second gap would be highly salient and might serve as an artificial segmentation cue that simplifies the reordering task. However, for models that cannot accept multiple separate inputs, concatenation is the only option, and the silence gap at least provides a clear boundary that the prompt can reference.

---

Summary of Design Choices and Their Justifications

• Two-level hierarchical structure (Foundation + Holistic): Enables diagnostic decomposition—perceptual failures can be distinguished from knowledge/reasoning failures by comparing foundational perception performance with holistic reasoning performance on the same attributes.

• Procedural synthesis for foundational tasks: Provides ground-truth control over every acoustic parameter, enabling precise difficulty calibration (specific $\Delta$ values) and eliminating annotation ambiguity that would plague real-world audio for perceptual threshold measurement.

• Physics simulation (Pyroomacoustics) for spatial foundational tasks: Ensures accurate binaural cues with known ground-truth source positions, avoiding the circularity of using recorded audio where true source location is approximate or unknown.

• Six attributes × two regimes: Provides comprehensive coverage of the perceptual dimensions relevant to 4D intelligence while keeping the evaluation tractable. The absolute range regime establishes sensory limits; the relative sensitivity regime quantifies discrimination precision.

• Segment reordering for temporal reasoning: A single, unified task format that probes deep understanding of temporal logic without requiring generative free-text answers (which are difficult to evaluate automatically). The three-segment design balances statistical power against combinatorial complexity.

• Five subcategories for temporal reasoning: Systematic enumeration of the types of logical principles governing event sequences (physical evolution, functional procedures, social scripts, causal triggers), ensuring that high performance requires generalizable reasoning rather than narrow heuristics.

• Dual native/channel-wise evaluation for spatial tasks: Acknowledges the reality that most current models cannot process true stereo while still enabling assessment of whether binaural information, when preserved, enables spatial reasoning. Serves as a built-in diagnostic for the channel-averaging problem.

• Four-stage curation pipeline with human performance validation: The most rigorous quality assurance approach among audio benchmarks. Each stage adds a quality gate: automated filtering removes obvious unsuitables, human cross-validation resolves annotation ambiguities, expert spot-checking catches systematic errors, and human performance validation ensures tasks are well-posed and solvable.

• Two-thirds expert agreement for retention: Sets a high but achievable bar—excludes ambiguous or overly difficult items while retaining challenging but solvable tasks. Guarantees that every benchmark item has a verified correct answer that domain experts can reliably identify.

• CircularEval and multi-run temporal evaluation: Addresses the well-documented position bias in LLMs by averaging over cyclic permutations, producing accuracy metrics that cannot be inflated by always selecting the first option. The All-Correct Rate metric additionally measures response stability.

• "Unable to determine" option universally available: Prevents forced-choice artifacts and enables measurement of metacognitive calibration—models that appropriately express uncertainty when they cannot perceive the relevant cue are rewarded, while models that guess are penalized.

4. Key Insights and Innovations

Innovation 1: The Caption-Only Audit as a Diagnostic for Benchmark Validity

The most intellectually distinctive contribution of this paper is not a new task or metric, but a meta-evaluation methodology—a way to test whether an audio benchmark actually measures auditory intelligence or merely language comprehension of audio descriptions. The caption-only experiment in Figure 1 (left) is conceptually simple but profoundly consequential: take a powerful multimodal model, have it generate detailed textual captions of benchmark audio, then evaluate the same model on the benchmark questions using only those captions. The magnitude of the performance drop tells you how much the benchmark depends on linguistically hard-to-describe acoustic cues.

This is a fundamental diagnostic move, not an incremental refinement. Prior to this work, the audio benchmarking community evaluated benchmarks by expanding task coverage, increasing question counts, or adding reasoning categories—all within the assumption that if a model answers correctly, it must be using audio. The caption-only audit falsifies that assumption for existing benchmarks: a ~6–9% drop on MMAU and MMAR means these benchmarks are, at least largely, testing text-recoverable semantics. The ~32–35% drop on STAR-Bench's temporal and spatial tasks—roughly 4× larger than on prior benchmarks—validates that the benchmark genuinely measures something different.

The significance goes beyond validating STAR-Bench itself. The caption-only audit introduces a principled falsification criterion that any audio benchmark can and should be subjected to. If a benchmark's questions can be answered from text captions at near-audio accuracy, the benchmark is not measuring audio intelligence—it is measuring language intelligence applied to audio-derived text. This reframes the evaluation problem from "how well does this benchmark cover the task space?" to "does this benchmark measure the modality it claims to measure?"—a more fundamental question that the field had largely overlooked.

The paper contextualizes this through the tripartite capability decomposition in Figure 1 (right)—fine-grained perception, world knowledge, and multi-step reasoning—arguing that the caption-only audit specifically isolates perception. When a model with perfect caption access performs substantially worse than one with raw audio, the deficit must be attributable to perceptual information that the caption failed to encode. This provides a clean experimental separation between "knowing what sounds mean" and "hearing what sounds are."

Innovation 2: Hierarchical Diagnostic Benchmarking—Separating Perception from Reasoning

The paper's second major intellectual contribution is the two-level diagnostic architecture that structurally separates perceptual capability measurement from reasoning evaluation. This is not merely "having two types of tasks"—it is a principled design where the foundational level serves as a capability prerequisite check for interpreting holistic-level failures, enabling causal attribution of errors rather than opaque aggregate scoring.

Before STAR-Bench, audio benchmarks produced single-dimensional accuracy scores per task category. If a model achieved 30% on "temporal reasoning," there was no way to determine whether the remaining 70% of errors were due to perceptual limitations (the model literally cannot hear the pitch differences needed to order pouring water segments), knowledge gaps (the model hears the differences but doesn't know that higher pitch implies a fuller container), or reasoning failures (the model perceives and knows but cannot chain observations into the correct sequential inference). These are fundamentally different deficits requiring fundamentally different remedies—better audio encoders, more diverse training data, or improved reasoning architectures, respectively—but prior benchmarks couldn't tell them apart.

STAR-Bench's hierarchy enables this discrimination. The foundational perception tasks produce quantitative sensitivity curves for six acoustic attributes under controlled, parameterized conditions (Figure 8, second row). If a model shows near-chance pitch discrimination at small $\Delta$ values, its failure on temporal tasks that depend on pitch tracking (pouring sounds, Doppler shifts) can be attributed, at least partially, to a perceptual bottleneck. Conversely, Gemini 2.5 Pro's strong foundational perception performance combined with its 84% perception error rate on temporal tasks (Figure 6, left) reveals a more subtle problem: its perception is good but not fine-grained enough for the specific contrasts required by complex temporal reasoning—it can discriminate pitch differences in isolated tones but struggles to track continuous pitch evolution in ecologically complex sounds.

This diagnostic architecture is a conceptual advance beyond anything in prior audio benchmarking. It transforms the benchmark from a ranking tool ("model X is 5 points better than model Y") into a profiling tool ("model X fails temporal reasoning primarily because of coarse perception, while model Y fails because of knowledge gaps"). The paper's error analysis (Figure 6) operationalizes this by manually classifying 200 failed predictions into perception errors, knowledge gaps, reasoning errors, and other categories—but the hierarchical design is what makes such classification possible and principled, since the foundational tasks provide independent evidence for or against a perceptual-deficit attribution.

Innovation 3: Identifying Channel-Averaging as a Structural Bottleneck, Not a Training Deficiency

The paper's analysis of the channel-averaging problem (Section 3.2.2, Figure 3) makes a contribution that is simultaneously obvious in retrospect and largely ignored in practice: most current audio-language models are architecturally incapable of spatial hearing, regardless of how well they are trained, because their preprocessing pipeline destroys the binaural information that spatial perception depends on before the model ever sees it.

This is not a training problem that more data could fix—it is a structural bottleneck at the hardware/software interface. Interaural time differences (ITD) and interaural level differences (ILD) are the two primary cues for horizontal-plane sound localization in human hearing. ITD arises from the finite speed of sound (~343 m/s): a source to the left reaches the left ear before the right ear, with the time difference being proportional to the azimuth angle and head size. ILD arises from head shadowing: the head blocks high-frequency sound, making the far ear signal quieter. Both cues require access to the raw per-channel waveforms. Averaging the left and right channels—computing $(x_L(t) + x_R(t))/2$—is mathematically equivalent to collapsing the ITD and ILD to zero. The spatial information is not merely degraded; it is eliminated.

The paper's pseudo-stereo experiment (Figure 3) provides an elegant proof. By assigning a sound to the left channel and its additive inverse to the right channel, the channel-averaged mono signal is $(x(t) + (-x(t)))/2 = 0$—complete silence. All tested models output random guesses because they receive no audio at all. The fact that this simple experiment produces such stark results across models from OpenAI, Google, and multiple open-source projects reveals that the channel-averaging problem is pervasive and severe, yet has gone largely unremarked in the audio-language model literature.

The conceptual significance is that it reframes the spatial reasoning challenge from "can models learn spatial audio?" to "do models have the architectural prerequisites to access spatial information?" The answer, for almost all current models, is no under their default configurations. This means that reported performance on spatial tasks in prior benchmarks—where models might have achieved above-chance accuracy through monaural loudness cues or dataset-specific heuristics—may have created a false impression of spatial capability. The dual native/channel-wise evaluation strategy in STAR-Bench not only acknowledges this problem but transforms it into a diagnostic: comparing native and channel-wise performance reveals whether spatial failures are due to information destruction (large improvement with channel-wise input) or genuine reasoning inability (no improvement).

Innovation 4: Empirical Characterization of the Closed-Source / Open-Source Capability Hierarchy

The paper's comprehensive evaluation of 19 models reveals a qualitatively different error profile between closed-source and open-source models that has implications beyond benchmark scores. This is not merely "closed-source models are better"—that would be unsurprising. The distinctive finding is where each group's bottleneck lies.

The error analysis in Figure 6 shows that Gemini 2.5 Pro's temporal failures are overwhelmingly perceptual (84% perception errors, only 8% knowledge gaps and 2% reasoning errors). This model has the knowledge and reasoning capability to solve temporal tasks—it demonstrates sophisticated physical understanding in its successful responses (as in the water-pouring example in Figure 7, where it produces a remarkably detailed acoustic description). Its bottleneck is that its audio perception, while strong relative to other models, is not sufficiently fine-grained to reliably extract the subtle acoustic cues needed for discrimination.

Open-source models like Qwen-2.5-Omni show the opposite profile: 54% knowledge gaps, only 23% perception errors for temporal tasks. These models are not primarily failing because they cannot hear—they are failing because even when they can perceive the sounds, they lack the physical world knowledge to interpret what the perceptual differences mean. A model might detect that a sound's pitch is changing, but without understanding that rising pitch during pouring indicates a filling container, it cannot use that perceptual information to determine temporal order.

This asymmetry has practical implications for model development. For closed-source models, the path forward is improving audio encoder resolution—finer-grained perceptual sensitivity, better temporal tracking of continuous acoustic changes, and native stereo processing. The reasoning and knowledge foundations are already strong. For open-source models, the more urgent priority—as the paper argues in the discussion—is improving dense audio captioning and world knowledge integration. The ablation study in Figure 9 provides direct evidence: when provided with global captions or uncut reference audio, open-source models show minimal improvement while Gemini 2.5 Pro's performance scales effectively, suggesting open-source models cannot leverage additional information because they lack the knowledge framework to interpret it.

This finding also contextualizes the "think" mode results. Audio Flamingo 3 Think and Xiaomi-MiMo-Audio Think perform worse than their non-thinking counterparts (Table 2). The paper interprets this as evidence that "without sufficiently solid perceptual and knowledge foundations, reasoning can be ineffective or even detrimental." Extended reasoning chains amplify errors when the premises—perceptual observations and world knowledge—are unreliable. This is a negative result with positive diagnostic value: it implies that reasoning enhancements (chain-of-thought, test-time compute) should not be applied uniformly but should be conditioned on the quality of the perceptual and knowledge substrate.

Innovation 5: The ACR Reliability Metric as a Window into Model Instability

The paper introduces All-Correct Rate (ACR) alongside Average Accuracy (AA) and, more importantly, uses the gap between them diagnostically. This goes beyond the standard practice of reporting multiple metrics by treating the AA-ACR gap as a signal about model behavior rather than just a robustness check.

The finding is stark: even Gemini 2.5 Pro shows an average AA-to-ACR drop of 25.01 percentage points (Appendix E.1). For most open-source models, ACR is near zero—meaning that under minor input perturbations (cyclic option rotation or segment reordering), these models cannot consistently produce the correct answer on any question. The paper notes that this instability often manifests as "a tendency to concentrate predictions on a specific option, suggesting a reliance on superficial biases rather than genuine understanding."

The conceptual contribution is that ACR measures response stability under task-preserving transformations—input variations that should not change the correct answer. A model with genuine understanding should be invariant to these transformations. High AA with low ACR indicates that the model is sometimes right for the wrong reasons (e.g., position bias placing the correct answer in a preferred slot on some runs) or that its reasoning is fragile and disrupted by superficial input changes.

This insight connects to broader concerns about LLM evaluation validity. If a model's answer depends on the arbitrary order of options or segments—features irrelevant to the task—then single-run accuracy scores overestimate true capability. The ACR metric partially corrects for this by requiring consistent correctness, and the large AA-ACR gaps observed on STAR-Bench suggest that much of the reported accuracy on audio benchmarks (including this one) may be inflated by lucky alignments between model biases and question formatting. This is not unique to STAR-Bench, but the paper's explicit measurement and discussion of the gap makes the problem visible in a way that single-metric benchmarks obscure.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use the STAR-Bench benchmark introduced in this paper, comprising 2,353 total questions: 951 foundational perception questions (spanning six acoustic attributes under absolute range and relative discrimination sensitivity), 900 temporal reasoning questions (segment reordering across continuous processes and discrete event sequences), and 502 spatial reasoning questions (static localization, multi-source relations, and dynamic trajectory tracking). The real-world audio for holistic tasks is sourced from Clotho (Drossos et al., 2019), FSD50K (Fonseca et al., 2022), STARSS23 (Shimada et al., 2023), and internet-sourced recordings; foundational perception tasks use procedurally synthesized or Pyroomacoustics-simulated audio. The paper reports statistics in Figure 4; the average audio length for reasoning tasks is 14.03 seconds.

• Base model(s). The evaluation covers 19 models: 16 open-source and 3 closed-source. Open-source models span three categories: Large Audio Language Models (SALMONN-13B, Qwen2-Audio-Instruct-8.4B, Audio Flamingo 3-8.4B with its Think variant, DeSTA2.5-Audio-8.8B, Kimi-Audio-7B, Step-Audio-2-mini-7B, MiDashengLM-7B, Xiaomi-MiMo-Audio-7B with its Think variant), a specialized spatial audio model (BAT-7B), and Omni Language Models (Qwen-2.5-Omni-7B, Phi4-MM-5.5B, Gemma-3n-E4B-it-7.5B, Ming-Lite-Omni-1.5-18.9B, MiniCPM-O-v2.6-8B). Closed-source models include Gemini 2.5 Pro, Gemini 2.5 Flash (both updated June 2025), and GPT-4o-audio-preview (version 2025-06-03). The paper states that the open-source models were chosen to span "representative of the capabilities of many contemporary" audio-language and omni-language models, with sizes ranging from 5.5B to 18.9B parameters.

• Metrics. All questions are multiple-choice and evaluated using classification accuracy with string matching of option labels or full text. Two primary metrics are reported: Average Accuracy (AA), the mean accuracy across all evaluation runs, and All-Correct Rate (ACR), the proportion of questions answered correctly in every single run. For foundational perception and spatial tasks, CircularEval (from MM-Bench; Liu et al., 2024b) is applied, presenting each question N times (where N equals the number of options) with cyclically rotated option orders to mitigate position bias. For temporal tasks, each question is evaluated three times with different temporal segment orders. The paper primarily reports AA in the main text tables due to space constraints, with complete AA/ACR breakdowns in Appendix D. Due to API costs, GPT-4o Audio was evaluated only once per question (no CircularEval).

• Baselines. The paper reports random guess baselines specific to each task type (25.00% for 4-option perceptual tasks, 14.29% for 6-option temporal reordering, 33.33% for 3-option spatial tasks) and a human performance baseline. The human baseline was established by having 10 university students (explicitly excluding annotators and domain experts) evaluate a randomly sampled ~10% subset of each task. Human performance on the full benchmark is reported as: 75.60% AA on foundational perception (79.42% range, 74.55% sensitivity), 88.00% AA on temporal reasoning (90.12% continuous, 85.51% discrete), and 73.72% AA on spatial reasoning (70.00% localization, 80.00% relation, 77.00% trajectory). The paper additionally notes a separate domain-expert baseline used during curation validation, but the human baseline in Table 2 is the non-expert version.

• Generation budget / compute accounting. STAR-Bench is a static benchmark evaluation, not a test-time compute scaling study, so there is no "generation budget" in the sense of varying inference compute per question. All models are evaluated under their standard inference configuration with the same set of fixed benchmark questions. The paper notes that for models without multi-audio support (specifically Audio Flamingo 3 and its Think variant), multiple clips are concatenated with a 2-second silence gap; other models receive multiple audios sequentially with textual indices. Channel-wise spatial evaluation presents left and right channels as separate audio streams with explicit textual labels.

• Cross-validation / statistical protocol. No cross-validation over benchmark splits is performed—the full benchmark is used for evaluation. Robustness is addressed through the CircularEval and multi-run temporal evaluation designs rather than through data splits. For the manual error analysis (Section 4.2), 200 failed predictions were sampled equally from temporal and spatial tasks of three representative models (Gemini 2.5 Pro, GPT-4o-audio, Qwen-2.5-Omni), though the paper does not specify further sampling stratification or inter-annotator agreement for the error classification. The human performance evaluation uses a random 10% subset per task, but no confidence intervals or standard deviations are reported for human or model scores.

---

Main Quantitative Results

Overall Benchmark Difficulty: Humans Far Outperform All Models

The headline finding from Table 2 is that STAR-Bench presents a substantial challenge to all evaluated models, with even the strongest model (Gemini 2.5 Pro) achieving only 49.59% macro-averaged accuracy across the three task categories, compared to 79.11% for humans. This ~30 percentage point gap is consistent across task types: 46.64% vs. 75.60% on foundational perception, 58.52% vs. 88.00% on temporal reasoning, and 43.62% vs. 73.72% on spatial reasoning. The gap is largest for spatial reasoning (30.1 percentage points), intermediate for temporal reasoning (29.5 points), and smallest but still substantial for foundational perception (29.0 points).

Most open-source models perform close to random guessing in aggregate. The macro-averaged accuracy (MA) across all three task categories ranges from 4.29% (BAT) to 31.28% (MiDashengLM), with the majority of open-source models clustering in the 23–28% range. The random baseline across all tasks is 24.32% (computed per the paper as the weighted average of task-specific random baselines), meaning several open-source models (SALMONN at 23.33%, DeSTA2.5-Audio at 23.30%, Gemma-3n-E4B-it at 22.92%) perform at or below random chance in aggregate. This is a damning result for current open-source audio models—their aggregate performance on STAR-Bench is essentially equivalent to guessing.

The All-Correct Rate (ACR) analysis reveals a more severe picture. As noted in Appendix E.1, even Gemini 2.5 Pro exhibits an average AA-to-ACR drop of 25.01 percentage points, and "the majority of open-source models record an ACR near zero." This means that under minor input perturbations (cyclic option rotation, segment reordering), these models cannot consistently produce the correct answer on any question. The paper attributes this partly to "a tendency to concentrate predictions on a specific option, suggesting a reliance on superficial biases rather than genuine understanding."

Foundational Perception: Closed-Source Models Show Superior Range and Sensitivity

Table 4 (Appendix D) provides the full foundational perception breakdown with both AA and ACR for every attribute and regime. The headline numbers from Table 2 are the macro-averaged accuracy: Gemini 2.5 Pro leads at 46.64% (39.90% range, 51.13% sensitivity), followed by Gemini 2.5 Flash at 39.72% (33.46% range, 43.88% sensitivity). The best open-source model is Audio Flamingo 3 at 34.15% (31.79% range, 35.72% sensitivity), followed by MiDashengLM at 33.24% (36.94% range, 30.78% sensitivity) and Xiaomi-MiMo-Audio at 32.93% (34.95% range, 31.59% sensitivity).

A notable pattern emerges in the range-versus-sensitivity breakdown. For absolute perception range, the best open-source model (Audio Flamingo 3 at 31.79%) trails Gemini 2.5 Pro (39.90%) by 8.1 percentage points—a substantial but not catastrophic gap. For relative discrimination sensitivity, the gap nearly doubles to 15.4 points (Audio Flamingo 3 at 35.72% vs. Gemini 2.5 Pro at 51.13%). This asymmetry suggests that discriminating fine differences between similar sounds is substantially harder for current models than detecting whether a sound is present at all, and that closed-source models have a particular advantage in fine-grained discrimination.

The BAT result as a revealing negative baseline. BAT (Zheng et al., 2024), a model specifically designed for spatial audio reasoning, achieves 12.87% overall foundational perception accuracy, with a striking 6.25% on sensitivity tasks. Its absolute range scores are non-zero for spatial attributes (26.04% azimuth, 41.67% elevation, 23.53% distance) but zero for pitch-and-loudness detection, and all sensitivity scores are zero except azimuth (37.50%). This profile suggests BAT's audio encoder is specialized for spatial cues but catastrophically fails on non-spatial perceptual tasks—an extreme case of the perception-reasoning tradeoff that makes it incapable of the integrative reasoning STAR-Bench's holistic tasks require.

The audiogram visualization (Figure 8, first row). The paper provides audiogram-style visualizations comparing Gemini 2.5 Pro, GPT-4o Audio, and Qwen-2.5-Omni across the pitch–loudness space. Gemini 2.5 Pro achieves "much broader coverage than the other two models, where greener regions indicate higher accuracy and the covered area reflects the perceptual range." The paper notes that human listeners with normal hearing are expected to achieve near-full coverage, implying that all models—including the best—fall well short of human perceptual range.

The sensitivity curves (Figure 8, second row). For pitch, loudness, and duration sensitivity, the paper plots ACR as a function of stimulus difference Δ. The key findings:

• For pitch sensitivity (Figure 8e), Gemini 2.5 Pro shows a clear monotonic improvement from near-zero ACR at Δ = 0 and 50 cents to ~80% ACR at Δ = 1200 cents. GPT-4o Audio and Qwen-2.5-Omni show flatter curves with lower asymptotes, never exceeding ~30–40% ACR even at the largest Δ. The human baseline (dashed line) is near-perfect across all non-zero Δ levels.

• For loudness sensitivity (Figure 8f), the most striking finding is that human performance is near-ceiling even at Δ = 4 dB, while all models show dramatically lower sensitivity. Gemini 2.5 Pro's ACR rises from near-zero at Δ = 0–8 dB to only ~50% at Δ = 48 dB. GPT-4o Audio and Qwen-2.5-Omni remain below ~30% ACR at all Δ levels. The paper states this reveals "a stark performance gap between all models and the human baseline, particularly in the perception of fine-grained loudness differences."

• For duration sensitivity (Figure 8g), Gemini 2.5 Pro shows a distinctive advantage, reaching ~90% ACR at Δ = 200% compared to ~50% for the other models. The paper interprets this as "showcasing temporal grounding capabilities superior to those of other models by accurately assessing audio segment lengths."

Temporal Reasoning: Gemini 2.5 Pro Dominates; Open-Source Models Are Near-Random

Table 5 (Appendix D) provides the full temporal reasoning breakdown. The headline numbers from Table 2: Gemini 2.5 Pro achieves 58.52% overall accuracy (54.88% continuous processes, 62.74% discrete event sequences), far ahead of the second-best model, Gemini 2.5 Flash at 30.70% (27.55% continuous, 34.38% discrete). The drop from first to second place is 27.82 percentage points—an enormous gap that underscores the qualitative difference between Gemini 2.5 Pro and all other models on temporal reasoning.

Among open-source models, the best performer is Xiaomi-MiMo-Audio at 18.63% (18.22% continuous, 19.15% discrete), followed by Kimi-Audio at 18.52% (18.71% continuous, 16.83% discrete). These are only ~4–5 percentage points above the random baseline of 14.29%, meaning even the best open-source models achieve only marginally above-chance performance on temporal reasoning. Audio Flamingo 3 achieves 8.67%—below the random baseline—suggesting systematic errors that push performance below guessing level.

The continuous vs. discrete breakdown reveals an interesting inversion for Gemini 2.5 Pro. While most models perform roughly equally (or slightly worse) on continuous processes compared to discrete event sequences, Gemini 2.5 Pro shows the opposite pattern: 54.88% on continuous vs. 62.74% on discrete. This 7.86 percentage point advantage for discrete sequences suggests Gemini 2.5 Pro's reasoning strengths lie more in understanding functional procedures and causal scripts than in tracking continuous acoustic evolution—consistent with the error analysis finding that fine-grained perception (needed for continuous tracking) is its primary bottleneck.

The subcategory breakdown (Table 5) provides finer diagnostic resolution:

• Object Spatial Motion (Doppler-based trajectory reconstruction): Gemini 2.5 Pro achieves 63.82%, the highest subcategory score for any model on temporal tasks, suggesting strong integration of pitch and loudness tracking with physical knowledge of motion.
• In-Situ State Evolution (physical process tracking): 43.72%—substantially lower than object spatial motion, indicating that tracking continuous state changes without spatial cues is more challenging even for the strongest model.
• Tool and Appliance Operation: 69.77%—the single highest subcategory score, indicating strong functional knowledge.
• Daily Scene Scripts: 57.22%—strong but substantially below tool operation, suggesting commonsense script knowledge is more variable than functional knowledge.
• Event-Triggered Consequences: 48.96%—the lowest discrete subcategory, suggesting causal reasoning remains challenging even for the strongest model.

Among open-source models, Event-Triggered Consequences shows the highest scores for several models (e.g., Gemma-3n-E4B-it at 20.83%, MiDashengLM at 21.88%, Xiaomi-MiMo-Audio at 26.04%), potentially reflecting dataset biases where causal sequences are more stereotyped and thus easier to guess.

Spatial Reasoning: All Models Struggle; Channel-Wise Input Partially Helps

Table 6 (Appendix D) provides the spatial reasoning breakdown under both native and channel-wise input. The headline numbers from Table 2: the best model on spatial reasoning is MiDashengLM at 44.29% (native input), followed by Gemini 2.5 Pro at 43.62% and GPT-4o Audio at 41.70%. These are only ~10–11 percentage points above the random baseline of 33.33%—a much smaller absolute gain than on temporal or foundational tasks. This is consistent with the paper's argument that the channel-averaging problem fundamentally limits what any model can achieve on spatial tasks.

Native vs. channel-wise input comparison. The channel-wise evaluation produces mixed and often counterintuitive results:

• Models that improve with channel-wise input: GPT-4o Audio improves from 41.70% to 45.42% (+3.72 points), Gemini 2.5 Flash improves dramatically from 28.35% to 41.23% (+12.88 points), and Audio Flamingo 3 improves from 38.91% to 44.35% (+5.44 points). These gains suggest that when binaural information is explicitly preserved, these models can partially exploit it—though the improvements are modest relative to the human baseline.

• Models that do not improve or degrade with channel-wise input: Gemini 2.5 Pro decreases from 43.62% to 40.77% (−2.85 points), Qwen2-Audio-Instruct decreases from 20.78% to 8.76% (−12.02 points), and Kimi-Audio decreases from 33.60% to 22.84% (−10.76 points). These degradations are puzzling—preserving more information should not reduce performance unless the model cannot process multi-audio inputs and the channel-wise format introduces confusion.

• The Qwen2-Audio-Instruct collapse (20.78% → 8.76%) is the most extreme degradation. This suggests the model is actively harmed by receiving separate left/right channels—perhaps because its training did not include multi-audio conversations, and the unusual input format triggers maladaptive behavior.

The paper's interpretation (Appendix E.2) is that "most existing models are not trained to handle multi-audio inputs. As a result, they consistently struggle to align channel representations and fail to make reliable use of interaural differences." In other words, the channel-wise evaluation reveals a training-data limitation: these models have not learned that two audio streams can represent binaural perspectives, so preserving the channels does not help if the model lacks the learned capability to extract ITD and ILD from them.

Subcategory breakdown (Table 6, native input):

• Single-Source Static Localization: MiDashengLM leads at 43.11%, followed by Gemini 2.5 Pro at 40.87% and Qwen-2.5-Omni at 39.46%. Even the best models are only ~10 points above random (33.33%).
• Multi-Source Spatial Relation: Gemini 2.5 Pro leads at 48.97%, followed by MiDashengLM at 45.43% and Xiaomi-MiMo-Audio at 41.30%. This is the highest-performing subcategory, possibly because relational judgments can sometimes be made using monaural loudness comparisons without requiring full binaural processing.
• Dynamic Trajectory Tracking: MiDashengLM leads at 46.23%, followed by Xiaomi-MiMo-Audio at 45.28% and Gemini 2.5 Pro at 45.28%. The relatively strong open-source performance on this subcategory (several open-source models exceed Gemini 2.5 Pro) is unexpected and the paper does not provide a clear explanation—it may reflect that trajectory tracking in the benchmark relies on simpler cues than intended, or that open-source models have encountered similar trajectory patterns in training data.

BAT's zero performance (0.00% across all spatial tasks) is a critical negative result. BAT was specifically designed for spatial audio reasoning (Zheng et al., 2024) and trained on spatial audio tasks. Its complete failure on STAR-Bench spatial tasks—under both native and channel-wise input—indicates either a severe distribution shift between its training data and STAR-Bench, a fundamental architectural limitation in generalizing spatial reasoning beyond its training distribution, or possibly an evaluation configuration error. The paper does not diagnose this further, but the result raises questions about the transferability of specialized spatial audio training to diverse real-world spatial tasks.

Error Analysis: The Capability Hierarchy Revealed

The manual error analysis of 200 failed predictions (Figure 6) is one of the paper's most informative results. For temporal tasks:

• Gemini 2.5 Pro: 84% perception errors, 8% knowledge gaps, 2% reasoning errors, 2% hallucination, 2% other (misalignment). The dominance of perception errors (84%) is the key finding—Gemini 2.5 Pro knows what to do and how to reason, but cannot reliably extract the fine-grained acoustic cues needed to execute.

• GPT-4o Audio: 70% perception errors, 4% reasoning errors, 4% hallucination, 9% reasoning-answer misalignment, 13% generation collapse. The generation collapse (13%)—where the model produces incomplete or incoherent outputs—is a distinctive failure mode not seen in the other two models.

• Qwen-2.5-Omni: 23% perception errors, 54% knowledge gaps, 17% reasoning errors, 8% reasoning-answer misalignment. The dominant knowledge gap (54%) is the distinguishing feature—this model lacks the physical world knowledge needed to interpret perceptual observations.

For spatial tasks, the error profiles are less clearly differentiated, with all three models showing substantial perception and knowledge errors. The paper also notes "a prevalent type of error involves vision-centric hallucinations (e.g., '...based on the car's trajectory in the video...')," suggesting that models trained on visual spatial reasoning may misapply visual reasoning patterns to auditory inputs—a form of modality confusion.

Temporal Reasoning Ablation: Open-Source Models Cannot Leverage Additional Information

The ablation study in Figure 9 progressively simplifies the temporal reasoning task by providing additional information:

• Baseline: Standard audio segment reordering (three shuffled clips).
• + Global Caption: A single sentence describing the overall scene is provided as contextual guidance.
• + Uncut Audio: The complete, unsegmented audio track is provided as a reference, reducing the task to comparing and grounding each segment within the full audio.

The results reveal a dramatic divergence between Gemini 2.5 Pro and open-source models:

• Gemini 2.5 Pro: Scales effectively from 58.52% (baseline) → 76.33% (+ global caption) → 99.00% (+ uncut audio). The near-perfect 99% in the uncut audio condition confirms that Gemini 2.5 Pro can perform segment grounding when given a reference—its baseline failures are primarily due to the difficulty of the pure reordering task, not a fundamental inability to compare across audio segments.

• Gemini 2.5 Flash: Improves from 30.70% → 50.89% → 57.78%, showing scaling but at a much lower asymptote.

• Qwen-2.5-Omni: Shows minimal improvement: 16.96% → 16.44% → 17.04%. Essentially flat across conditions.

• Xiaomi-MiMo-Audio: 18.63% → 18.93% → 21.85%. Slight improvement but remains very low.

• Xiaomi-MiMo-Audio Think: 18.00% → 25.33% → 24.04%. Modest improvement but still far below Gemini 2.5 Flash.

The paper's interpretation is direct: "This finding starkly exposes a core weakness in current open-source models: a fundamental inability to effectively compare, ground, and integrate information from multiple audio inputs." The flat curves for Qwen-2.5-Omni and Xiaomi-MiMo-Audio, even in the uncut audio condition where the task reduces to straightforward segment-to-reference matching, indicate that these models lack even the basic capability to align audio segments with a reference track—a prerequisite for any form of comparative audio reasoning.

---

Ablation Studies and Robustness Checks

Caption-only benchmark auditing (Figure 1, left): The paper demonstrates that Gemini 2.5 Pro's accuracy on MMAU drops by only 5.9% (from 88.00% with audio to 82.20% with captions) and on MMAR by 9.0% (from 77.80% to 73.70%), while on STAR-Bench's temporal and spatial tasks, the drops are 31.5% and 35.2% respectively (from 58.50% to 27.00% on temporal, from 43.60% to 8.40% on spatial). This 4× larger degradation validates STAR-Bench's emphasis on linguistically hard-to-describe cues and demonstrates that the benchmark measures something fundamentally different from prior benchmarks.

Channel-averaging diagnostic (Figure 3): The pseudo-stereo experiment tests 20 samples where the left channel contains a sound and the right channel contains its additive inverse. Humans achieve 100% accuracy, but Gemini 2.5 Pro achieves 20%, GPT-4o-audio-preview achieves 0%, and Qwen-2.5-Omni achieves 0%—all at or below random chance (25% for four options). A dedicated spatial reasoning model, BAT, achieves 100%, confirming that its architecture preserves channel information while all other evaluated models destroy it through channel averaging. This ablation isolates a structural preprocessing limitation that makes spatial reasoning impossible regardless of model quality.

Think vs. non-think variants (Table 2): Audio Flamingo 3 Think performs worse than Audio Flamingo 3 on foundational perception (30.66% vs. 34.15% MA), temporal reasoning (13.59% vs. 8.67% OA—note: think is higher here), and spatial reasoning (36.45% vs. 38.91% OA native). Xiaomi-MiMo-Audio Think performs worse than its non-think counterpart on foundational perception (26.92% vs. 32.93% MA) and temporal reasoning (18.00% vs. 18.63% OA), with mixed spatial results. The paper interprets this as evidence that reasoning enhancements can be "ineffective or even detrimental" when the underlying perceptual and knowledge foundations are insufficient.

Human performance validation (Table 2): Human non-experts achieve 75.60% on foundational perception, 88.00% on temporal reasoning, and 73.72% on spatial reasoning. These baselines serve as the calibration point against which model performance is measured. The gap between human and the best model (Gemini 2.5 Pro) ranges from ~29–30 percentage points across tasks, establishing a clear capability deficit. The higher human performance on temporal reasoning (88%) compared to spatial reasoning (73.7%) suggests that the temporal tasks, while still very challenging for models, are more intuitive for humans.

Multi-audio concatenation vs. sequential input (implicit ablation, Section 4): For models without multi-audio support (Audio Flamingo 3 and its Think variant), clips are concatenated with 2-second silence. For models with multi-audio support, clips are fed sequentially with textual indices. While not presented as a formal ablation, the performance differences between these two groups (both perform poorly) suggest that the input format is not the primary bottleneck—the fundamental capability to compare across audio segments is missing in both cases.

Oracle difficulty estimation cost (not evaluated): The foundational perception tasks themselves serve as a form of difficulty estimation—by profiling a model's perceptual range and sensitivity, one can predict which holistic tasks it will struggle with. However, the paper does not evaluate how many foundational task samples are needed to reliably predict holistic task performance, nor does it explore whether a lightweight perceptual screening could substitute for the full foundational evaluation. This is a missed opportunity for establishing the diagnostic efficiency of the hierarchical design.

---

Critical Assessment

Does the Benchmark Genuinely Measure Audio 4D Intelligence?

The paper's central claim is that STAR-Bench measures audio 4D intelligence—deep reasoning over sound source dynamics in time and 3D space, grounded in physical understanding. The evidence partially supports this claim, but there are important gaps between the construct and its operationalization.

What the evidence shows. The caption-only audit (Figure 1) convincingly demonstrates that STAR-Bench's tasks depend on acoustic information that cannot be recovered from text captions—the 31.5% and 35.2% drops are genuine and substantially larger than for prior benchmarks. The hierarchical design is genuinely diagnostic: the foundational perception results (Table 4, Figure 8) provide independent evidence for perceptual limitations that can be cross-referenced with holistic task failures. The error analysis (Figure 6) operationalizes the tripartite decomposition into perception, knowledge, and reasoning errors, and reveals qualitatively different bottleneck profiles across models. The ablation study (Figure 9) demonstrates that open-source models cannot leverage additional information to improve temporal reasoning, confirming a fundamental capability gap rather than a task-format artifact.

What the evidence does not show. Several aspects of the audio 4D intelligence construct are not fully tested:

1. Genuine physical reasoning vs. pattern matching. The benchmark cannot distinguish between a model that applies the Doppler equation to infer motion direction and a model that has learned, from thousands of training examples, that "siren pitch going up then down" correlates with "passed from left to right." The paper's use of diverse real-world audio sourced from multiple datasets partially addresses this by reducing the chance that any single pattern dominates, but without adversarial or counterfactual test cases (e.g., sounds with artificially inverted Doppler shifts), it is impossible to verify whether correct answers reflect physical understanding or surface-level correlations. The foundational tasks are synthetic and thus novel, but the holistic tasks use real-world audio from widely-used datasets (FSD50K, Clotho) that are likely in models' training data—the paper's argument that the task formulation is novel does not fully address the concern that models may have learned spurious correlations between acoustic patterns and temporal orders from caption-level training on these same datasets.

2. The "4D" in 4D intelligence is only partially tested. Spatial reasoning tasks are exclusively about azimuth (left/right), elevation, and distance—there is no integration of spatial and temporal reasoning within a single task except for dynamic trajectory tracking. A true 4D intelligence test would require reasoning about how a sound source's position in 3D space changes over time, with tasks that simultaneously demand temporal ordering and spatial localization (e.g., "the explosion happened behind you, then the debris fell to your left—which order did you hear the sounds in?"). The current benchmark treats temporal and spatial dimensions as parallel tracks rather than integrated ones.

3. "Grounded in an understanding of the physical world" is assumed rather than tested. The paper frames tasks as requiring physical principles (fluid dynamics, thermodynamics, Doppler effect), but never verifies that models solving these tasks actually use those principles. A model could order pouring-water segments correctly by tracking pitch trends without understanding why pitch rises as a container fills. The error analysis classifies failures into perception/knowledge/reasoning, but for successful predictions, there is no mechanism to verify that the model's internal reasoning process invoked physical principles rather than heuristic pattern completion.

Strength of Model Comparison Claims

Claim: "Open-source models exhibit fundamental weaknesses across all three core capabilities" (perception, knowledge, reasoning). This is strongly supported by the data. Open-source models' aggregate accuracy is near random (most in the 23–28% range, with random at 24.32%), their ACR is near zero (no consistent correct answers under input perturbation), and the ablation study (Figure 9) shows flat curves even when provided with additional information. The error analysis provides qualitative confirmation: Qwen-2.5-Omni shows 54% knowledge gaps, 23% perception errors, and 17% reasoning errors—weakness across all three pillars. However, the claim is asserted generically about "open-source models" while the evaluation covers 16 specific open-source models with heterogeneous architectures and training procedures. The results might not generalize to the broader class of open-source audio-language models, especially larger or differently-trained variants not included in the evaluation.

Claim: "Closed-source models are bottlenecked by fine-grained perception." This is supported for Gemini 2.5 Pro, for which the error analysis shows 84% perception errors on temporal tasks with only 8% knowledge gaps and 2% reasoning errors. The foundational perception sensitivity curves (Figure 8, second row) independently confirm that Gemini 2.5 Pro's perceptual discrimination, while best-in-class, degrades substantially as stimulus differences shrink. However, the claim is less clearly supported for GPT-4o Audio, which shows 70% perception errors but also 13% generation collapse—a different bottleneck profile. For Gemini 2.5 Flash, no detailed error analysis is provided, so its bottleneck profile is unknown. The claim therefore holds for the strongest closed-source model but may not characterize the category as a whole.

Claim: "The caption-only experiment demonstrates that STAR-Bench measures linguistically hard-to-describe cues." This is well-supported by the 4× larger accuracy drops on STAR-Bench compared to prior benchmarks. However, the control experiment has limitations. The caption quality depends on Gemini 2.5 Pro's audio captioning capability—if the captions are poor, the drop may reflect captioning inadequacy rather than the intrinsic difficulty of describing the acoustic cues verbally. The paper provides one example of Gemini 2.5 Pro's captioning output (Appendix B.1: "Please provide a detailed description of the audio...") but does not evaluate caption quality or compare different captioning models. A stronger design would use multiple captioning models or human-written captions to bound the effect of captioner quality.

Weaknesses and Missing Experiments

Small evaluation sample for human baseline. Human performance is evaluated on ~10% of each task (roughly 235 total questions spread across three task categories). With 10 human evaluators, this yields ~2,350 human judgments—sufficient for aggregate accuracy estimation but insufficient for subcategory-level or difficulty-level human baselines. Several model-vs-human comparisons at the subcategory level are thus underpowered.

No confidence intervals or statistical testing. All reported accuracies are point estimates without confidence intervals, standard errors, or significance tests. This makes it impossible to determine whether, for example, MiDashengLM's 44.29% spatial accuracy is statistically distinguishable from Gemini 2.5 Pro's 43.62%, or whether the 1–2 percentage point differences between adjacent models in the open-source rankings are meaningful. Given the relatively small test set (502 spatial questions, 900 temporal questions, 237–357 questions per foundational subcategory), sampling variability could account for some of the reported rank ordering.

The BAT model's zero performance is not adequately explained. BAT was specifically designed for spatial audio reasoning and achieves perfect accuracy on the pseudo-stereo test (Figure 3), yet scores 0.00% on all spatial reasoning tasks in STAR-Bench. The paper notes this result but does not investigate whether it reflects a genuine capability gap, a catastrophic distribution shift, a prompt formatting issue, or a configuration error. For a model that serves as the only specialized spatial audio baseline, this diagnostic failure is a significant gap—we learn that BAT fails, but not why.

No evaluation on model scale variation. All evaluated models have different architectures, training procedures, and scales. The paper cannot isolate whether performance differences are due to model size, training data composition, architectural choices, or training objectives. Evaluating a single model family at multiple scales (e.g., 1B, 7B, 13B, 70B variants of the same architecture) would provide scaling-law insights that the current heterogeneous evaluation cannot.

Spatial task design may conflate several challenges. The spatial reasoning tasks require simultaneous sound source segregation (identifying which source is which in multi-source scenes), spatial localization, and relational reasoning. A model could fail because it cannot separate simultaneous sounds (auditory scene analysis) even if its spatial perception is intact. The benchmark does not include monaural control conditions that would isolate spatial perception from source segregation, making spatial failure attribution ambiguous.

The 2-second silence concatenation for multi-audio models may introduce artifacts. For models that cannot accept multiple separate audio inputs, clips are concatenated with a 2-second silence gap. This gap is highly salient and could serve as an artificial segmentation cue that simplifies the reordering task compared to models receiving clips as separate inputs. The paper does not verify that concatenated and sequential input formats produce comparable difficulty.

The foundational perception tasks may not transfer to ecologically complex sounds. The foundational tasks use pure tones for non-spatial attributes and curated broadband sounds for spatial attributes—clean, isolated stimuli under controlled conditions. The holistic tasks use real-world recordings with background noise, reverberation, overlapping sources, and natural variability. A model could perform well on pure-tone pitch discrimination but fail to track pitch in noisy, reverberant, real-world recordings, making the foundational-to-holistic diagnostic link imperfect. The paper does not evaluate whether foundational task performance predicts holistic task performance at the per-model or per-attribute level.

No cross-benchmark correlation analysis. The paper argues that STAR-Bench measures something different from prior benchmarks, but does not report correlations between model performance on STAR-Bench and on MMAU, MMAR, or other audio benchmarks. Such correlations would quantify whether STAR-Bench is measuring a distinct capability dimension or simply a harder version of the same dimension. Models that excel on prior benchmarks but fail on STAR-Bench (and vice versa) would strengthen the claim of complementary measurement.

Conditional Scope of Claims

The paper's claims are best understood as establishing what current models cannot do on a specific, carefully constructed benchmark, rather than as making universal claims about audio reasoning capabilities. Several conditions bound the findings:

• The results are specific to the 19 evaluated models. The claim that "open-source models lack fundamental audio reasoning capabilities" might not hold for larger open-source models (beyond 18.9B parameters), differently-trained models (e.g., those explicitly trained on binaural audio or multi-event reasoning), or models released after the evaluation cutoff.

• The benchmark's difficulty is relative to current models, not absolute. Human non-experts achieve 73–88% accuracy, indicating the tasks are solvable by humans without specialized training. The fact that the best model achieves only 49.59% suggests a genuine capability gap, but does not imply that the tasks are fundamentally unsolvable by AI—future models with improved audio encoders, multi-channel processing, and physical reasoning capabilities could close this gap.

• The channel-averaging problem means spatial task results are a lower bound on potential performance. Since most models destroy binaural information before processing, their spatial reasoning scores reflect both genuine reasoning limitations and an information bottleneck. The channel-wise evaluation partially addresses this but introduces the confound of multi-audio processing capability. The true upper bound for what these models could achieve with native stereo processing is unknown.

• The benchmark measures linguistically hard-to-describe reasoning, not all forms of audio 4D intelligence. The paper's construct is specifically operationalized through tasks where text captions fail to capture the necessary acoustic information. There may be aspects of audio 4D intelligence that are linguistically describable but still require deep physical reasoning—for example, "the engine sputtered for 2 seconds before dying" is linguistically expressible but still requires reasoning about mechanical failure sequences. STAR-Bench's focus on non-linguistic cues means it samples a specific region of the audio intelligence space, not the entire construct.

• The diagnostic link between foundational and holistic tasks is hypothesized but not empirically validated. The paper asserts that foundational perception performance explains holistic task failures, and the error analysis manually classifies failures into perception/knowledge/reasoning, but there is no quantitative analysis showing that a model's pitch sensitivity (from the foundational tasks) predicts its performance on temporal tasks that depend on pitch tracking. Without this validation, the hierarchical structure remains a useful conceptual framework but not a proven causal diagnostic.

6. Limitations and Trade-offs

6.1 Difficulty Estimation Costs Are Not Accounted For

The assumption or constraint. The compute-optimal framework in this paper conditions strategy selection on a difficulty estimate per prompt. However, the paper uses an approach for difficulty estimation that is prohibitively expensive: generating 2048 samples per question and computing either ground-truth pass@1 (oracle) or average PRM final-answer score (predicted). The paper explicitly acknowledges this:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity" (Section 3.2)

The consequence. In a realistic deployment, the total compute cost would be difficulty estimation plus strategy execution. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. Since the difficulty estimation step alone (2048 samples) consumes 4–8× more compute than the largest test-time budgets studied (256–512 generations), the net efficiency gain when accounting for difficulty estimation would be negative—the system would use more total compute than simply running best-of-N at a high budget on every problem. The paper's headline "4× better efficiency" claim is therefore an upper bound that does not reflect deployable cost.

What evidence exists in the paper. The paper provides no experiment that measures or amortizes difficulty estimation cost. The difficulty estimation protocol is described in Section 3.2, and the compute-optimal results in Figures 4 and 8 explicitly exclude this cost. The authors flag the issue as "a key avenue for future work" but provide no empirical bounds on how cheap difficulty estimation would need to be for the approach to be net-beneficial.

Mitigation status. Not addressed experimentally. The paper suggests (Section 8) that future work could train models to directly predict difficulty from question text, or use adaptive schemes that estimate difficulty from initial samples and allocate remaining budget accordingly. No such model or scheme is developed or evaluated. Until the difficulty estimation cost is reduced to a small fraction of the total inference budget (e.g., 4–8 samples rather than 2048), the compute-optimal framework is an analytical tool rather than a practical deployment strategy.

---

6.2 Hard Problems Remain Completely Unsolved Regardless of Compute Budget

The assumption or constraint. The paper assumes that test-time compute can substitute for pretraining compute, and demonstrates this on easy-to-medium problems. However, the approach has a hard failure boundary: on the hardest difficulty quintile (bin 5), no amount of test-time compute produces meaningful improvement, and the base model's pass@1 is near zero. The paper is transparent about this:

"On the hardest questions (bin 5), no method makes meaningful progress — the base model simply lacks the capability to produce correct solutions regardless of how the budget is allocated." (Section 5.3)

The consequence. Test-time compute can amplify existing capability but cannot create it from nothing. For problems outside the base model's approximate capability range—where the model's pass@1 is effectively zero—search, revisions, and compute-optimal allocation all fail. This means the approach offers no path for genuinely novel or out-of-distribution reasoning tasks that exceed the base model's training distribution. For such problems, scaling pretraining (more parameters, more data, different data mixtures) remains the only viable path, and the compute-optimal framework provides no guidance for when a given problem will fall into this regime.

What evidence exists in the paper. Figure 3 (right) shows bin 5 accuracy hovering at 1–3% for all methods (beam search, best-of-N) across all budgets from 4 to 256 generations. Figure 7 (right) shows bin 5 at roughly 2–3% accuracy regardless of sequential-to-parallel ratio at 128 generations. Figure 9 shows the bin 5 scaling line essentially flat near 0–5% for all compute budgets, well below the 14× larger model's performance. Section 5.3 explicitly notes that "on the hardest questions (bin 5), no method makes meaningful progress."

Mitigation status. None. The paper frames this as a fundamental boundary condition rather than a solvable limitation. The FLOPs-matched analysis (Section 7) quantifies where test-time compute is preferable versus where pretraining is necessary, but does not propose methods to expand the base model's capability range. The paper's contribution is precisely characterizing where this boundary lies, not pushing it outward.

---

6.3 Single Benchmark, Single Model Family—No Evidence of Cross-Domain or Cross-Architecture Generalization

The assumption or constraint. All experiments use the MATH benchmark (500 test questions) with the PaLM 2-S* model family. The paper asserts that "this model is representative of the capabilities of many contemporary LLMs" (Section 4) and that math reasoning is a domain where test-time compute should help, but provides no evidence that the findings generalize to other benchmarks, other model families, or other reasoning domains.

The consequence. The compute-optimal strategies, the difficulty thresholds, the optimal sequential-to-parallel ratios, and the magnitude of the 4× efficiency gain may all be specific to PaLM 2-S* on MATH. A model with different calibration, different error patterns, or different in-context learning capabilities might exhibit qualitatively different difficulty-dependent scaling curves—for example, beam search might cause over-optimization at different budget levels, or the optimal difficulty bin thresholds might shift. The finding that "beam search helps on medium problems but hurts on easy problems" may not hold for models with differently-calibrated PRM scores. The paper's central contribution—the framework of difficulty-conditioned allocation—may transfer, but the specific policies and quantitative gains likely do not.

What evidence exists in the paper. All experiments in Sections 5–7 use MATH with PaLM 2-S*. There is no evaluation on GSM8K, HumanEval, or any other benchmark. There is no evaluation with a different base model (e.g., LLaMA, Qwen, Gemini). The paper does not provide ablation studies on how compute-optimal strategy selection changes with model scale within the same family. The authors acknowledge this limitation implicitly by leaving "extension to other domains" as future work (Section 8).

Mitigation status. Not addressed. The paper's contribution is framed as establishing the framework and methodology; replication on other benchmarks and model families is explicitly deferred to future work. The single-model-family design is a reasonable scope for an initial study, but it means practitioners cannot apply the specific difficulty thresholds or strategy recommendations from this paper to their own models without re-running the full analysis.

---

6.4 Sequential Revision Strategies Have Latency Costs That Are Not Addressed

The assumption or constraint. The paper measures test-time compute in "generations"—the number of complete solutions sampled—which is a reasonable proxy for total FLOPs but ignores wall-clock latency. The compute-optimal revision policy (Figure 7) often favors high sequential-to-parallel ratios on easy problems (purely sequential is optimal at low-to-moderate budgets), and a balanced ratio with substantial sequential depth on medium problems. Since each sequential revision depends on the previous one, these strategies introduce serial dependencies that cannot be parallelized.

The consequence. For latency-sensitive applications—interactive assistants, real-time decision-making, any user-facing system—a compute-optimal policy that allocates 64 generations as "16 parallel chains × 4 sequential revisions each" takes approximately 4× longer wall-clock time than running 64 parallel samples simultaneously, even though the total FLOPs are identical. If latency is the binding constraint (not total FLOPs budget), the compute-optimal policy under the paper's formulation may be suboptimal or even infeasible. The paper provides no analysis of the latency-throughput tradeoff inherent in sequential revision strategies.

What evidence exists in the paper. Figure 7 (left) shows that at lower budgets (8–32 generations), fully sequential revision is optimal. Figure 6 (right) shows that sequential revision outranks parallel sampling in aggregate. The sequential architecture is described in Section 6: "it produces an initial answer, then conditions on that answer to produce a revision, then conditions on the revision to produce another revision, and so on"—inherently serial. The paper never reports wall-clock time, latency, or the compute-latency tradeoff.

Mitigation status. Not addressed. The paper's cost model (generations as the unit of compute) is standard in the scaling literature but is a deliberate simplification that abstracts away the latency dimension. The paper does not propose latency-aware allocation policies, speculate on how to balance accuracy gains against latency constraints, or measure the actual wall-clock overhead of sequential versus parallel strategies. This is a significant gap for practitioners who must consider both dimensions when deploying inference strategies.

---

6.5 The FLOPs-Matched Comparison Uses a Potentially Weak Pretraining Baseline

The assumption or constraint. The FLOPs-matched comparison (Section 7) pits PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters, using greedy decoding with no test-time compute augmentation. The paper scales only model parameters (not training data) when increasing pretraining compute, following the LLaMA paradigm (Touvron et al., 2023) rather than compute-optimal pretraining. The paper explicitly acknowledges:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work." (Section 7)

The consequence. A Chinchilla-optimal model trained with 14× more total FLOPs—scaling both parameters and training tokens—would likely outperform a parameter-only-scaled model. The pretraining baseline in this paper is therefore weaker than it could be, and the reported advantages of test-time compute over pretraining (e.g., +27.8% relative improvement on easy questions at R << 1 with revisions) may shrink or reverse against a properly compute-optimal larger model. Additionally, the larger model uses only greedy decoding with no test-time augmentation—no majority voting, no best-of-N, no search—making it a "bare" baseline against a highly optimized test-time strategy. A fairer comparison would give the larger model at least a modest test-time budget (e.g., best-of-8 or best-of-16, which adds minimal inference FLOPs relative to the pretraining cost difference).

What evidence exists in the paper. Figure 9 and the bar charts in Figure 1 present the FLOPs-matched comparison with the 14× parameter-scaled, data-fixed baseline at three R values (0.16, 0.79, 22). The paper's explicit caveat about the compute-optimal pretraining baseline appears in Section 7. The paper does not evaluate a Chinchilla-optimal larger model, does not give the larger model any test-time compute budget, and does not analyze how sensitive the FLOPs comparison is to the pretraining scaling strategy.

Mitigation status. Partially addressed by transparency. The authors explicitly state this limitation and frame their results as specific to the LLaMA-style parameter-scaling paradigm. However, this transparency does not change the fact that the empirical claims about the test-time-vs-pretraining tradeoff are conditioned on a particular (and arguably suboptimal) pretraining baseline. Until the comparison is replicated with compute-optimal pretraining and a test-time-augmented larger model, the claim that "test-time compute can outperform a 14× larger model" should be understood as an existence proof under favorable conditions rather than a universal scaling law.

---

6.6 Revisions and PRM Search Are Studied Independently—the Combined Potential Is Unexplored

The assumption or constraint. The paper studies two mechanisms for test-time compute—PRM-guided search (Section 5) and iterative revision (Section 6)—as independent, parallel investigations. Each is evaluated separately, and the compute-optimal policies for each are derived independently. The paper never combines PRM tree-search with the revision model as the proposal distribution. The authors explicitly acknowledge this gap in Section 8:

"we did not experiment with PRM tree-search techniques in combination with revisions"

The consequence. The paper's results represent a lower bound on what a fully integrated system could achieve. The two mechanisms have complementary strengths: revisions improve the proposal distribution (generating better candidates by conditioning on previous mistakes), while PRM search improves candidate selection (finding the best among generated candidates by guided exploration). Applying beam search to revision model outputs—using the PRM to decide which revision branches to pursue—could yield gains beyond either method alone. The paper's finding that revisions help most on easy problems and PRM search helps most on medium problems (Sections 5.3, 6) further suggests that a combined approach could broaden the difficulty range over which test-time compute is effective. The current study design leaves this combination unexplored, so the true upper bound of what test-time compute can achieve with both mechanisms is unknown.

What evidence exists in the paper. The search experiments (Section 5) use the few-shot prompted base model as the proposal distribution. The revision experiments (Section 6) use majority voting or a standalone ORM for answer selection, not the PRM. The paper never reports results for "beam search with the revision model" or "PRM-guided revision chain selection." The compute-optimal policies for search and revisions are derived and presented separately (Figures 4 and 8). Section 8 lists the combination as a key direction for future work.

Mitigation status. Not addressed experimentally. The paper provides conceptual justification for why the mechanisms are complementary (the proposal-verifier decomposition in Section 2) and does not claim that the independent evaluation is optimal. However, the pragmatic consequence is that the paper's central claim—that compute-optimal scaling achieves 4× efficiency over best-of-N—is established only for each mechanism independently. Whether combining them yields >4× or whether the mechanisms interfere (e.g., revision model outputs may be poorly calibrated for the PRM, or PRM-guided revision selection may cause over-optimization similar to what affects beam search) is an open empirical question that the paper leaves unanswered.

7. Implications and Future Directions

How This Work Changes the Landscape

STAR-Bench introduces a diagnostic reframing of audio benchmark evaluation rather than an incremental extension of task coverage. The core shift is from measuring what models know about audio (semantic content recoverable from text captions) to measuring what models can hear and reason about from acoustic signals directly (linguistically hard-to-describe cues requiring fine-grained perception integrated with physical world knowledge). This is a conceptual reframing with methodological consequences: it changes what counts as evidence of audio intelligence and provides tools to detect when benchmarks are measuring the wrong thing.

The paper's most consequential methodological contribution is the caption-only audit (Figure 1): evaluating a model on benchmark questions using only textual descriptions of the audio, with the performance drop quantifying how much the benchmark depends on genuinely acoustic (non-text-recoverable) information. This is a principled falsification criterion that any audio benchmark can and should be subjected to. Prior to this work, the field lacked a simple, decisive test for whether an audio benchmark measures auditory perception or language comprehension applied to audio-derived text. The caption-only audit provides that test, and its application to existing benchmarks (MMAU drops 5.9%, MMAR drops 9.0%) reveals that much of the field's reported progress in audio understanding may reflect progress in language understanding of audio captions. This finding does not invalidate prior benchmarks—they serve their purpose for semantic-level understanding—but it reframes their scope: they measure a necessary but insufficient component of audio intelligence.

The paper also introduces hierarchical diagnostic benchmarking as a paradigm. The two-level structure—foundational perception tasks that quantitatively profile six acoustic attributes under controlled conditions, and holistic reasoning tasks that demand synthesis of perception, knowledge, and reasoning—enables causal attribution of errors rather than opaque aggregate scoring. Before STAR-Bench, if a model scored 30% on audio temporal reasoning, there was no principled way to determine whether failures were perceptual (can't hear the pitch change), knowledge-based (hears it but doesn't know what it means), or reasoning-based (perceives and knows but can't chain inferences). The foundational level provides independent evidence for or against a perceptual-deficit attribution, and the paper's error analysis (Figure 6) operationalizes the tripartite decomposition to reveal qualitatively different bottleneck profiles across models. This transforms benchmarks from ranking tools into profiling tools—a shift with direct implications for model development priorities.

Reconciling conflicting prior findings. The paper's error analysis provides a framework for understanding why different models fail on the same tasks for different reasons, which in turn explains why the audio reasoning literature has produced apparently contradictory results. Prior work has alternately claimed that LLMs can (Ma et al., 2025) or cannot (by implication from the near-random performance of most models on STAR-Bench) perform audio reasoning. The resolution is that these claims tested different models on different implicit difficulty distributions. The paper's finding that closed-source models are bottlenecked by fine-grained perception while open-source models are bottlenecked by knowledge gaps (Figure 6) means that a benchmark skewed toward tasks requiring world knowledge will show a large open-source/closed-source gap, while a benchmark skewed toward tasks requiring perceptual acuity might show a narrower gap or even different rank-ordering. The tripartite decomposition (perception, knowledge, reasoning) provides the vocabulary for precise claims about where a model's capabilities break down, replacing coarse "can/cannot reason about audio" statements.

Research directions that become more attractive. The paper's findings redirect research attention in several ways:

• Audio encoder resolution and native stereo processing become first-order priorities. The finding that even Gemini 2.5 Pro—the strongest model by a large margin—is bottlenecked by fine-grained perception (84% of temporal errors, Figure 6) means that improving audio frontends is likely higher-impact than improving reasoning architectures for the current generation of models. The channel-averaging diagnostic (Figure 3) further demonstrates that native stereo processing is a prerequisite for spatial reasoning, and that most current models lack this prerequisite entirely.

• Dense audio captioning as a path to knowledge integration becomes a high-priority research direction for open-source models. The paper's finding that open-source models suffer from 54% knowledge gaps (vs. 8% for Gemini 2.5 Pro, Figure 6) and cannot leverage additional information when provided (flat ablation curves in Figure 9) suggests that training on fine-grained, acoustically-detailed audio captions—rather than coarse semantic labels—may be the most direct path to building the world knowledge needed for audio reasoning. The paper explicitly argues: "enabling them to generate fine-grained acoustic descriptions is critical toward more robust reasoning."

• Multi-audio comparison and grounding becomes a recognized capability gap. The ablation study (Figure 9) demonstrates that open-source models show essentially flat performance even when given the uncut reference audio—a task that reduces to straightforward segment-to-reference matching. This identifies the inability to compare across multiple audio inputs as a fundamental limitation that no amount of additional context can compensate for. Training paradigms that explicitly teach models to align, compare, and integrate information across multiple audio streams become a clear priority.

Research directions that become less attractive. The paper's negative results also close off certain lines of inquiry:

• Reasoning enhancements (chain-of-thought, "think" modes) without corresponding perceptual improvements. The paper's finding that Audio Flamingo 3 Think and Xiaomi-MiMo-Audio Think perform worse than their non-thinking counterparts (Table 2) suggests that adding reasoning depth to models with inadequate perceptual or knowledge foundations is counterproductive. The paper's interpretation—"without sufficiently solid perceptual and knowledge foundations, reasoning can be ineffective or even detrimental"—implies that reasoning enhancements should not be a universal add-on but should be conditioned on perceptual capability.

• Generic audio QA benchmarks that do not audit for text-recoverability. The caption-only audit establishes a new validity standard. Benchmarks that cannot demonstrate substantial caption-only performance drops (substantially above the ~6–9% seen for MMAU and MMAR) will be increasingly viewed as measuring language comprehension rather than audio intelligence. The paper provides both the diagnostic method and the evidence threshold (~30%+ drops on genuinely acoustic tasks), making it straightforward for future benchmark authors to validate their designs.

• Spatial audio evaluation without addressing the channel-averaging problem. The paper demonstrates that most current models destroy binaural information during preprocessing (Figure 3), making spatial task performance under native input a measure of preprocessing limitations rather than reasoning capability. Future spatial audio benchmarks will need to adopt the dual native/channel-wise evaluation strategy—or models will need to support native stereo processing—for spatial task performance to be interpretable as a capability measure rather than a preprocessing artifact.

---

Follow-Up Research This Work Enables

1. Training open-source models for dense, fine-grained audio captioning and measuring the effect on reasoning. The paper's error analysis identifies knowledge gaps (54% of Qwen-2.5-Omni's temporal errors) as the dominant bottleneck for open-source models, and the paper argues that "advanced world knowledge is deeply embedded within detailed audio-text captioning." A direct follow-up would be to fine-tune an open-source audio-language model (e.g., Qwen2-Audio-Instruct or Audio Flamingo 3) on a dataset of temporally-dense, acoustically-detailed captions—descriptions that include pitch trajectories ("the pouring sound starts low and gradually rises"), loudness dynamics ("the siren swells then fades"), and spatial qualifiers ("the voice moves from left to center to right")—and then re-evaluate on STAR-Bench. The hypothesis is that fine-grained captioning training would reduce knowledge gaps specifically, leaving perception errors (which require encoder improvements) largely unchanged. A strong follow-up would compare: (a) baseline model, (b) model fine-tuned on standard captions, (c) model fine-tuned on dense acoustic captions, measuring both overall accuracy and the error category distribution (perception/knowledge/reasoning) before and after training. The paper's error taxonomy provides the measurement framework for this experiment.

2. Developing and evaluating native stereo audio encoders for spatial reasoning. The channel-averaging diagnostic (Figure 3) and the spatial reasoning results (Table 6) demonstrate that current models are architecturally incapable of spatial hearing under default configurations. A direct engineering follow-up would be to modify an existing open-source audio encoder (e.g., Whisper, CLAP, or the Qwen2-Audio encoder) to preserve binaural channels through the encoding pipeline—either by encoding left and right channels separately and fusing at a later stage, or by designing interaural feature extractors that compute ITD and ILD explicitly—and then evaluate the modified model on STAR-Bench's spatial tasks under native input. The metric of success is whether native-input spatial performance approaches or exceeds channel-wise performance (since native processing would avoid the multi-audio alignment problem that degrades channel-wise performance). Given that the paper shows most models achieve only ~40–45% on spatial tasks even with channel-wise input (vs. 73.7% human), a follow-up would also need to investigate whether better spatial perception translates to better spatial reasoning, or whether reasoning capability is an independent bottleneck.

3. Testing physical understanding through adversarial and counterfactual audio examples. The paper frames its tasks as requiring "physical world knowledge" but cannot distinguish genuine physical reasoning from dataset correlations. A stress-test follow-up would construct adversarial variants of STAR-Bench temporal tasks where the acoustic cues are manipulated to conflict with physical expectations. For example: (a) a pouring-water sequence where the pitch is artificially lowered as the container fills (inverting the normal acoustic-physical relationship), testing whether models follow the physical principle (fill order: empty → full, based on pitch) or the acoustic pattern (rising pitch = later time); (b) a Doppler sequence where the frequency shift is reversed (approaching sounds have decreasing pitch), testing whether models use the Doppler cue or the semantic association "approaching → louder"; (c) a causal sequence where the trigger and consequence sounds are swapped temporally but the causal relationship is preserved linguistically (e.g., glass shattering sound followed by firework explosion, but captioned as "the firework caused the glass to shatter"). A model using genuine physical understanding should correctly identify the physical sequence despite the manipulated acoustics; a model using pattern matching should follow the acoustic cues to the wrong answer. This experiment would directly test the paper's central construct validity claim—that STAR-Bench measures physically-grounded reasoning rather than acoustic pattern recognition.

4. Cross-modal transfer: do vision-capable models show visual spatial biases on audio spatial tasks? The paper's error analysis notes that some spatial task errors involve "vision-centric hallucinations (e.g., '...based on the car's trajectory in the video...')," suggesting modality confusion in omni-modal models. A systematic follow-up would evaluate a set of models with varying degrees of visual training (audio-only LALMs, audio+vision OLMs, text-only LLMs with audio encoders) on STAR-Bench's spatial tasks, measuring both accuracy and the rate of visual-referencing errors in model outputs. The hypothesis is that models extensively trained on visual spatial reasoning (e.g., video QA, embodied navigation) will show higher rates of inappropriate visual reasoning on audio-only spatial tasks, reflecting negative transfer. A strong follow-up would also test whether explicit instructions to "ignore visual knowledge and reason only from the audio" reduce hallucination rates, distinguishing between automatic cross-modal interference and strategic over-reliance on vision. The paper's dual native/channel-wise spatial evaluation provides the appropriate baseline for measuring spatial capability independent of visual confounds.

5. Scaling laws for audio perceptual resolution: how does model size, encoder quality, and training data volume affect foundational perception sensitivity? The paper's foundational perception tasks produce quantitative sensitivity curves (Figure 8, second row) for pitch, loudness, and duration discrimination at six difficulty levels. These curves enable a scaling-law analysis that the paper does not perform: evaluate a single model architecture at multiple scales (e.g., 1B, 3B, 7B, 13B, 70B parameters) or with varying audio encoder capacities (e.g., different Whisper model sizes paired with the same LLM backbone) on the foundational perception battery, and measure how discrimination sensitivity scales with model size and encoder quality. The hypothesis is that perceptual sensitivity follows a power-law relationship with encoder capacity (more parameters → finer JND thresholds), but may plateau once the encoder resolution exceeds the information content of the audio signal. This analysis would provide the first empirical characterization of audio perceptual scaling laws, analogous to how vision models show predictable improvements in acuity and texture discrimination with scale. It would also inform the practical question of how large an audio encoder needs to be to support the level of perceptual resolution required for holistic reasoning tasks.

6. Dynamic difficulty estimation: can initial samples predict problem difficulty well enough to guide strategy allocation without the prohibitive cost of 2048-sample estimation? While this is a "compute-optimal" paper implication carried forward from the prior sections, it applies with equal force to audio 4D intelligence evaluation: the paper's difficulty estimation approach for temporal and spatial tasks is implicit rather than explicit (the benchmark is static), but the diagnostic hierarchy (foundational → holistic) enables a related question. A follow-up would test whether a small number of foundational perception probe trials (e.g., 4–8 pitch discrimination questions at varying Δ) can predict a model's holistic temporal reasoning accuracy at above-chance levels. If a model's pitch sensitivity curve from 8 trials correlates strongly with its segment reordering accuracy on pitch-dependent tasks (pouring water, Doppler motion), then a lightweight perceptual screening could serve as a "difficulty estimator" for whether a given model is likely to succeed on a given holistic task category. This would address a practical limitation of the hierarchical design: running the full foundational battery plus all holistic tasks is expensive. If a sparse sampling of foundational tasks can predict holistic performance, the benchmark becomes more efficient to administer while retaining its diagnostic value. The specific experiment: compute the correlation between per-model foundational attribute sensitivity (pitch AA at Δ=50, 100, 200 cents) and per-model accuracy on temporal subcategories that depend on that attribute (object spatial motion, in-situ state evolution), using the 19 models already evaluated as data points. A high correlation would validate the hierarchical diagnostic link that the paper asserts but does not empirically verify.

---

Practical Applications and Downstream Use Cases

1. Audio encoder selection and procurement for specific deployment domains. Organizations building audio AI systems for domain-specific applications can use STAR-Bench's foundational perception battery to select audio encoders matched to their perceptual requirements. For example, a factory monitoring system that needs to detect subtle changes in machine pitch (indicating bearing wear) requires high pitch discrimination sensitivity at specific frequency ranges. The audiogram visualization (Figure 8, first row) and sensitivity curves (Figure 8, second row) enable direct comparison of encoder performance on the perceptual dimensions relevant to the application, without requiring the full holistic benchmark. The paper's finding that even Gemini 2.5 Pro shows substantial sensitivity degradation at small Δ values (e.g., ~20% ACR at Δ=50 cents for pitch, Figure 8e) provides a quantitative baseline for what level of perceptual resolution is currently achievable and where custom encoder training may be necessary.

2. Diagnostic profiling for model improvement prioritization. The paper's error taxonomy and hierarchical design provide a decision framework for model development teams. A team working on an open-source audio-language model can run STAR-Bench, examine their error distribution (analogous to Figure 6), and identify their primary bottleneck. If the distribution shows >50% knowledge gaps (as Qwen-2.5-Omni does on temporal tasks), the priority is improving training data diversity and caption quality. If it shows >70% perception errors (as GPT-4o Audio does), the priority is improving audio encoder resolution or temporal tracking capability. If the AA-ACR gap is near-total (ACR ≈ 0 with AA modestly above random), the priority is addressing output instability and position bias—the model is sometimes right for wrong reasons and cannot be relied upon. This profiling use case does not require the benchmark to be solved; it requires the benchmark to reveal how a model fails, which STAR-Bench demonstrably does.

3. Safety validation for embodied and autonomous systems that rely on auditory situational awareness. The paper's motivating examples—inferring the trajectory and distance of an approaching vehicle, monitoring machine operations through sound—are safety-critical. STAR-Bench's spatial reasoning tasks (dynamic trajectory tracking, multi-source spatial relations) and temporal reasoning tasks (object spatial motion via Doppler cues) directly test these capabilities. Before deploying an audio AI system in a safety-critical context (e.g., an autonomous vehicle using auditory cues for emergency vehicle detection, a robot operating near humans), STAR-Bench can serve as a capability audit: if a model achieves near-random performance on dynamic trajectory tracking under native input (as most models do, with only 28–45% accuracy where random is 33%), it should not be trusted to make spatial inferences from audio. The paper's human baseline (77% on trajectory tracking) establishes the performance level that a reliable system should approach. The benchmark's explicit inclusion of "unable to determine" options in foundational tasks also provides a calibration signal for whether a model appropriately expresses uncertainty—a critical safety property that binary accuracy metrics miss.

4. Curriculum design for audio-language model training. The paper's hierarchical task structure (foundational → holistic) and the finding that open-source models fail on holistic tasks partly because they lack fine-grained perceptual sensitivity suggest a training curriculum strategy. Models could be trained first on foundational perception objectives—explicitly learning to discriminate pitch, loudness, duration, and spatial attributes at multiple Δ levels—before being trained on holistic reasoning tasks. The foundational tasks' parameterized difficulty levels (4–6 Δ values per attribute, Table 3) naturally support curriculum learning, with models progressing from large-Δ (easy) to small-Δ (hard) discrimination. The hypothesis is that perceptual pretraining on controlled synthetic data would improve sample efficiency and final performance on holistic reasoning compared to training directly on complex real-world tasks. The foundational battery provides both the training data generation recipe (procedural synthesis with Pyroomacoustics for spatial, pure tone generation for non-spatial) and the evaluation framework to measure whether perceptual training transfers to reasoning improvement.