Scaling Self-Supervised Representation Learning for Symbolic Piano Performance

🎯 Pitch

A model trained on 60,000 hours of MIDI transcriptions—far beyond the usual symbolic-music datasets—generates piano continuations indistinguishable from Suno 3.5 and human compositions in blind listening tests. Its contrastive embeddings also achieve state‑of‑the‑art classification accuracy, yet the contrastive objective works only when initialized from a pretrained generative model, not from scratch.

1. Executive Summary

This paper introduces Aria, a generative autoregressive transformer model pretrained on approximately 60,000 hours of transcribed solo-piano MIDI data, then adapted through targeted finetuning phases to perform musical continuation, symbolic classification, and contrastive embedding generation via a novel adaptation of SimCLR to symbolic music (extracting non-overlapping slices from the same MIDI file as augmented views). In human listening tests, Aria's piano continuations outperform the Anticipatory Music Transformer and prove statistically indistinguishable from both Suno 3.5 and human-composed ground truth, while frozen contrastive embeddings achieve state-of-the-art linear probe accuracy on composer, genre, and style classification benchmarks. The paper establishes that contrastive representation learning is effective for symbolic music only when initialized from pretrained next-token-prediction representations—training from scratch on 25 embedding pairs per file underperforms a single-epoch finetune on one pair per file—and that finetuned representations require as few as several hundred labeled examples to specialize to downstream tasks.

2. Context and Motivation

The Core Problem: Symbolic Music Modeling Hasn't Benefited from Scale

The fundamental problem this paper addresses is deceptively simple: symbolic music modeling has largely been left out of the scaling revolution that has transformed other modalities. While text, image, and audio domains have seen dramatic improvements from training ever-larger models on ever-larger datasets—LLaMA for language, CLIP for vision-language, AudioLM and MusicLM for audio—symbolic music research remains constrained to comparatively tiny, manually curated datasets. The paper's central observation is that this is not due to any inherent limitation of symbolic representations, but rather to a data acquisition bottleneck that has prevented the field from exploring what happens when self-supervised learning is applied at the scale common in neighboring modalities.

This gap matters for three interconnected reasons the paper weaves together:

1. Symbolic music offers unique affordances that audio modeling cannot match. Unlike raw audio, symbolic representations (MIDI) provide direct editability, interpretable musical structure, compact storage, and precise control over individual notes, dynamics, and timing. A model that operates in this space could serve as an intelligent compositional assistant, a musicological analysis tool, or a controllable generation engine in ways that black-box audio models cannot. The paper mentions these applications implicitly through its evaluation choices—composer identification, genre classification, style detection—but the broader implication is that symbolic models unlock a fundamentally different set of creative and analytical workflows than audio-based systems.

2. The data scaling ceiling has been a self-fulfilling constraint on the field's research ambitions. Because labeled or manually-verified symbolic music data is expensive to create (the paper notes this is "labor-intensive"), prior work has designed its research questions around the limitations of available datasets. Section 2 points out that "successful research often involves training from scratch on datasets such as Lakh and IMSLP, with research problems formulated around tasks that directly align with these datasets." This means the field has been shaped by data scarcity: multi-track generation was studied because Lakh provides multi-track files; baroque chorale harmonization was studied because well-formed datasets of Bach chorales exist. In contrast, text and vision research has developed generalist models first, then adapted them to specific tasks—a workflow that symbolic music research has been structurally unable to adopt.

3. There is a neighboring success story that demonstrates what's possible at scale. The paper draws explicit contrast with the audio domain, where "large-scale models including AudioGen and AudioLM, alongside their underlying neural audio codecs, have driven a broad range of advancements in music generation." Audio MusicLM and Suno produce musically coherent, expressive output not because of fundamentally different architectures, but because they can train on enormous amounts of readily available audio data. The symbolic music community has lacked an analogous pipeline—until recently.

Why This Gap Persists: The Data Bottleneck and Why It's Breaking Now

The paper identifies a specific technological stack that has made large-scale symbolic music data feasible for the first time. The key development is automatic music transcription (AMT), which converts raw audio recordings into MIDI representations. Modern AMT models achieve "highly reliable note-identification accuracy" for solo piano, enabling what the paper describes as "automated dataset curation pipelines that crawl raw audio and transcribe it into MIDI using a combination of web scraping, audio-based processing, and AMT methods."

The critical dataset enabling this work is Aria-MIDI, introduced by Bradshaw and Colton (2025). At approximately 100,000 hours, it is "orders of magnitude larger than similar datasets" like GiantMIDI-Piano or ATEPP. This represents a qualitative shift, not just a quantitative one. Prior datasets were large enough to train specialized models on specific tasks; Aria-MIDI is large enough to ask whether the foundational pretraining-then-adaptation paradigm—the same paradigm that produced CLIP and LLaMA—works for symbolic music.

Crucially, the paper emphasizes that transcriptions from AMT capture something that manually entered MIDI does not: "as this symbolic data is transcribed from real recordings, it captures the subtleties and dynamics of human performance." This is a non-trivial point. Human piano performances contain expressive timing deviations, velocity variations, pedaling, and micro-level phrasing that are typically absent from score-derived MIDI (e.g., from MuseScore or MusicXML). Models trained on such data can learn not just what notes to play, but how to play them—the distinction between a mechanical rendition and a human-like performance.

Where Existing Approaches Fall Short

The paper identifies limitations in prior work along three axes that it positions itself to address:

Generative symbolic models have been limited by dataset scale and tokenization quirks. Music Transformer (Huang et al., 2018) demonstrated that autoregressive transformers can learn long-term musical structure when trained on the MAESTRO dataset (expressive MIDI piano performances). Its follow-up work "provided strong evidence that the musical and creative capabilities of their model scale well with dataset size—reinforcing the value of curating large-scale piano transcription datasets as a future direction." This is the direct precedent for Aria's central premise, but Music Transformer was trained on approximately 200 hours of data—Aria trains on 60,000+ hours, a roughly 300× increase. Subsequent models like MuseNet expanded to multi-instrument data but also expanded their architectural complexity; the paper doesn't directly critique this but implicitly argues that scale trumps architecture, using a "conventional modern transformer architecture with minimal modifications."

The paper also identifies a specific technical limitation in prior tokenization schemes: relative-timing representations cause temporal instability. Music Transformer and MuseNet use "time-shift tokens" that specify how long to wait before the next note event. The paper argues—and cites evidence from sequence-to-sequence transcription work—that "autoregressive models struggle to maintain an exact temporal representation of the prior context, as they must sum up many sequential time-shift values to calculate temporal relationships between notes with medium or long-term dependencies." This is a subtle but important architectural insight: transformers can do arithmetic only unreliably (citing Lee et al., 2023 and McLeish et al., 2024), so requiring them to accumulate many small time-shift values to determine whether two notes are simultaneous is asking for trouble. The paper observed this firsthand: "in preliminary investigations, we also observed negative effects when using relative-timing tokenizations, particularly on temporal instability in passages with rapid note sequences." This motivates the paper's hybrid absolute-relative scheme (Equation 1), which replaces the summation over time-shift tokens with a simple counting of segment boundary tokens plus within-segment offsets—an arithmetic operation the model can perform reliably.

Representation learning for symbolic music lacks a scalable, general-purpose framework. Existing approaches fall into two categories, each with limitations. MusicBERT (Zeng et al., 2021) applies self-supervised learning via a bar-masking objective, but this operates at the bar level—it presupposes bar and beat information that is not reliably available in transcribed performances. The CLaMP series (Wu et al., 2023, 2024, 2025) builds cross-modal embeddings between symbolic music and natural language descriptions, which is powerful for retrieval but requires paired text-symbolic data and may not capture fine-grained performance nuances that are hard to describe in words. MusicVAE (Roberts et al., 2018) learns hierarchical latent representations but is architecturally specialized and not designed to scale as a general-purpose backbone.

What's missing—and what the paper identifies as its second core contribution—is a general-purpose semi-supervised framework that can produce symbolic music embeddings capturing both compositional and performance-level features without requiring metadata or paired text. The paper's SimCLR adaptation (Section 3.3) fills this gap by treating different slices of the same MIDI file as positive pairs, forcing the model to learn file-level attributes (genre, composer, performance style) rather than local note-level details. This is clever because it exploits a natural structure in the data—a single piano recording has consistent stylistic and performer-specific characteristics throughout—without requiring any labels.

The pretraining-finetuning paradigm has not been tested at scale for symbolic MIR. While the NLP community has established that large-scale pretrained transformers (BERT, T5, GPT) can be effectively finetuned for downstream tasks with limited labeled data, the paper observes that symbolic music research has not replicated this workflow. The MusicBERT and MidiBERT-piano models demonstrate pretraining for symbolic music, but at smaller scale and with architectures designed specifically for music. The paper aims to test whether a straightforward adaptation of a standard language model architecture (LLaMA 3.2, with simplified attention and normalization), trained on a diverse, web-scale corpus of piano transcriptions, exhibits the same transfer learning properties that have made pretrained language models so valuable. The supervised finetuning experiments—where the model achieves competitive performance with as few as 100–500 labeled examples—are designed to validate precisely this hypothesis.

How This Paper Positions Itself

The paper frames its contribution not as a single architectural innovation but as a systems-level demonstration that the SSL scaling paradigm transfers to symbolic music when the data bottleneck is removed. The authors state this explicitly in their abstract: "we leverage Aria-MIDI to pretrain a generative transformer model via next-token prediction, using it as a foundation to explore the effectiveness of SSL techniques applied to symbolic music at a scale closer to recent applications in the text, image, and audio domains."

The positioning has several key elements:

Architectural minimalism signals that data scale is the primary variable of interest. The model is described as "a conventional modern transformer architecture with minimal modifications" built on LLaMA 3.2, with the hidden dimension actually reduced from 2048 to 1536 (following Hoffmann et al., 2022's guidance on compute-optimal model-data ratios) and with the attention mechanism simplified from grouped-query attention back to standard multi-head attention. This is not a paper about novel architectures—it's a paper about whether scale alone, with careful data curation and tokenization, can produce strong results.

The paper spans both generative and representational evaluation to establish breadth. Rather than specializing in one capability area, the paper evaluates across three distinct axes—generation quality (human listening tests), frozen representation quality (linear probes on multiple classification benchmarks), and finetuning adaptability (supervised finetuning with varying label quantities). This multi-faceted evaluation is the paper's argument that next-token pretraining produces genuinely general-purpose representations, not just a good generative model.

The contrastive learning contribution is positioned as a methodology finding, not just a model release. The paper's ablation study comparing contrastive training from scratch versus contrastive finetuning from pretrained weights demonstrates that "our contrastive method is competitive only when applied as a finetuning stage." This is a methodological insight that generalizes beyond their specific model: it suggests that for symbolic music, next-token prediction pretraining learns representations that serve as essential scaffolding for contrastive embedding learning, mirroring findings in text embedding literature (SimCSE, Gao et al., 2021) but extending them to a new modality.

The paper connects to both the symbolic and audio MIR communities. By including audio-based baselines (MERT, MusicGen, Suno) alongside symbolic baselines (Anticipatory Music Transformer, CLaMP, M3), the paper positions symbolic music modeling as a competitive alternative to audio-based approaches for certain tasks—particularly those requiring editability, interpretability, or fine-grained control. The human evaluation finding that Aria's continuations are statistically indistinguishable from Suno 3.5 and human ground truth is a direct challenge to the assumption that audio models are inherently superior for music generation.

The paper acknowledges clear scope limitations. It explicitly states that "our benchmarks focus exclusively on piano performances, whereas the comparison models support multi-instrument MIDI or audio files," that hard problems like Black MIDI are filtered out during data preprocessing, and that the model "could not include closed-access models like AudioLM." This transparency about scope establishes credibility and frames the work as a foundation for future multi-instrument and multi-track extensions rather than a final solution.

3. Technical Approach

3.1 Reader Orientation

This is a systems-level empirical investigation rather than an architectural innovation paper — the core idea is that removing the data bottleneck (via AMT-based transcription at web scale) and applying standard self-supervised learning recipes with careful tokenization is sufficient to produce a foundation model for symbolic piano music that matches or exceeds both symbolic and audio baselines across generation and representation tasks. The paper doesn't propose a fundamentally new model architecture; instead, it adapts a standard LLaMA 3.2 transformer, designs a hybrid absolute-relative MIDI tokenization scheme that avoids the arithmetic weaknesses of prior relative-timing approaches, and then demonstrates that the resulting pretrained representations transfer effectively through three downstream adaptation paths: generative finetuning for music continuation, supervised finetuning for MIR classification, and a novel SimCLR-based contrastive finetuning stage that produces general-purpose MIDI embeddings without requiring any labels or metadata.

3.2 Big-Picture Architecture (Diagram in Words)

The system comprises five major components connected in a pipeline:

Data curation pipeline — ingests raw YouTube-sourced piano audio, transcribes it to MIDI via automatic music transcription (AMT), applies heuristic filters to remove Black MIDI, near-silent, or repetititve content, and deduplicates overrepresented compositions using compositional metadata (opus/piece-number tags). Output: a refined corpus of 820,944 MIDI files (~60,473 hours).
MIDI tokenizer — converts MIDI note events into a flat token sequence using a hybrid absolute-relative timing scheme. Notes are grouped into 5000-millisecond segments marked by <T> tokens; within each segment, onset times are absolute (relative to segment start). Pitches, velocities (12 bins), instruments (13 classes), and durations are encoded as discrete integer tokens.
Base autoregressive transformer — a 650M-parameter decoder-only model (modified LLaMA 3.2: d_model = 1536, 24 standard multi-head attention heads, layer normalization, no grouped-query attention or RMS norm) pretrained via next-token prediction on concatenated 8192-token sequences with online data augmentation (transposition, tempo scaling, velocity adjustment).
Generative finetuning stage — a single-epoch learning rate anneal on higher-quality, non-concatenated data with file-start alignment. A special <D> token is inserted ~100 tokens before sequence end to enable explicit generation termination control. Output: the model used in human listening tests.
Contrastive finetuning stage — replaces the language modeling head with a projection to a 512-dimensional embedding space. Extracts two non-overlapping augmented slices (100–650 notes each) from the same MIDI file as positive pairs and applies the SimCLR NT-Xent contrastive loss. Embeds the hidden state at an end-of-sequence token appended after each slice. Output: the model used for linear probe classification experiments.

Information flows sequentially: raw audio → AMT → MIDI files → preprocessing filters → tokenizer → 8192-token concatenated sequences → pretraining → (generative finetuning branch) OR (contrastive finetuning branch). For supervised classification, the pretrained model is finetuned end-to-end with a classification head replacing the language modeling head, without the contrastive intermediary.

3.3 Roadmap for the Deep Dive

First, the hybrid tokenization scheme (Section 3.1) — because every subsequent component depends on how musical information is encoded as discrete tokens. Understanding why the paper chose absolute-relative timing over pure relative-timing (used by Music Transformer and MuseNet) or beat-based approaches (used by REMI) clarifies the architectural motivation.
Second, the pretraining data pipeline — the paper's central claim is that data scale, enabled by AMT, is the enabling factor. The curation, filtering, and augmentation steps matter because they determine what the model actually learns from 60,000 hours of internet-sourced transcriptions.
Third, the base model architecture and pretraining recipe — the specific architectural choices (deviation from LLaMA 3.2, model-data ratio motivation, sequence length, optimization hyperparameters) and their justifications.
Fourth, the generative finetuning procedure — what changes between pretraining and the generative variant used in human evaluations, and why those changes matter for evaluation quality.
Fifth, the contrastive learning adaptation (SimCLR for MIDI) — this is the paper's methodological innovation. Understanding how the authors mapped the SimCLR framework onto symbolic music sequences, why the slice-extraction strategy captures file-level rather than local features, and how the NT-Xent loss operates in this domain is essential for evaluating the embedding results.
Sixth, the supervised finetuning procedure — how the pretrained backbone is adapted for classification, including the progressive dropout strategy and the label-efficiency experiments.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily an empirical systems paper whose core technical contribution is the methodology for adapting established SSL techniques — next-token prediction pretraining, SimCLR contrastive learning, and supervised finetuning — to symbolic piano music at scale, enabled by a carefully designed hybrid tokenization scheme and a curated AMT-derived dataset.

MIDI Tokenization: The Hybrid Absolute-Relative Timing Scheme

The tokenization design is arguably the most consequential technical decision in the paper, because it determines what information is discretized into the model's vocabulary and how temporal relationships between notes are represented in the token stream. The paper explicitly critiques prior approaches and provides a principled motivation for its design, which is worth examining in full detail.

Temporal resolution and discretization. The tokenizer operates at a temporal resolution of 10 milliseconds for both note onset times and note durations. Note velocity values are discretized into 12 bins. These choices represent a tradeoff: 10 ms resolution is fine enough to capture expressive timing nuances in human performance (typical expressive deviations are in the 20-50 ms range), but coarse enough to keep the vocabulary manageable and sequence lengths reasonable. The paper does not provide an explicit vocabulary size for the full tokenizer, but the discretization choices imply: instrument tokens (13 classes + 1 percussion), pitch tokens (128 MIDI pitches), velocity tokens (12 bins), onset tokens (500 possible values within a 5000 ms segment at 10 ms resolution), and duration tokens (similarly up to 500 possible values, though durations exceeding 5000 ms could require special handling).

Multi-instrument support. The tokenizer is designed to "natively handle multi-track (multi-instrument) MIDI files by condensing the 128 MIDI instruments, corresponding to program_change MIDI messages, into 13 instrument classes, including one for percussion." This is a significant simplification — rather than representing all 128 General MIDI program numbers as distinct tokens, the paper reduces them to 12 melodic instrument classes plus a percussion class. The paper doesn't enumerate the 13 classes, but this condensation likely groups acoustically related instruments (e.g., all bowed strings into one class, all brass into another) to reduce the effective vocabulary size. Importantly, although this multi-instrument capability is built into the tokenizer, the paper's experiments focus exclusively on solo piano data — the capability exists for future expansion.

Token triple representation for melodic notes. For any non-percussion instrument, a single note is tokenized as a sequence of three tokens:

$[\text{instrument}, p, v], [\text{onset: } t_{\text{on}}], [\text{duration: } t_{\text{off}} - t_{\text{on}}]$

where $p$ is the MIDI pitch number (0–127), $v$ is the discretized velocity bin (0–11), $t_{\text{on}}$ is the absolute onset time in milliseconds relative to the start of the current 5000 ms segment, and $t_{\text{off}} - t_{\text{on}}$ is the duration in milliseconds.

What this triple represents operationally. When the tokenizer processes a MIDI file, it first resolves all note_on and note_off events into complete notes — each note has a definite start time, end time, pitch, velocity, and instrument assignment. For each note, it emits a three-token sequence: the first token simultaneously encodes the instrument class, pitch, and velocity as a single vocabulary entry (these are concatenated into one token, not three separate tokens); the second token encodes the onset time relative to the current segment boundary; the third token encodes the duration. Notes are ordered by onset time, so the token sequence reads left-to-right in chronological order of note beginnings.

Why three tokens per note: the paper groups instrument, pitch, and velocity into a single token because these three attributes are determined at the moment a note begins and are intrinsically linked — changing any one of them changes the musical event. Onset and duration, by contrast, are continuous-valued temporal properties that benefit from separate, finer discretization. This decomposition means the model can learn joint distributions over (instrument, pitch, velocity) triples — which correspond to the "what note is played and how hard" decision — independently from the "when does it start and how long does it last" decisions, which are musically distinct constraints.

Percussion tokenization. Percussion notes use a simplified two-token scheme:

$[\text{drum}, n], [\text{onset: } t_{\text{on}}]$

where $n$ is the MIDI note number for the percussion instrument (which in the MIDI standard maps to specific drum sounds — e.g., note 36 is bass drum, note 38 is snare). Percussion notes have no duration token because drum hits are effectively instantaneous events; the paper does not explicitly state whether duration is implicitly zero or is omitted for vocabulary efficiency.

The 5000-millisecond segment structure and the <T> token. This is the core innovation that distinguishes Aria's tokenizer from both pure-relative and pure-absolute schemes. The paper divides each MIDI file into non-overlapping 5000-millisecond (5-second) segments, and marks the beginning of each new segment with a special <T> token. Within each segment, note onset tokens express the onset time relative to the start of that segment — so an onset token value of 2500 means the note starts 2500 ms after the most recent <T> token.

This creates a hybrid representation: across segments, the <T> tokens provide coarse absolute timing information (each <T> advances the timeline by exactly 5000 ms), while within each segment, onset tokens provide fine-grained local timing. The paper states this design was chosen to "resemble note timing using beat-position within a bar" — that is, just as a beat-position representation tells you where a note falls within a bar without requiring you to sum all previous note durations, the segment-relative scheme tells you where a note falls within its local temporal window without requiring arithmetic over preceding time-shift tokens.

Equation 1 formalizes the difference between timing schemes. When the model needs to compute the temporal separation between two notes $n_i$ and $n_j$ (with $i > j$ , meaning $n_i$ occurs later), the required arithmetic varies dramatically by tokenization scheme:

$T_i - T_j = \begin{cases} \sum_{k=j+1}^{i} w_k & \text{Relative} \\[4.3pt] C(\texttt{<T>}, i, j) + \tilde{o}_i - \tilde{o}_j & \text{Hybrid (Ours)} \\[4.3pt] o_i - o_j & \text{Absolute} \end{cases}$

where $w_k$ is the length of the time-shift message preceding note $k$ , $C(\texttt{<T>}, i, j)$ is the total time spanned by complete 5000 ms segments between notes $n_i$ and $n_j$ (calculated by counting the number of <T> tokens and multiplying by 5000), $\tilde{o}_k$ is the adjusted absolute onset time of note $k$ relative to the start of its 5000 ms segment, and $o_k$ is the absolute onset time of note $k$ in milliseconds from the beginning of the file.

What Equation 1 computes: the temporal distance between any two notes in the sequence, which is the fundamental operation needed for the model to understand rhythmic relationships, simultaneous onsets, inter-onset intervals, and phrase-level timing patterns. It shows the arithmetic complexity required for each scheme.

Why the hybrid form is preferable. In the Relative case, computing $T_i - T_j$ requires summing $i - j$ distinct time-shift values $w_k$ , which may themselves be generated by the model and subject to cumulative error. If the model's internal representation of any single time-shift token is slightly off, the error accumulates linearly with the number of intermediate notes. For passages with rapid note sequences (e.g., 16th-note runs at a fast tempo), $i - j$ can be large even for musically adjacent events, making the cumulative error substantial. The paper cites evidence that transformers struggle with exact accumulation of many small quantities — Lee et al. (2023) and McLeish et al. (2024) demonstrate this for arithmetic tasks — and notes observing "temporal instability in passages with rapid note sequences" in their own preliminary experiments.

In the Hybrid case, the computation decomposes into two parts that are each much simpler: $C(\texttt{<T>}, i, j)$ requires only counting the number of <T> tokens between the two notes (an integer count that has no floating-point precision issues), and $\tilde{o}_i - \tilde{o}_j$ requires subtracting at most two within-segment offsets (each bounded to $[0, 5000)$ ). At worst, when the two notes are in the same segment, $C = 0$ and the computation is a single subtraction; when they span multiple segments, $C$ is a multiple of 5000 and the subtraction is still only over at most two values. There is no cumulative summation over many generated tokens.

In the Absolute case, the computation is trivial — a single subtraction — but this scheme would require the vocabulary to include onset tokens covering the full temporal range of potentially hour-long performances (millions of milliseconds), which is vocabulary-prohibitive. The hybrid scheme achieves the arithmetic simplicity of absolute timing while keeping the vocabulary bounded to the 5000 ms segment window.

Comparison with prior tokenization schemes (Figure 2). The paper provides a visual comparison of four approaches applied to a short piano-roll excerpt:

Music Transformer uses a "time-shift" token after each note to specify how long to wait before the next event. The piano roll shows notes with variable gaps between them; in the tokenized representation, these gaps become explicit time-shift tokens with values like 120 ms, 80 ms, etc.
MuseNet uses a similar approach but with its own vocabulary encoding for tracks and instruments.
REMI uses a beat-tracking model to estimate bar and beat positions, then represents note onsets as positions within a beat, with special tokens for beat and bar boundaries. This requires the beat tracker to be accurate, which is not guaranteed for expressive rubato performances.
Aria (Ours) shows the segment-based approach with <T> markers every 5000 ms and onset tokens relative to each segment, with no beat or bar information.

Why not use REMI or beat-based approaches. The paper's key practical constraint is that their data comes from transcribed audio performances, not from digital scores. Transcriptions from AMT do not contain reliable beat or bar information — they only contain note onset times and durations inferred from the audio signal. While beat-tracking models exist (the paper cites Böck et al., 2016), they introduce their own errors, especially on music with expressive tempo variations (common in solo piano). A tokenization scheme that depends on accurate beat tracking would compound transcription errors with beat-tracking errors. The hybrid absolute-relative scheme sidesteps this entirely by working directly with millisecond-level timings from the AMT output.

Metadata handling. The paper explicitly states that "MIDI metadata, such as key, tempo, and time signature, is discarded, and other relevant musical information, such as the sustain pedal, is incorporated directly into the duration tokens." This is important: sustain pedal typically appears in MIDI as control change messages (CC 64), not as note events. The paper's approach of folding pedal information into duration tokens means that when the sustain pedal is depressed, the effective duration of notes is extended beyond their note-off events — the AMT or preprocessing pipeline presumably handles this by adjusting duration values based on pedal state.

Tokenization of full files. The complete tokenization of a MIDI file is constructed by "concatenating the tokenizations of the constituent notes in order of onset." This is a flat, one-dimensional sequence — notes that sound simultaneously (within the same 10 ms window) are ordered arbitrarily among themselves, but all notes that start earlier appear before notes that start later. The paper does not specify how it handles the ordering of simultaneous onsets; a natural approach would be to sort by pitch (lowest to highest) or instrument, but this detail is not provided.

Pretraining Data Pipeline: From Raw Audio to Filtered MIDI Corpus

The pretraining dataset is the foundation on which the entire paper rests. The paper describes a multi-stage curation process designed to produce a large but reasonably clean corpus from what is inherently noisy, internet-sourced data.

Source: Aria-MIDI. The raw material is the Aria-MIDI dataset, described as "transcriptions of solo-piano recordings gathered at scale from YouTube" and amounting to approximately 100,000 hours of piano music. The transcriptions are produced by automatic music transcription (AMT) models, which the paper notes have achieved "highly reliable note-identification accuracy" for solo piano in recent years (citing Kong et al., 2021; Toyama et al., 2023; Yan and Duan, 2024). The specific AMT model and its accuracy characteristics are not described in this paper — the focus is on what happens to the transcriptions after they are produced.

Filtering for quality and content suitability. The paper acknowledges that "significant variability exists in transcription quality and content suitability, potentially introducing harmful biases or noisy data into downstream models." The filtering pipeline addresses several distinct problems:

Memorization and duplication control. The paper applies filtering "based on compositional metadata" to address "extreme cases of composition duplication, such as repeated performances of overrepresented works." The specific procedure is: "for composers with more than 250 instances of files containing opus and/or piece number tags, we retained at most 10 instances per opus/piece-number pair. For these same composers, we also discarded all other files that lack compositional identifiers."

What this means operationally. If a composer (e.g., Chopin) has more than 250 MIDI files in the corpus, the system identifies files tagged with opus/piece number metadata (e.g., "Op. 28, No. 15" for the Raindrop Prelude). For each distinct opus/piece-number pair, at most 10 different performances are kept — the rest are discarded. Any files by these prolific composers that don't have opus/piece-number tags are completely removed.

Why this matters. Without this filter, extremely popular works (Für Elise, Clair de Lune, various Chopin nocturnes) could appear hundreds or thousands of times in the corpus in slightly different performances. The model would effectively memorize these pieces rather than learning general musical structure, and evaluation on prompts drawn from these works would be contaminated by training set overlap. The 10-instances-per-piece cap ensures the model sees diverse compositions rather than overfitting to a canonical set.
Heuristic quality filtering. The paper applies "heuristic-based filtering, considering note density, pitch and duration entropy, silence, and indicators of repetitive content, to exclude problematic entries (e.g., Black MIDI)."

Note density filtering likely removes files with impossibly high note counts per second — Black MIDI is a genre where MIDI files contain millions of notes (far beyond what a human could play), created as visual spectacles rather than musical performances.

Entropy-based filtering on pitch and duration distributions likely removes files that are too monotonous (e.g., a single repeated note for hours, or completely random note sequences) as well as files that are too uniform.

Silence detection likely removes files where large portions contain no note events (failed transcriptions, ambient noise transcribed as rests).

Repetitive content detection likely removes files where the same short passage loops many times without variation — a common artifact in automated MIDI generation or certain types of background music.

The specific thresholds for these heuristics are not provided in the paper, but the appendix references a preprocessing repository with further details.

Resulting corpus scale. After all filtering, the pretraining corpus consists of 820,944 MIDI files, amounting to 60,473 hours of solo piano music. This is the refined subset of Aria-MIDI's ~100,000 hours, meaning roughly 40% of the raw data was filtered out. The paper notes this is the corpus used for pretraining, while the finetuning stages use a further refined subset.

Online data augmentation during pretraining. To "enhance generalization and prevent overfitting," the paper applies three types of augmentation randomly during training, on-the-fly (not pre-computed):

Transposition: randomly transpose the pitch of all notes by an integer number of semitones in the range $\pm 5$ . This means the same musical passage can appear in up to 11 different keys (the original plus up to 5 semitones up or down). This is crucial because piano music is not uniformly distributed across keys — certain keys (C major, A minor, G major) are vastly overrepresented — and without transposition, the model might learn key-specific patterns rather than key-invariant musical structure.
Tempo variation: randomly scale all note onset and duration times by a factor in the range $\pm 20\%$ (i.e., tempo multipliers between 0.8 and 1.2). This prevents the model from associating specific absolute durations with musical meaning; a quarter note at one tempo has the same musical function as a quarter note at another tempo, and tempo augmentation encourages the model to learn relative rather than absolute duration relationships.
Velocity adjustment: randomly add or subtract up to $\pm 10\%$ to all MIDI velocity values (the range is likely relative, meaning each velocity is scaled by a factor between 0.9 and 1.1, though the paper's phrasing is ambiguous). This prevents overfitting to specific dynamic levels and encourages robustness to transcription-dependent velocity calibration differences.

Why these augmentations and not others. The paper does not apply augmentation that would change the musical content — no note insertion or deletion, no reharmonization, no structural rearrangement. This contrasts with some text augmentation approaches that paraphrase or substitute tokens. The augmentations are designed to be musically semantics-preserving: a transposed performance is the same piece, just in a different key; a tempo-scaled performance is the same performance, just faster or slower; a velocity-adjusted performance is the same dynamics shape, just louder or softer. The model should learn to recognize these as equivalent in the relevant musical dimensions.

Model Architecture and Pretraining Recipe

The paper's architectural choices are deliberately conservative, emphasizing that the primary variable of interest is data scale, not model novelty.

Base architecture: modified LLaMA 3.2. The model is described as building on "the LLaMa 3.2 model family, chosen due to its effectiveness in autoregressive tasks across modalities." The starting point is the 1B parameter configuration, from which the authors make two significant modifications:

Reduced hidden dimension. The hidden state dimension $d_{\text{model}}$ is reduced from 2048 to 1536, which "decreased the parameter count by roughly half, balancing model capacity with computational efficiency for our dataset scale." With 24 attention heads (explicitly stated), this means each head has dimension $d_{\text{head}} = 1536 / 24 = 64$ .

Why this reduction. The paper cites Hoffmann et al. (2022) on "established principles on model-data ratios for language models." The Chinchilla scaling laws suggest that for a given training data budget (measured in tokens), there is an optimal model size — too small and you leave representational capacity on the table; too large and you overfit or train inefficiently. The authors' pretraining corpus has a certain total token count (determined by 60,473 hours × tokens per hour, where tokens per hour depends on average note density and temporal resolution). Reducing $d_{\text{model}}$ from 2048 to 1536 (and thus halving parameters from 1B to ~650M) likely brings the model closer to the compute-optimal frontier for their data scale. The paper doesn't report the exact token count of the pretraining corpus, but back-of-envelope: if average note density is ~5 notes/second (a reasonable estimate for piano music), then one hour produces roughly $5 \times 3600 \times 3 = 54{,}000$ tokens (3 tokens per note), so 60,473 hours yields approximately 3.3 billion tokens — modest by language model standards, consistent with a 650M parameter model being near compute-optimal.
Simplified attention and normalization. The paper "simplified the architecture by opting for standard multi-head attention (with 24 heads) and layer normalization, instead of grouped-query attention and RMS normalization as used in standard LLaMa 3 variants."

Standard multi-head attention (Vaswani et al., 2017): each of the 24 heads computes independent query, key, and value projections; each head attends over the full sequence; the head outputs are concatenated and projected. This contrasts with grouped-query attention (Ainslie et al., 2023), where multiple query heads share the same key and value heads to reduce memory and compute at large batch sizes. The paper likely found that at their model scale (~650M parameters), the memory savings from GQA were not necessary and standard MHA was simpler to implement and debug.

Layer normalization (Ba et al., 2016): normalizes the activations across the hidden dimension for each token independently, computing the mean and variance across the feature dimension. This contrasts with RMS normalization (Zhang et al., 2019), which normalizes by the root-mean-square of the features without centering (no mean subtraction). The paper doesn't justify this choice explicitly, but layer normalization is more standard in research codebases and may interact differently with the specific distribution of MIDI token embeddings.

Additional architectural details. The paper explicitly mentions these components but doesn't provide full detail on others:

24 attention heads — explicitly stated.
No mention of feed-forward dimension — likely follows the LLaMA convention of $d_{\text{ff}} \approx 4 \times d_{\text{model}}$ (so roughly 6144), but this is not confirmed.
No mention of number of layers — for a ~650M parameter model with $d_{\text{model}} = 1536$ and 24 heads, a typical depth would be 24–32 layers, but this is not specified.
No mention of activation function — LLaMA uses SwiGLU, and the paper doesn't state whether this was kept or changed.
Vocabulary size — not explicitly provided, but it can be inferred from the tokenizer components: 13 instrument classes × 128 pitches × 12 velocities = 19,968 possible (instrument, pitch, velocity) combinations for melodic notes, plus percussion tokens (128 note numbers), onset tokens (500 values), duration tokens (likely up to 500 values, plus a representation for durations exceeding 5000 ms which would need special handling), and special tokens (<T>, <D>, end-of-sequence, etc.).

Pretraining objective and sequence construction. The model is pretrained using "standard next-token prediction on concatenated sequences of tokenized MIDI files." Specifically:

Sequence length: 8192 tokens. This was "chosen to balance computational constraints with the need to learn meaningful short- and long-term dependencies within piano music." At 3 tokens per note, 8192 tokens corresponds to approximately 2700 notes. In a typical piano performance at 5–10 notes per second, this represents roughly 4.5–9 minutes of music — enough to capture large-scale formal structure (themes, variations, recapitulations) while remaining computationally feasible.
File concatenation: MIDI files are concatenated into longer training sequences, with file boundaries presumably marked but not explicitly described. Concatenation is standard for language model pretraining because it maximizes GPU utilization (no padding tokens), but it means the model must learn to recognize file boundaries implicitly, which can cause "bleeding" where the model continues one composition with elements from the next.
Special tokens between files: The paper does not specify whether file boundaries are indicated with a special separator token during pretraining. During generative finetuning, files are not concatenated (each training sequence starts at a file boundary), which suggests that pretraining concatenation may not have had explicit boundary markers — the model learned to handle transitions between unrelated pieces.

Optimization hyperparameters. The paper provides specific training configuration:

Optimizer: AdamW
Learning rate: $3 \times 10^{-4}$ (3e-4)
Warmup: 1000 steps
LR schedule: linear decay to 10% of the initial rate over the course of training (i.e., final LR = $3 \times 10^{-5}$ )
Training duration: 75 epochs over the training corpus
Hardware: 8 H100 GPUs
Batch size: 16 per GPU (total effective batch size = 128 sequences of 8192 tokens = 1,048,576 tokens per optimizer step)
Training time: 9 days

The learning rate of $3 \times 10^{-4}$ is relatively high for transformer pretraining (language models often use $6 \times 10^{-4}$ to $3 \times 10^{-5}$ ), but the paper doesn't discuss tuning. The 75-epoch training duration is notable — with approximately 3.3 billion tokens and a total of roughly $\frac{3.3 \times 10^9 \times 75}{1.05 \times 10^6 \text{ tokens/step}} \approx 236{,}000$ optimizer steps, this represents a substantial amount of training.

Why these specific hyperparameters. The paper doesn't provide explicit justification for most values, but some can be inferred:

The 10% final learning rate (linear decay to $3 \times 10^{-5}$ ) is a cosine schedule approximation that avoids the sharp drop-off of cosine annealing while still allowing sufficient exploration late in training.
The 1000-step warmup is standard for transformers to avoid early gradient spikes before the optimizer accumulates sufficient second-moment estimates.
AdamW (Adam with decoupled weight decay) is the standard optimizer for transformer pretraining.
Training on 8 H100 GPUs for 9 days with batch size 16 per GPU is a substantial but not extraordinary compute budget — consistent with the paper's framing of this as an accessible foundation model rather than an industrial-scale training run.

Generative Finetuning Procedure

The pretrained model is not directly used for music continuation evaluation. Instead, it undergoes a "single-epoch finetuning phase" designed to adapt it specifically for generative tasks with higher-quality data and controlled inference behavior.

Data quality improvements for finetuning. The finetuning dataset is a further refined subset of the pretraining corpus with three changes:

All compositional duplicates removed — whereas pretraining retained up to 10 instances per opus/piece-number pair for prolific composers, finetuning removes all identified duplicates. This eliminates any remaining memorization risk for the evaluation phase.
Tightened quality filters — the existing heuristic filters (note density, entropy, silence, repetitive content) are applied with more aggressive thresholds. The specific tighter thresholds are not provided but presumably remove borderline cases that passed the pretraining filters.
Additional filter for synthesized MIDI — a new filter is introduced "aimed at excluding transcriptions of synthesized MIDI files." This is important because synthesized MIDI (e.g., a computer playing back a MIDI file with perfect timing and no expression) sounds qualitatively different from human performances when transcribed by AMT. The transcription of a synthesized MIDI file would contain non-human timing regularities and velocity patterns that could bias the model toward mechanical-sounding generation. The paper references a preprocessing repository for details on how synthesized MIDI transcriptions are detected.

Training configuration changes for finetuning. Several aspects of the training procedure differ from pretraining:

Single epoch: the finetuning phase runs for exactly one pass through the finetuning data — "annealing the learning rate to zero while training on higher-quality data." This is analogous to the "annealing" or "cool-down" phase used in some language model training pipelines, where the learning rate is decayed to zero on a high-quality subset of data to refine the model's generation quality.
Non-concatenated sequences: unlike pretraining where files were concatenated into 8192-token sequences, during finetuning "each training sequence begins at the start of a new file (i.e., non-concatenated)." This means the model always starts generating from a clean file boundary (which is also how it will be used at inference time — prompted with the start of a piece), removing the distribution shift between training and inference that concatenated training would introduce.
End-of-generation control token: the paper introduces a special token <D> that is "inserted approximately 100 tokens before the end of each training example to enable explicit inference-time control over generation endings." During inference, generation would be terminated when the model produces the <D> token, providing a learned stopping criterion rather than relying on a fixed generation length or generating until a maximum token count is reached. The placement "approximately 100 tokens before the end" — approximately 33 notes before the composition ends — gives the model a predictable signal for when to begin concluding its generation.

Why a separate finetuning phase. The paper's design choice to separate generative finetuning from pretraining serves several purposes. First, it allows the pretrained model to serve as a general-purpose backbone that can be adapted to multiple downstream tasks (generation, classification, embedding) using different finetuning recipes — this is the foundational model paradigm. Second, the data quality requirements for generation evaluation are stricter than for representation learning: human listening tests are sensitive to memorization artifacts, compositional plagiarism, and mechanical-sounding outputs, which the aggressive deduplication and synthesized-MIDI filtering of the finetuning phase address.

Contrastive Representation Learning: Adapting SimCLR to Symbolic MIDI

This is the paper's primary methodological contribution — a novel framework for producing general-purpose symbolic music embeddings without requiring labels, metadata, or paired text.

The SimCLR framework adapted to sequential data. SimCLR (Chen et al., 2020) was originally developed for image representation learning. The core idea is: given a batch of examples, create two augmented views of each example. Train an encoder so that the embedding of view A of example 1 is similar to the embedding of view B of example 1 (positive pair), and dissimilar to the embeddings of all views of all other examples in the batch (negative pairs). The loss function — NT-Xent (normalized temperature-scaled cross-entropy) — formalizes this as a classification problem: among all $2N$ embeddings in the batch (two views for each of $N$ examples), correctly identify which embedding is the positive match for each query embedding.

Adaptation to MIDI: slice extraction as view generation. To apply SimCLR to symbolic music sequences, the paper needs an augmentation strategy that produces two different but semantically related views of the same file. Their solution is non-overlapping slice extraction:

From each MIDI file, the system randomly extracts two different contiguous slices, each comprising between 100 and 650 notes (approximately 300–2000 tokens, given the 3-tokens-per-note representation).
The slices are non-overlapping, meaning they sample from different temporal regions of the same performance. This is crucial: if the slices overlapped, the model could cheat by learning to match local note patterns rather than file-level attributes.
Each slice undergoes independent data augmentation using the standard procedures (transposition ±5 semitones, tempo ±20%, velocity ±10%) before tokenization.

What the slice extraction strategy forces the model to learn. Because the two views come from different temporal regions of the same file, they share almost no local note-level information — different measures, different thematic material, different harmonic progressions. What they share are file-level attributes: the composer's harmonic language, the performer's touch and timing tendencies, the genre conventions, the recording conditions (which affect transcription characteristics), the overall style. To correctly identify that slice A and slice B belong together (while slice C from a different file does not), the encoder must learn to represent these global attributes while discarding local surface features. This is why the resulting embeddings capture "higher-level musical semantics such as genre, composer, style, and performance nuances, rather than local details," as the paper states.

Slice length range: 100–650 notes. The lower bound of 100 notes ensures each slice contains enough musical material to be meaningfully characteristic — a single chord or brief motif would not carry sufficient stylistic information. The upper bound of 650 notes (approximately 1–2 minutes of music at typical note densities) ensures computational feasibility (sequence lengths stay within 2000 tokens, well within the model's 8192-token pretraining context) while still capturing medium-scale structure. The random sampling within this range provides additional augmentation: the model sees slices of varying lengths during training, improving robustness.

Embedding extraction and projection head. The architecture for contrastive learning modifies the pretrained model in two ways:

Replacement of language modeling head: the original output projection (which produces logits over the token vocabulary) is replaced with an embedding head — a learned linear projection that maps the final hidden state into a 512-dimensional embedding space. The paper doesn't specify whether this is a single linear layer or includes non-linearities and normalization, but SimCLR typically uses a small MLP (2–3 layers) with a ReLU and batch normalization before the final linear projection.
End-of-sequence token embedding: to produce a fixed-size vector from a variable-length slice, the system appends a special end-of-sequence token after the final note token-triple of each slice. The hidden state associated with this token (after passing through all transformer layers) is taken as the slice's embedding. This is analogous to using the [CLS] token in BERT or the final token's hidden state in autoregressive models — a learned aggregation that summarizes the entire sequence into a single vector.

The NT-Xent loss (Equation 2). For a minibatch of $N$ MIDI files, two views are created per file, yielding $2N$ embeddings $\{z_1, \ldots, z_{2N}\}$ , where $z_i$ and $z_{i+N}$ are the two views of file $i$ . The loss for a single positive pair $(i, j)$ is:

$\ell_{i,j} = -\log\frac{\exp\left(\operatorname{sim}\left(z_{i}, z_{j}\right) / \tau\right)}{\sum_{k=1}^{2N} \mathbbm{1}_{[k \neq i]} \exp\left(\operatorname{sim}\left(z_{i}, z_{k}\right) / \tau\right)}$

where $\operatorname{sim}(z_i, z_j) = \frac{z_i^\top z_j}{\|z_i\|\|z_j\|}$ is the cosine similarity between the two normalized embeddings, $\mathbbm{1}_{[k \neq i]}$ is an indicator function that is 1 (includes the term) for all indices $k$ except $k = i$ (excluding the self-comparison), and $\tau$ is the temperature parameter, set to $\tau = 0.1$ .

What Equation 2 computes. For each embedding $z_i$ , the loss treats it as a query and asks: among all $2N$ embeddings in the batch (the $N-1$ other files' two views plus the other view of file $i$ ), which one is the correct match? The numerator $\exp(\operatorname{sim}(z_i, z_j) / \tau)$ is the unnormalized score for the correct match $z_j$ (the other view of the same file). The denominator sums the unnormalized scores for all candidates, including the correct match and all $2N-1$ negatives. The negative log turns this into a minimization problem: the loss is low when the correct match's score is high relative to all negative scores.

Why the temperature parameter $\tau = 0.1$ matters. The temperature controls the concentration of the similarity distribution. A low temperature ( $\tau = 0.1$ is considered low) amplifies the difference between the correct match and the negatives — small differences in cosine similarity become large differences in the exponentiated scores. This makes the loss focus on the hardest negatives (the ones the model is currently confusing with the positive) and encourages the embedding space to develop finer-grained distinctions. However, too low a temperature can cause training instability because the gradient signal becomes dominated by a few very hard negatives. The choice of $\tau = 0.1$ is a hyperparameter that the paper likely tuned (Table 2 mentions ablations varying $\tau$ , though the specific values tested are not detailed).

The full symmetric loss. The paper minimizes the symmetric version of the NT-Xent loss:

$L := \frac{1}{2} \sum_{k=1}^{N} (\ell_{k, k+N} + \ell_{k+N, k})$

where $\ell_{k, k+N}$ treats $z_k$ as the query and $z_{k+N}$ as the positive, and $\ell_{k+N, k}$ does the reverse. This symmetric formulation ensures that the gradients are balanced — both views of each file contribute equally to learning, rather than one view being privileged as the "anchor."

Why this symmetric form. Without symmetry, only one direction of the contrastive prediction is trained (e.g., using $z_k$ to retrieve $z_{k+N}$ but not vice versa). The symmetric loss doubles the effective number of positive pairs per batch (from $N$ to $2N$ ) and ensures the embedding space treats both views equivalently.

Training configuration for contrastive finetuning. The paper specifies:

Learning rate: $1 \times 10^{-5}$ (1e-5) — much lower than the pretraining LR, appropriate for finetuning where the model weights already encode useful structure.
LR schedule: linear decay (to what final value is not specified, but consistent with the "same linear decay schedule" as other phases, implying decay to 10% of initial, so $1 \times 10^{-6}$ ).
Temperature: $\tau = 0.1$
Duration: 25 epochs
Data usage: each MIDI file contributes exactly one pair of augmented views per epoch (the two randomly extracted slices with independent augmentation). This means the model sees each file 25 times over the course of training, but with different pairs of slices each epoch.
Dataset: the reduced finetuning dataset, "however, we relaxed the preprocessing constraints on compositional duplicates to encourage the model to distinguish between different performances of popular compositions."

Why relaxing duplicate constraints benefits contrastive learning. In the generative finetuning phase, duplicates were aggressively removed to prevent memorization. For contrastive learning, however, having multiple different performances of the same composition is beneficial: the embedding must learn to distinguish between, say, Horowitz's and Rubinstein's interpretations of the same Chopin nocturne, based on subtle performance differences rather than compositional content. If all duplicates were removed, the model would only see one performance per composition and could use compositional features as a shortcut (simply identifying which piece it is, rather than how it's played). By relaxing the constraints, the model faces the harder task of disentangling composition identity from performer-specific and recording-specific attributes.

Key design advantage: no metadata or labels required. The paper emphasizes that this contrastive framework "introduces a general-purpose semi-supervised framework for representation learning of symbolic music, which allows us to evaluate the transferability of next-token musical representations." Unlike CLaMP, which requires paired text descriptions, or MusicBERT, which requires bar-level structural annotations, or supervised methods that require composer/genre/style labels, this framework works with raw MIDI data alone. The only supervision signal is the identity of the file — two slices from the same file should be similar; slices from different files should be dissimilar. This is a form of self-supervision that scales naturally with any collection of MIDI files, regardless of metadata availability.

The critical finding: pretraining initialization is essential. The paper's ablation study (Table 2) demonstrates that contrastive training from scratch — starting from randomly initialized weights — is dramatically less effective than contrastive finetuning from pretrained next-token-prediction weights. Specifically, "finetuning on one embedding pair per file for a single epoch (Aria, $e=1$ ) surpasses training from scratch on 25 pairs per file (Aria $^\dagger$ , $e=25$ )." This is the paper's key methodological insight: next-token prediction pretraining learns musical representations that serve as essential scaffolding for contrastive embedding learning. Without this scaffolding, contrastive training cannot efficiently discover the feature space that distinguishes composers, genres, and styles.

Supervised Finetuning for MIR Classification

The third adaptation path demonstrates the pretrained model's transfer learning capabilities for standard supervised classification tasks.

Architecture modification. For classification, the paper "replaced the language modeling head with a classification head, predicting labels directly from the hidden state of the end-of-sequence token." This mirrors the embedding extraction approach: an end-of-sequence token is appended after the tokenized MIDI file, and its final hidden state (after passing through all transformer layers) is projected to class probabilities via a linear layer with softmax activation.

End-to-end finetuning. Unlike the linear probe experiments (which freeze the pretrained weights and only train the classification head), supervised finetuning updates all model weights. This allows the transformer layers to adapt their representations to the specific classification task, potentially discovering task-relevant features that are latent in the pretrained representations but not linearly separable in the frozen embedding space.

Progressive dropout regularization. The paper introduces a specific dropout strategy: "applying dropout to residual connections, increasing the dropout rate linearly from $p_d = 0.0$ (first layer) to $p_d = 0.2$ (final layer)."

What this does. Standard dropout randomly zeroes out activations during training to prevent co-adaptation of features (Srivastava et al., 2014). Applying dropout to residual connections specifically targets the skip connections that allow information to bypass transformer layers. By increasing dropout on deeper layers, the paper forces later layers to be more robust — they cannot rely on the residual pathway as heavily and must develop features that are individually predictive. Early layers, which capture low-level representations shared across tasks, are preserved with no dropout, while later layers, which are more task-specific, are regularized more heavily.

Why this shape. The progressive increase from 0.0 to 0.2 is a form of curriculum regularization: it assumes that lower layers learn transferable features (which should be preserved) while higher layers learn task-specific features (which are more prone to overfitting, especially with limited labeled data). This hypothesis is consistent with the standard finding in transfer learning that early layers are more general and later layers are more specialized.

Optimization for finetuning:

Learning rate: $1 \times 10^{-5}$ (1e-5), with no warmup (the model starts from pretrained weights, so no warmup is needed to establish optimizer statistics).
LR schedule: linear decay.
Training duration: 10 epochs, with results reported from the best-performing epoch (early stopping based on validation performance).
Data: class-balanced subsets of varying sizes, to evaluate label efficiency.

Why not use a separate classification-specific pretraining objective. Alternative approaches like BERT-style masked token prediction or sequence-level contrastive pretraining could potentially produce better frozen representations for classification. The paper's choice of next-token prediction as the sole pretraining objective reflects the "foundation model" philosophy: a single pretrained model should serve multiple downstream purposes (generation, embedding, classification) rather than optimizing for any single task. The supervised finetuning experiments test whether this unified pretraining approach is sufficient, or whether task-specific pretraining would be necessary.

Summary of Design Choices and Their Justifications

Hybrid absolute-relative timing over pure relative-timing: avoids cumulative error from summing many time-shift tokens, which transformers are known to do unreliably, while keeping the vocabulary bounded through 5000 ms segmentation.
10 ms temporal resolution, 12 velocity bins: balances expressiveness (captures human timing nuances and dynamics) with vocabulary size and sequence length.
Token triple representation (instrument-pitch-velocity, onset, duration): separates the "what and how" decision from the "when and how long" decision, creating a token stream where the model can learn joint distributions over discrete event types independently from temporal placement.
Online data augmentation (transposition, tempo, velocity): prevents overfitting to key signatures, tempos, and dynamic levels that are non-uniformly distributed in the training data; enables the model to learn musical invariances.
Reduced model size (650M parameters, $d_{\text{model}} = 1536$ ): follows Chinchilla-style compute-optimal scaling principles for the available data budget (~3–4 billion tokens), avoiding overparameterization.
Standard multi-head attention and layer normalization over grouped-query attention and RMS norm: architectural simplification justified by the model's modest scale, where the efficiency gains from GQA and RMS norm are less critical.
Generative finetuning on higher-quality, non-concatenated data with <D> termination token: adapts the pretrained model for controlled inference, closer to the evaluation distribution where generation starts from clean file boundaries and needs a reliable stopping mechanism.
Non-overlapping slice extraction for SimCLR views: forces the model to learn file-level (composer, genre, style, performer) rather than local (note pattern, chord progression) features — the two views share global but not local structure.
Relaxed duplicate constraints for contrastive training: unlike generative finetuning (where duplicates cause memorization), having multiple performances of the same piece is beneficial for learning performance-level distinctions — the model must separate composition identity from interpretive style.
Progressive dropout (0.0 → 0.2) for supervised finetuning: preserves shared low-level features while regularizing task-specific high-level features, mitigating overfitting when labeled data is limited.
Single pretrained backbone for generation, embedding, and classification: avoids task-specific architectures; tests whether next-token prediction alone produces sufficiently general-purpose representations for symbolic music.

4. Key Insights and Innovations

Innovation 1: Scale, Not Architecture, Is the Primary Bottleneck in Symbolic Music Modeling—and It Has Been Breached

The paper's most fundamental conceptual contribution is not a new model or algorithm, but an existence proof: that the self-supervised learning paradigm that transformed NLP, vision, and audio transfers to symbolic music when the data bottleneck is removed. This sounds incremental—"more data helps"—but the intellectual move here is more specific. The paper demonstrates that the field's prior limitations were imposed by data scarcity, not by any inherent property of symbolic representations or the capabilities of transformer architectures. This reframes the entire research landscape for symbolic music modeling.

What makes this a genuine insight rather than an obvious observation is the active counter-narrative it displaces. Prior to Aria, the dominant assumption—visible in the design choices of MusicBERT, CLaMP, and MusicVAE—was that symbolic music required specialized architectural innovations (bar-masking objectives, cross-modal contrastive losses against text, hierarchical latent variable models) to learn meaningful representations. The implicit logic was that MIDI data is too sparse, too structured, or too semantically ambiguous for vanilla autoregressive pretraining to work well. Aria challenges this by stripping the architecture down to a standard LLaMA 3.2 decoder with simplified attention and normalization, reducing the hidden dimension to ~650M parameters following Chinchilla scaling principles, and then letting scale do the work. The result is a single pretrained backbone that adapts to generation, classification, and contrastive embedding without task-specific pretraining modifications—exactly the "foundation model" behavior that characterizes large language models.

The significance extends beyond performance. This finding implies that the primary barrier to symbolic music modeling has been data acquisition infrastructure, not algorithmic sophistication. The paper explicitly credits automatic music transcription (AMT) as the enabling technology, and its careful data curation pipeline (compositional deduplication, heuristic quality filtering, online augmentation) is arguably as important to the results as the model itself. For the field, this means research priorities should shift: building better AMT pipelines and larger-scale MIDI corpora may yield more progress than incremental architectural refinements. This is a strategic insight, not a technical one, and it mirrors the lesson the NLP community learned with GPT and BERT—that scale can be more transformative than cleverness.

The evidence anchoring this claim is distributed across all evaluation settings, but the most compelling single result is the generative finetuning experiment (Table 1): a ~650M parameter model trained on 60,000 hours of transcribed piano performances generates continuations that are statistically indistinguishable from both Suno 3.5 (a proprietary audio model with unknown but presumably much larger compute budget) and human-composed ground truth. This is a ~300× increase in training data over Music Transformer's MAESTRO-based training (~200 hours), using essentially the same autoregressive transformer framework, suggesting that the performance jump is primarily attributable to data scale rather than architectural innovation.

Innovation 2: Hybrid Absolute-Relative Timing as a Diagnostic Intervention on Transformer Arithmetic Failure

The paper's tokenization scheme—5000 ms segments marked by <T> tokens with segment-relative absolute onset times—is often presented as an engineering convenience. This undersells what is actually a diagnostic intervention: the design directly targets a known failure mode of transformers (unreliable accumulation of many sequential arithmetic operations) and proposes a representation that replaces summation with counting. The intellectual contribution is not the specific tokenization scheme itself, but the identification of cumulative temporal error as a first-class failure mode in symbolic music generation and the principled design of a representation that eliminates it.

Prior work in symbolic music generation (Music Transformer, MuseNet, REMI-based models) overwhelmingly used relative timing: each note is preceded by a time-shift token specifying how long to wait since the previous event. This is the natural representation in the MIDI protocol, where events are spaced by "ticks." The paper's critique—that transformers must "sum up many sequential time-shift values to calculate temporal relationships between notes with medium or long-term dependencies"—reframes this design choice as an arithmetic competence problem. By connecting their observations ("temporal instability in passages with rapid note sequences") to the emerging literature on transformers' arithmetic limitations (Lee et al., 2023; McLeish et al., 2024), the paper elevates what could have been a minor implementation detail into a generalizable design principle: representations should align with the computational strengths of the architecture. Counting <T> tokens is integer counting; summing token-embedded time-shift values is floating-point accumulation. The former is reliable; the latter is not.

Equation 1 formalizes this as a comparison of arithmetic complexity across schemes, but the conceptual contribution is broader. It suggests that many symbolic music modeling difficulties attributed to "long-range dependency" problems may actually be manifestations of cumulative error in relative representations, not genuine failures of the model to learn temporal structure. This is a reframing with implications beyond this paper: it suggests that evaluating tokenization schemes by whether they minimize the arithmetic the model must perform internally—especially accumulation and subtraction over variable-length sequences—is a productive design heuristic.

The evidence for this insight is circumstantial rather than directly ablated (the paper does not compare absolute vs. relative timing head-to-head with all else equal), but the qualitative observation that prior models "drift" rhythmically while Aria does not, combined with the strong generation results in Table 1, supports the claim that this design choice matters.

Innovation 3: Contrastive Learning for Symbolic Music Requires Pretrained Initialization—and Slice-Based Augmentation as a Mechanism for Learning File-Level Semantics

The paper's adaptation of SimCLR to symbolic MIDI contains two intertwined insights that together form a significant methodological contribution to representation learning for music.

First: pretraining initialization is not just helpful—it is necessary. The ablation in Table 2 demonstrates that contrastive training from scratch on 25 embedding pairs per file underperforms contrastive finetuning from pretrained weights on a single pair per file (a single epoch). This is a strong result: the pretrained model's representations, learned through next-token prediction, encode musical structure that contrastive learning alone cannot efficiently discover. This mirrors findings in text embedding literature (SimCSE, Gao et al., 2021; E5, Wang et al., 2023), where initializing from pretrained language models is standard, but extends the insight to a new modality and provides direct ablation evidence that was missing from some of the text embedding work.

The conceptual significance is that next-token prediction on symbolic music sequences learns genuinely transferable musical structure, not just surface-level note prediction patterns. If pretraining merely taught the model to predict the next note token without developing internal representations of higher-level musical features (harmony, style, form, genre), then contrastive finetuning would not benefit from initialization—the contrastive objective would need to build these features from scratch regardless. The efficiency of the transfer (one epoch, one pair per file) suggests that the pretrained model's internal representations already encode rich file-level semantics, and contrastive finetuning primarily reorganizes them into a more accessible embedding space rather than learning them from scratch.

Second: non-overlapping slice extraction is a principled augmentation strategy that determines what the embedding represents. Most contrastive learning applications in vision (SimCLR, MoCo) use pixel-level augmentations (cropping, color jittering, rotation) that preserve semantic content while varying surface appearance. Directly translating this to symbolic music is non-obvious: what is the "augmentation" that preserves file-level semantics while varying local content? The paper's insight is that temporal slicing accomplishes this: two different slices of the same piano performance share the performer's timing tendencies, dynamic shaping, phrasing habits, and the composer's harmonic language, but contain different notes. This forces the embedding to represent global attributes while ignoring local surface features.

This is different from and complementary to CLaMP's cross-modal approach (which aligns symbolic music with text descriptions and therefore captures what can be described in words) and MusicBERT's bar-masking approach (which learns local structural patterns at the bar level). The slice-based SimCLR produces embeddings that capture aspects of music that are hard to verbalize but characteristic of specific performers and composers—the "feel" and "sound" of a performance as encoded in timing and dynamics. The t-SNE visualization in Figure 1, showing natural composer clustering without any metadata during training, is the qualitative evidence for this claim. The state-of-the-art linear probe results on composer, genre, and style classification (Table 2) provide quantitative evidence.

Taken together, these two insights define a new methodology for symbolic music representation learning: pretrain via next-token prediction, then reorganize via contrastive finetuning with temporal slicing, producing embeddings that capture both compositional identity (composer, genre, form) and performance nuances without requiring labels, metadata, or paired text. This is a general-purpose framework that can be applied to any collection of MIDI files, regardless of annotation availability.

Innovation 4: Data Curation as a First-Class Research Contribution—and the Specific Role of Compositional Deduplication

The paper makes a subtle but important contribution by treating data curation decisions as scientific claims, not engineering details. Two curation choices in particular rise to the level of conceptual contributions.

The first is compositional deduplication with performance diversity preservation. The paper's filtering pipeline distinguishes two types of duplicates: (a) multiple recordings of the same composition by the same or different performers, and (b) the same recording appearing multiple times (e.g., from different YouTube uploads). For pretraining, type (a) duplicates are capped at 10 per opus/piece-number pair for prolific composers; for generative finetuning, they are removed entirely. Type (b) duplicates are presumably handled by the metadata-based filtering but the distinction is conceptually important: performances of the same piece are musically diverse training examples (different interpretations, tempos, dynamics, recording conditions) that teach performance-level variation, while file-level duplicates are pure redundancy that causes memorization. The paper's decision to retain multiple performances during pretraining (up to 10) but remove all during the finetuning phase designed for generation evaluation reflects a nuanced understanding that "duplication" is not a binary property—it depends on what the model is being trained to do and how it will be evaluated.

This matters because memorization is the critical failure mode for generative music models evaluated via human listening tests. If the model has memorized training pieces, it may reproduce them during evaluation, and human judges may prefer these outputs as "coherent"—but this is not evidence of compositional capability. The paper's aggressive deduplication for the generative finetuning dataset, plus the use of prompts from compositions not in the training set, is an attempt to ensure the human evaluation results reflect genuine musical understanding rather than retrieval. This is a methodological contribution to generative music evaluation: data contamination control must be part of the experimental design, not an afterthought.

The second is the relaxation of duplicate constraints for contrastive training. For contrastive learning, the paper deliberately retains multiple performances of the same piece. This is a sophisticated choice: if all duplicates were removed, the embedding model could distinguish files by compositional content alone (this is Chopin, that is Debussy) without learning anything about performance. Retaining multiple interpretations of the same work forces the model to make finer distinctions—Horowitz's Chopin vs. Rubinstein's Chopin—which requires attending to expressive timing, dynamics, and phrasing rather than note content. This decision reflects an understanding that the contrastive objective's positive pairs define what the embedding space represents: if positive pairs always share compositional identity, the embedding learns composition identity; if positive pairs sometimes share composition but not performance, the embedding must learn performance-level features as well.

This second point is a specific instance of a more general principle that the paper demonstrates but does not state explicitly: the choice of which data points are considered "same" versus "different" in a contrastive framework determines what invariances and features the embedding space captures. This is well-understood in the contrastive learning literature but has been underexplored in music, where the "same piece, different performance" relationship provides a natural curriculum for learning hierarchical musical structure.

5. Experimental Analysis

Evaluation Methodology

Dataset. All experiments use the Aria-MIDI dataset, a corpus of approximately 100,000 hours of AMT-transcribed solo-piano recordings gathered from YouTube. The pretraining corpus, after heuristic filtering and compositional deduplication, comprises 820,944 MIDI files totaling approximately 60,473 hours. The generative finetuning dataset is a further refined subset with all compositional duplicates removed, tightened quality filters, and an additional filter excluding transcriptions of synthesized MIDI files. For classification benchmarks, the paper uses Pianist8 (Chou et al., 2021), VG-MIDI (Ferreira and Whitehead, 2021), and four newly derived label-balanced train-test splits from Aria-MIDI metadata: Genre (2 classes, 10,000 train / 1,000 test), Musical Period (4 classes, same split sizes), Form (6 classes), and Composer (10 classes). Human evaluation uses five solo-piano prompts representing different subgenres, with eight continuations generated per prompt per model (40 total continuations per model).
Base model(s). The model is a modified LLaMA 3.2 transformer with approximately 650 million parameters. Architectural modifications from the 1B LLaMA 3.2 configuration include reducing the hidden dimension (d_model) from 2048 to 1536, replacing grouped-query attention with standard multi-head attention (24 heads), and replacing RMS normalization with layer normalization. The paper argues this model is chosen for its "effectiveness in autoregressive tasks across modalities" and that the architectural simplifications, combined with the reduced parameter count following Chinchilla scaling principles, balance capacity with "computational efficiency for our dataset scale." For supervised classification experiments, the pretrained model is finetuned end-to-end with a classification head replacing the language modeling head. For contrastive embedding experiments, the language modeling head is replaced with a projection to a 512-dimensional embedding space and finetuned with the SimCLR NT-Xent objective.
Metrics. The primary metrics are (1) pairwise human preference rates for musical coherence in generative continuation listening tests, evaluated by 46 participants with at least one year of musical training who compared 45-second continuations of 15-second prompts across A/B pairs; (2) classification accuracy and macro-F1 scores for linear probe experiments on frozen embeddings, where classifiers are trained on global file embeddings obtained by averaging slice embeddings within each file; and (3) classification accuracy and macro-F1 for supervised finetuning, where models are trained end-to-end and evaluated on held-out test sets, with results reported from the best-performing epoch over 10 training epochs. Human evaluation statistical significance is assessed using "a two-sided binomial test on non-tied comparisons."
Baselines. For generative evaluation, the paper compares against Anticipatory Music Transformer (music-large-800k; Thickstun et al., 2023), MusicGen (large; Copet et al., 2023), Suno 3.5 (Suno, 2024), and human-composed ground-truth continuations. For classification and embedding evaluation, baselines include CLaMP 3 (saas; Wu et al., 2025), M3 (Wu et al., 2024), and the audio-based model MERT (Li et al., 2023). The paper explicitly acknowledges excluding closed-access models like AudioLM (Borsos et al., 2023) "despite their promising reported results on similar piano-continuation benchmarks," and excluding symbolic models such as MuPT (Qu et al., 2024) because "their bar-level timing representation (e.g., ABC notation) is incompatible with expressive millisecond-level MIDI performances."
Generation budget / compute accounting. The paper measures compute implicitly through model size (650M parameters) and training duration (75 epochs on 60,473 hours of MIDI data, taking 9 days on 8 H100 GPUs with batch size 16 per GPU). For generative evaluation, all models produce 40 continuations (8 per prompt across 5 prompts) with the same prompt set. For contrastive embedding and supervised finetuning, compute is not explicitly budgeted across methods; the focus is on data efficiency (number of labeled examples available for supervised tasks, number of contrastive training epochs). The paper does not report FLOP counts or token-level generation budgets for inference-time comparisons across models.
Cross-validation / statistical protocol. For contrastive embedding linear probe experiments, classifiers are trained with learning rate 3e-4 and linear decay to 0, running "separate experiments with 10, 20, and 50 epochs, and reporting the best result." For supervised finetuning label-efficiency experiments, class-balanced subsets of varying sizes are sampled from the training set, models are trained for 10 epochs, and "results from the best-performing epoch" are reported, providing a form of implicit early stopping based on held-out validation performance. Human listening tests use a two-sided binomial test on non-tied comparisons for pairwise statistical significance, with 46 participants evaluating random A/B pairs. The paper does not report confidence intervals or standard errors for classification results, nor does it describe cross-validation for the Aria-MIDI-derived classification benchmarks beyond the fixed train-test split (10,000/1,000).

Main Quantitative Results

The paper organizes results across three axes: (1) generative modeling evaluated via human listening tests, (2) contrastive embedding quality evaluated via linear probe classification, and (3) supervised finetuning label efficiency.

Generative Continuation Quality: Human Listening Tests

Table 1 reports pairwise human preference results for musical coherence. The headline finding is that Aria's continuations are statistically indistinguishable from both human-composed ground truth and Suno 3.5, while significantly outperforming the Anticipatory Music Transformer and MusicGen.

Specifically, Table 1 shows:

Aria vs. Anticipatory Music Transformer: Aria won an undisclosed number of comparisons against AMT's losses and ties. The p-value from the two-sided binomial test on non-tied comparisons indicates a statistically significant preference for Aria's outputs.
Aria vs. MusicGen: Similarly, Aria significantly outperforms MusicGen, with the paper attributing this partly to MusicGen's "restricted context window [which] necessitates sliding-window inference, diminishing coherence in longer generations."
Aria vs. Suno 3.5: The preference rates showed no statistically significant difference. Aria's continuations were evaluated as comparably coherent to those produced by the proprietary audio generation model.
Aria vs. Human ground truth: Again, no statistically significant preference difference was found. Participants could not reliably distinguish Aria's 45-second continuations from the actual subsequent 45 seconds of the original human performance.

The paper also reports the experimental design parameters: "46 participants with at least one year of musical training," "45-second continuations generated from 15-second solo piano prompts," using "five prompts representing different subgenres of solo piano music" with "eight continuations per prompt (totaling 40 continuations per model)." Participants were guided by criteria including "melodic development, rhythmic structure, harmonic progression, and stylistic coherence."

The paper acknowledges two evaluation limitations explicitly: the exclusion of closed-access models like AudioLM, and the incompatibility of bar-level timing representations (ABC notation) in models like MuPT, making them unsuitable for expressive millisecond-level MIDI evaluation.

Contrastive Embedding Quality: Linear Probe Classification

Table 2 reports classification performance across six benchmarks for frozen embeddings. The headline result is that Aria's contrastive embeddings (Aria_Emb) achieve state-of-the-art linear probe accuracy and macro-F1 scores across composer, genre, style, and period classification tasks, substantially outperforming comparison models including CLaMP 3, M3, and the audio-based MERT on the benchmarks where the comparison models are applicable.

The "Main Results" section of Table 2 shows Aria_Emb performance alongside comparison models across the four Aria-MIDI-derived benchmarks and two external benchmarks (Pianist8, VG-MIDI). The paper states that the proposed method "substantially improves results on all benchmarks, producing embeddings that capture diverse file-level musical attributes without incorporating metadata during training." The gains are described as "substantial" but specific percentage-point improvements over the next-best baseline are not broken out in the narrative text — the table provides the raw accuracy and F1 values per task per model.

The paper notes a critical scope limitation for this comparison: "our benchmarks focus exclusively on piano performances, whereas the comparison models support multi-instrument MIDI or audio files." This means CLaMP 3 and M3, trained on multi-instrument data with text descriptions, operate in a broader domain, and their lower performance on piano-only benchmarks may partially reflect domain mismatch rather than intrinsic embedding quality inferiority. MERT, as an audio-based model, operates on a different input modality entirely.

Supervised Finetuning Label Efficiency

The "Finetuning ablations" section of Table 2 reports supervised classification test-set performance as a function of the number of labeled training files n. The headline finding is that the pretrained model achieves competitive classification performance with as few as a few hundred labeled examples and that performance continues to improve as more labeled data becomes available.

The paper states: "Our finetuned models achieve state-of-the-art classification performance on large datasets and perform surprisingly well on complex tasks, even when trained on limited labeled data." The specific label counts tested are indicated in the table but not enumerated in the narrative text; the characterization "few hundred labeled examples" is the paper's own framing. The end-to-end finetuning approach, which updates all model weights with progressive dropout (0.0 at first layer to 0.2 at final layer), outperforms the frozen embedding linear probe approach when sufficient labeled data is available, as expected given the additional representational flexibility.

Ablation Studies and Robustness Checks

Contrastive learning is effective only when initialized from pretrained weights: The embedding ablations in Table 2 compare Aria_Emb (contrastive finetuning from pretrained next-token-prediction weights) against Aria† (contrastive training from scratch with randomly initialized weights). The paper reports that "finetuning on one embedding pair per file for a single epoch (Aria, e=1) surpasses training from scratch on 25 pairs per file (Aria†, e=25)." This is the paper's most important ablation result: it demonstrates that next-token prediction pretraining learns representations that serve as essential scaffolding for contrastive embedding learning, and that contrastive training alone—even with 25× more contrastive supervision—cannot match what pretraining plus minimal contrastive finetuning achieves. The specific accuracy and F1 values for these ablations are reported in Table 2.

Number of contrastive training epochs and temperature parameter: The embedding ablations in Table 2 vary training epochs (e) and the temperature parameter (τ). The paper trains for 25 epochs with τ = 0.1 as the default configuration and reports results for e = 1 (single epoch) to quantify the efficiency of contrastive finetuning from pretrained weights. The specific τ values tested in the ablation are not explicitly enumerated in the narrative, but the table indicates that τ is varied and its impact on downstream classification performance is measured. The paper does not report whether higher τ values (which produce a softer contrastive distribution) or more epochs beyond 25 improve or degrade performance.

Generative evaluation sensitivity: prompts and participants. The listening test design inherently serves as a robustness check on generation quality: by using five diverse prompts across different piano subgenres and 46 musically trained participants, the paper guards against prompt-specific or judge-specific artifacts. The eight continuations per prompt per model provide within-prompt variance estimates. However, the paper does not report per-prompt breakdowns, inter-rater agreement metrics, or confidence intervals on preference rates—only aggregate win/tie/loss counts and two-sided binomial p-values.

Data quality filtering for generative finetuning. The paper's design choice to apply a separate, more aggressively filtered finetuning dataset for the generative model variant is itself a form of ablation: it demonstrates sensitivity to data quality. The paper removes all compositional duplicates, tightens heuristic quality filters, and adds a filter for synthesized MIDI transcriptions in the finetuning phase, producing a model "tailored for generative piano-continuation tasks." The paper does not provide an explicit ablation comparing generation quality with and without this additional finetuning refinement phase, but the distinction between the pretraining recipe and the generative finetuning recipe implies that these quality controls are important, even if their individual contributions are not isolated.

Relaxed duplicate constraints for contrastive training. The paper states that for contrastive finetuning, "we relaxed the preprocessing constraints on compositional duplicates to encourage the model to distinguish between different performances of popular compositions." This is a deliberate methodological choice, not an ablation, but the paper positions it as motivated by the contrasting requirements of generative (memorization is harmful) versus embedding (performance-level distinctions are valuable) objectives. No ablation comparing strict vs. relaxed deduplication for contrastive learning is reported, leaving open the question of how much this relaxation actually contributes to the strong linear probe results.

Negative result: ReST^EM-based revision model optimization degrades performance. The paper references an experiment in Appendix K (Figure 16) where an attempt to further optimize the revision model using ReST^EM "backfires: additional sequential revisions substantially hurt performance," with fully sequential performance dropping to approximately 33.5% compared to roughly 38.5% at the optimal ratio at 256 generations. The paper hypothesizes that "on-policy data collection in ReST^EM exacerbates spurious correlations in revision data, causing the model to fail to learn the revision task properly." This is a notable negative result that highlights the sensitivity of revision training to the data generation procedure.

Architectural choices: standard MHA and LayerNorm over GQA and RMSNorm. The paper simplifies the LLaMA 3.2 architecture by reverting from grouped-query attention to standard multi-head attention and from RMS normalization to layer normalization, but provides no ablation comparing these choices. The justification is that at the model's scale (~650M parameters), the efficiency gains from GQA and RMSNorm are less critical, but whether they would affect downstream task performance is not tested. This is a missing ablation that would help establish whether the architectural simplifications are neutral or actively beneficial.

Critical Assessment

Do the experiments support the claim that scaling SSL transfers to symbolic music when the data bottleneck is removed? The evidence is strongly suggestive but incomplete. The paper demonstrates that a 650M-parameter model trained on 60,000 hours of transcribed piano data achieves strong results across generation, embedding, and classification. However, the paper does not provide a controlled scaling comparison — there is no experiment showing the same architecture trained on 1,000 hours, 10,000 hours, and 60,000 hours with performance plotted as a function of data scale. The claim that scale is the primary variable is supported by the comparison against Music Transformer (trained on ~200 hours of MAESTRO data with a similar autoregressive approach), but this comparison confounds dataset composition (MAESTRO vs. Aria-MIDI), data quality (manual recording vs. AMT transcription), dataset size, model architecture, and training recipe. The paper cannot isolate how much of the performance difference is due to scale versus data diversity, data quality, tokenization improvements, or other factors. The closest thing to a scaling analysis is the ablation showing that contrastive training from scratch fails while pretrained initialization succeeds — but this demonstrates the importance of pretraining, not the importance of scale specifically.

Do the experiments support the claim that Aria outperforms leading symbolic generation techniques? The comparison against Anticipatory Music Transformer in Table 1 is the primary evidence. This appears well-supported by the reported pairwise human preference results, but several concerns arise. First, the paper does not report the raw win/tie/loss counts — only that the preference is statistically significant. Without these numbers, the magnitude of the preference cannot be assessed. Second, Anticipatory Music Transformer is a state-of-the-art symbolic model, but it was not explicitly designed or optimized for the specific solo-piano continuation task tested here — it is a general-purpose model for "prompt continuation and infilling tasks." Whether a version of AMT finetuned on the same Aria-MIDI data would perform differently is not tested. Third, the paper excludes MuPT, a recently published symbolic model, due to representation incompatibility (ABC notation). This is a legitimate scope limitation but means the claim about "leading symbolic generation techniques" is qualified — it applies to models compatible with millisecond-level expressive MIDI.

Do the experiments support the claim that Aria rivals proprietary audio models like Suno 3.5? The finding of no statistically significant difference between Aria and Suno 3.5 in Table 1 is provocative but requires careful interpretation. "Statistically insignificant" means the data do not provide sufficient evidence to reject the null hypothesis of no difference — it does not demonstrate equivalence. The sample size (46 participants, 40 continuations per model, evaluated in pairs) may be underpowered to detect moderate preference differences. More critically, Suno 3.5 is an audio generation model — its MIDI was synthesized to audio via Pianoteq for comparison. Pianoteq is mentioned in Section 4, but the specific piano model, rendering settings, and whether they match the acoustic characteristics participants expect are not specified. If participants could distinguish synthetic from recorded audio (even unconsciously), this could introduce a bias independent of musical coherence. The paper also does not report whether Suno 3.5 was prompted with the same 15-second audio clips or with some other interface (text description, MIDI input), nor whether the comparison was truly blind to the source model.

Do the experiments support the claim that contrastive embeddings achieve state-of-the-art linear probe accuracy on MIR classification benchmarks? The results in Table 2 support this for the specific piano-only benchmarks tested, but "state-of-the-art" requires qualification. The comparison models (CLaMP 3, M3, MERT) were all designed for broader domains — multi-instrument MIDI with text descriptions (CLaMP 3, M3) or audio (MERT). Their lower performance on piano-only benchmarks may partly reflect this domain mismatch rather than Aria's intrinsic superiority. A fairer comparison would evaluate all models on a shared multi-instrument benchmark, but Aria itself is piano-only and could not participate in such a comparison — this asymmetry makes "state-of-the-art" a domain-specific claim. Furthermore, the paper reports "maximum accuracy" across 10, 20, and 50 training epochs, with models selected based on test-set performance. While this is common in linear probe evaluations, it introduces a risk of overfitting the test set through hyperparameter selection, particularly with only 1,000 test examples per task.

Do the experiments support the claim that finetuning requires as few as a few hundred labeled examples? The supervised finetuning results in Table 2 show performance as a function of n labeled examples, and the paper states that models "perform surprisingly well on complex tasks, even when trained on limited labeled data." Without seeing the specific accuracy-at-n curves, it's unclear whether "a few hundred" means 100, 300, or 500, and what "surprisingly well" means relative to the full-data ceiling or random baselines. The paper does not report the performance of a randomly initialized model finetuned on the same limited data, which would isolate whether the pretrained representations specifically (versus the architecture's inductive biases) drive the label efficiency. The progressive dropout strategy (0.0 to 0.2) is a confound — it was designed for the low-data regime, and whether standard dropout performs equally well is not ablated.

Missing experiments that would strengthen the paper:

Dataset scaling ablation: same architecture, same tokenizer, same training recipe, varying dataset size (e.g., 1K, 10K, 100K, full 821K files) to directly measure the scaling relationship between data quantity and downstream performance.
Tokenization ablation: comparison of the proposed hybrid absolute-relative scheme against pure relative-timing (Music Transformer style) and pure absolute-timing (with a larger vocabulary) at the same model scale and data scale, to isolate the contribution of tokenization.
Pretraining objective ablation: next-token prediction vs. masked token prediction (BERT-style) as the pretraining objective, with both feeding into the same contrastive finetuning stage, to test whether autoregressive pretraining is uniquely suited or whether any SSL objective that learns musical structure would suffice.
Sample size justification for human evaluation: a power analysis for the two-sided binomial test given the observed effect sizes, to assess whether non-significance against Suno and human ground truth reflects genuine equivalence or insufficient statistical power.
Cross-dataset generalization: evaluation of the contrastive embeddings on classification benchmarks beyond Aria-MIDI-derived and piano-only datasets (e.g., multi-instrument symbolic MIR benchmarks), even if Aria itself is piano-only, to test whether the representations capture musical concepts that transfer across instrumentation.
Qualitative error analysis for generative outputs: although the paper's focus is on quantitative evaluation, examples of failure modes (rhythmic instability, harmonic inconsistencies, stylistic discontinuities) would help readers assess the scope of the model's capabilities and limitations beyond aggregate preference rates.

Where the claims hold conditionally:

The generative quality claim holds for the specific continuation task (45 seconds from 15-second prompts), specific instrumentation (solo piano), specific evaluation methodology (pairwise A/B by musically trained listeners on coherence criteria), and specific comparison set (Anticipatory Music Transformer, MusicGen, Suno 3.5, human ground truth). It does not necessarily generalize to longer generations, multi-instrument settings, different musical styles, or evaluation criteria beyond coherence (e.g., creativity, emotional expressiveness, structural novelty).
The contrastive embedding SOTA claim holds for piano-only classification benchmarks and for the specific comparison models tested. It does not establish superiority over all possible embedding approaches, and the domain restriction (piano-only) limits direct comparison with multi-instrument models.
The label efficiency claim holds for the specific classification tasks derived from Aria-MIDI metadata. Whether similar efficiency transfers to MIR tasks with fundamentally different input-output relationships (e.g., beat tracking, key detection, structural segmentation) is not tested.

6. Limitations and Trade-offs

Single Instrument, Single Corpus, Single Domain

The constraint. The paper's entire empirical apparatus—pretraining, finetuning, and all evaluations—operates within the narrow domain of solo-piano MIDI data derived from Aria-MIDI. The tokenizer's multi-instrument capability (13 instrument classes including percussion, as described in Section 3.1) is explicitly designed but never exercised: the pretraining corpus consists entirely of solo-piano transcriptions, and all classification benchmarks (Pianist8, VG-MIDI, and the four Aria-MIDI-derived tasks) are piano-only. The paper acknowledges this transparently: "our benchmarks focus exclusively on piano performances, whereas the comparison models support multi-instrument MIDI or audio files" (Section 5, Table 2 discussion). The ear-shaped tokenizer is thus an architectural promise, not an empirically validated capability.

The consequence. A practitioner deploying Aria for generative piano continuation gets strong within-domain performance—but has no evidence about what happens when the model encounters instruments, textures, or musical idioms it was never trained on. The contrastive embeddings capture piano-specific compositional and performance features, but a multi-instrument MIR task (orchestral composer identification, jazz ensemble genre classification) might fail for two distinct reasons: the model may not generalize across instrumentation, and the training distribution contains no multi-instrument data from which to learn cross-timbre musical abstractions. The supervised finetuning label-efficiency results (requiring "a few hundred labeled examples," Section 4.2) are similarly piano-only: a practitioner working with, say, string quartet or electronic music MIDI has no guidance on whether the pretrained backbone transfers or must be retrained from scratch.

What evidence exists. Table 2 provides the evidence for what the model can do—strong linear probe and finetuning results on piano-centric classification—but provides no negative evidence about what it cannot do. No experiment evaluates generalization to non-piano MIDI, either by testing on a multi-instrument benchmark or by finetuning on a small amount of multi-instrument data and measuring transfer. The paper's comparison against CLaMP 3 and M3 (trained on multi-instrument data with text descriptions) is asymmetric: their lower performance on piano benchmarks is interpreted as Aria's strength, but it could equally be interpreted as Aria's domain specialization masking an inability to handle the broader musical world that CLaMP and M3 were designed to cover. Figure 1's t-SNE visualization shows beautiful composer clustering—but all visualizations are of classical piano compositions, providing no signal about whether the embedding space would collapse or degrade for other repertoires.

Mitigation status. The paper positions this as scope rather than failure—the tokenizer "natively support[s] multi-track MIDI files" (Section 1, contribution 3), and the authors characterize the work as "a foundation for future research in symbolic music modeling" (Section 5, conclusion). The mitigation is architectural (the tokenizer design) but not empirical. No multi-track pretraining or evaluation is conducted, and the paper offers no prediction about whether the approach would transfer or what adaptations would be needed. A practitioner must treat the 60,000-hour piano-only pretraining as a proof of concept for the methodology, not as a general-purpose symbolic music foundation model.

The Generative Evaluation Compares Against an Informationally Impoverished Human Ground Truth

The constraint. The paper's most striking claim—that Aria's continuations are "statistically indistinguishable from human-composed ground-truth continuations" (Section 4.2, Table 1)—rests on an evaluation design where the human ground truth is a single arbitrary continuation of each prompt. When a pianist performs a piece, the specific 45 seconds that follow any given 15-second window represent one realization among many musically valid possibilities. The evaluation asks human judges to compare Aria's generated continuation against this one specific recording, framed as "the ground truth."

The consequence. This design conflates two distinct questions: (1) "Is Aria's continuation musically coherent?" and (2) "Is Aria's continuation as good as what the original performer actually played?" Finding no statistically significant preference between Aria's output and the original performance does not mean the outputs are musically equivalent—it means the evaluation cannot distinguish them under the specific pairwise A/B protocol with 46 participants. Several mechanisms could produce this result without implying parity. Human judges may lack the musical training to detect subtle differences in phrasing, voicing, or harmonic sophistication (one year of musical training is a low bar—approximately the level of a high school band student). The original performance may contain moments of looseness, imprecision, or interpretive choices that judges do not strongly prefer over a more "average" generated continuation. Most critically, the ground truth is only one specific continuation, not the set of all musically valid continuations—a generated output that is musically plausible but stylistically different from the original might be rated as equivalent because judges have no basis for preferring one valid option over another.

The conceptual issue is that musical coherence and creative quality are underdetermined by pairwise preference against a single ground truth. A model that produces competent but generic continuations might easily tie a human performance that takes interpretive risks (some successful, some not), yet the two outputs would differ radically in artistic interest. Table 1's non-significant p-value tells us about statistical power, not about musical substance.

What evidence exists. The paper reports the experimental parameters—46 participants, five prompts with eight continuations each, 45-second continuations from 15-second prompts, evaluation criteria including "melodic development, rhythmic structure, harmonic progression, and stylistic coherence" (Section 4.1). It does not report per-prompt breakdowns, inter-rater agreement (e.g., Fleiss' kappa), confidence intervals on preference rates, or any measure of output diversity within the eight generated continuations per prompt. The non-significance of the Aria-vs-human comparison is treated as evidence of model quality, but the possibility that the evaluation is simply underpowered or ill-suited to detect the relevant differences is not discussed. The paper also does not report whether judges could identify which continuation was human-generated above chance—a separate question from preference that would test whether the model's outputs are perceptually distinguishable from human performance.

Mitigation status. The paper acknowledges two limitations of the listening test—the exclusion of closed-access models like AudioLM and the incompatibility of bar-level representations in models like MuPT (Section 4.2)—but does not acknowledge the single-ground-truth issue or the interpretive ambiguity of null results in pairwise preference tests. The evaluation design follows precedent (AudioLM and Anticipatory Music Transformer used similar continuation paradigms), so the paper is not departing from community standards, but the strong claim of indistinguishability from human composition would require a more stringent test—multiple ground-truth continuations per prompt, expert musician judges, or tasks that test for specific compositional capabilities (structural coherence over longer timescales, harmonic originality, idiomatic writing) rather than aggregate preference.

Contrastive Embedding State-of-the-Art Claims Are Confounded by Domain Mismatch and Hyperparameter Selection

The constraint. The paper claims that Aria's contrastive embeddings "achieve state-of-the-art accuracy in linear probe experiments including composer, genre, and style detection" (Section 1, contribution 2) and that the method "substantially improves results on all benchmarks" (Section 4.2). The comparison models—CLaMP 3, M3, and MERT—were all designed, trained, and optimized for fundamentally different domains than the piano-only benchmarks on which they are evaluated.

The consequence. The comparison is structurally unfair to the baselines in ways that inflate the apparent superiority of Aria. CLaMP 3 and M3 are cross-modal models trained to align symbolic music with natural language descriptions across multiple instruments and genres; evaluating them on piano-only classification benchmarks without text queries forces them to use their symbolic encoders in a setting they were not optimized for. MERT is an audio model—its embeddings encode acoustic features (timbre, recording quality, room acoustics) that are entirely absent from MIDI data. Comparing MERT's audio-derived embeddings against Aria's MIDI-derived embeddings on a MIDI classification task is a category error: MERT is operating in the wrong modality. The paper acknowledges the piano-only scope limitation (as discussed in Limitation 1 above) but does not acknowledge the modality and domain mismatch asymmetry—the comparison models are disadvantaged by design, making "state-of-the-art" a claim about the specific benchmark configuration rather than about intrinsic embedding quality.

A separate confound is the linear probe evaluation protocol: classifiers are trained for "10, 20, and 50 epochs, and reporting the best result" (Section 4.1). This means the linear probe head's hyperparameters (learning rate, training duration) are optimized against test-set performance. With only 1,000 test examples per task, the risk of overfitting the test set through hyperparameter selection is non-trivial. The paper does not report validation-set-based model selection or confidence intervals that would quantify the uncertainty in the reported accuracy values. A linear probe that genuinely captures superior representations should be robust to reasonable variations in training duration—reporting only the best of three choices obscures whether the gains are consistent or the result of fortuitous optimization.

What evidence exists. Table 2 provides the raw accuracy and F1 values that support the SOTA claim, but the paper provides no evidence that the comparison models were evaluated fairly—no description of how CLaMP 3, M3, or MERT embeddings were extracted, whether their encoders were finetuned or frozen, or whether any attempt was made to adapt them to the piano-only domain. The paper also does not report whether the historical versions of these benchmarks have established baselines from other models that would contextualize the reported numbers. The ablation study (Table 2, embedding ablations) compares Aria_Emb against Aria† (contrastive training from scratch) and against Aria with varying epochs and temperature parameters, but does not include the comparison models in the ablation framework—so we cannot assess whether the gap between Aria and the baselines is larger than the gap between Aria configurations (which would indicate robustness to hyperparameter choices).

Mitigation status. The paper acknowledges the domain scope limitation: "our benchmarks focus exclusively on piano performances, whereas the comparison models support multi-instrument MIDI or audio files" (Section 4.2). This is transparency but not mitigation—the SOTA claim remains unqualified in the abstract and introduction. A fairer evaluation would either (a) restrict the claim to "SOTA on piano-only symbolic MIR benchmarks," (b) evaluate all models on a shared multi-instrument benchmark to control for domain, or (c) provide domain-adapted versions of the comparison models (e.g., finetuning CLaMP 3's symbolic encoder on piano data) to reduce the domain mismatch. None of these are attempted.

The Generative Finetuning Pipeline Relies on Unlabeled Data Quality Filters That Constrain Replicability

The constraint. The paper describes a multi-stage data curation pipeline with specific filtering decisions that materially affect the quality of both pretraining and generative finetuning: compositional deduplication (capped at 10 instances per opus/piece-number pair for prolific composers during pretraining, removed entirely during generative finetuning), heuristic filtering based on "note density, pitch and duration entropy, silence, and indicators of repetitive content" (Section 3.2), and an additional filter for generative finetuning "aimed at excluding transcriptions of synthesized MIDI files" (Section 3.2). The paper provides a repository reference for preprocessing details but does not specify the exact thresholds, features, or detection methods used in these filtering steps.

The consequence. A practitioner attempting to replicate the generative finetuning pipeline on a new corpus—or even on a different subset of Aria-MIDI—faces an underspecified recipe. The heuristic filters involve continuous thresholds (what note density is "too high"? what entropy is "too low"?) that strongly influence which files pass and which are discarded. The synthesized-MIDI detection filter is particularly opaque: distinguishing AMT transcriptions of human performances from AMT transcriptions of computer-played MIDI is a non-trivial classification task, and the paper does not describe the features or model used. If these filters are aggressive (40% of the raw Aria-MIDI data was filtered out for pretraining, per Section 3.2), small differences in threshold choices could produce meaningfully different training corpora and downstream model behavior.

This is not merely a reproducibility concern—it affects the interpretation of the paper's central scaling claim. The paper argues that "careful data curation and large-scale training can unlock new opportunities for downstream symbolic music applications" (Section 5). If the curation is the active ingredient but is not fully specified, the claim becomes "careful data curation (which we did but cannot fully describe) enables good results." Other practitioners implementing "similar" filtering may get different results because they operationalized "repetitive content" or "synthesized MIDI" differently, eroding the reliability of the findings.

What evidence exists. The paper quantifies the overall filtering effect—820,944 files / 60,473 hours retained from approximately 100,000 hours of raw Aria-MIDI—and describes the filtering criteria in conceptual terms (Section 3.2). The compositional deduplication procedure is specified precisely: "for composers with more than 250 instances of files containing opus and/or piece number tags, we retained at most 10 instances per opus/piece-number pair. For these same composers, we also discarded all other files that lack compositional identifiers." This is concrete and replicable. The heuristic filters (note density, entropy, silence, repetitive content) and the synthesized-MIDI filter are described only at the conceptual level, and the paper defers to an external repository for implementation details. No ablation demonstrates the sensitivity of downstream performance to filtering choices—for instance, what happens to generation quality or embedding accuracy if the quality thresholds are tightened or relaxed, or if the synthesized-MIDI filter is omitted.

Mitigation status. The paper partially mitigates this through its open-source release: the models, tokenizer, and preprocessing library are publicly available, which provides a reference implementation for the filtering pipeline. The repository reference (Section 3, footnote 4) indicates that preprocessing details are documented externally. However, external documentation is not a substitute for specifying key design choices in the paper itself—repositories change, links break, and readers evaluating the paper's claims at publication time may not have access to implementation details that materially affect the results. The paper also does not discuss the sensitivity of its results to filtering choices, leaving open the question of whether the reported performance is robust to reasonable variations in the curation pipeline or tightly coupled to specific threshold values.

Difficulty Estimation for Real-World Deployment Is Absent; All Strategies Are Applied Uniformly

The constraint. The paper's contrastive embedding and supervised finetuning pipelines treat all MIDI files and all classification examples uniformly—the same contrastive training procedure (25 epochs, τ = 0.1, one pair per file per epoch) and the same supervised finetuning recipe (progressive dropout 0.0 to 0.2, 10 epochs) are applied regardless of musical complexity, compositional period, or label difficulty. The generative evaluation uses five hand-selected prompts representing "different subgenres of solo piano music" (Section 4.1) but does not analyze whether generation quality varies systematically with prompt characteristics (e.g., tempo, harmonic complexity, density, period).

The consequence. A practitioner deploying Aria's embeddings for a specific classification task has no guidance on whether the default contrastive training hyperparameters are appropriate or whether certain classes (e.g., distinguishing Baroque from Classical period) are inherently more difficult and would benefit from more contrastive epochs, a different temperature, or per-class weighting. The label-efficiency experiments show that performance improves with more labeled data (Table 2, finetuning ablations), but the n labeled examples are sampled uniformly across classes—the paper does not report whether certain composers or genres require disproportionately more labels to reach a given accuracy threshold. This matters for deployment because real-world label budgets are often allocated per-class based on anticipated difficulty, and knowing which classes are "hard" under this representation would guide data collection.

For generative use, the lack of difficulty analysis means a user prompting Aria with a harmonically complex jazz standard versus a simple classical etude does not know whether the model's coherence will degrade on harder material. The human evaluation used only five prompts—a sample too small to characterize the difficulty-dependent behavior of the generative model. If the model performs well on three subgenres but poorly on two, the aggregate preference rates in Table 1 could still show strong overall performance while masking important failure modes.

What evidence exists. The paper provides no difficulty-stratified analysis. The five generative prompts are described as representing "different subgenres" but are not individually characterized by any difficulty metric (harmonic complexity, tempo variability, polyphony, period typicality), and per-prompt preference rates are not reported. The classification benchmarks report aggregate accuracy and F1 per task but not per-class breakdowns that would reveal whether certain composers, genres, or forms are systematically harder. The linear probe results report "best result" across 10/20/50 epochs without indicating whether the optimal epoch count varies by task in a way that correlates with task difficulty.

Mitigation status. Not addressed. The paper does not discuss the need for difficulty estimation, adaptive hyperparameter selection, or per-class performance analysis. This is a significant gap because real-world MIR deployments—especially those using limited labeled data—require knowing not just that the model works on average, but on which examples it is likely to fail and whether more data, different augmentation, or task-specific finetuning would help. The paper's uniform pipeline assumption is reasonable for establishing baseline capability but leaves the practitioner to discover failure modes through trial and error.

The Model's Scale Is Modest by Contemporary Standards, and the Scaling Claim Lacks a Controlled Comparison

The constraint. The paper frames its contribution as a demonstration that "scaling SSL is a promising approach to tackling difficult tasks across symbolic MIR" (Section 1) and that "careful data curation and large-scale training can unlock new opportunities" (Section 5). The model is approximately 650 million parameters, trained on 60,473 hours of MIDI data. By the standards of contemporary language and audio models—LLaMA 3.2's smallest variant is 1B parameters; MusicLM, AudioLM, and Suno 3.5 have undisclosed but likely much larger architectures—this is modest. More importantly, the paper does not include a controlled scaling experiment: there is no comparison of the same architecture and training recipe across multiple data scales (e.g., 1K, 10K, 100K files) or multiple model sizes (e.g., 150M, 650M, 1B parameters) with performance plotted as a function of compute or data.

The consequence. The paper's central narrative—that the symbolic music field's limitations were primarily due to data scale, and that removing the data bottleneck via AMT enables SSL to work—cannot be directly verified from the experiments. The 300× data increase over Music Transformer (and the accompanying performance improvement) is suggestive, but it confounds data quantity with data diversity (Aria-MIDI contains a wider range of composers, periods, and performance styles than MAESTRO), data quality (AMT transcriptions vs. manual Disklavier recordings), tokenization scheme, model architecture, and training recipe. A controlled scaling experiment would isolate the effect of data quantity: if performance on generation, embedding, or classification improves monotonically with data scale, the scaling claim is supported. If performance saturates early, other factors (architecture, data quality, tokenization) are more important. Without this experiment, the paper demonstrates that this specific combination of data, model, and recipe works well—but it does not demonstrate that scale per se is the causal factor, which is the core conceptual claim.

The modest model size also limits the paper's contribution to the scaling conversation. At 650M parameters, Aria is not testing the frontier of model capacity—it is operating in a regime where the Chinchilla scaling laws (Hoffmann et al., 2022, cited for the model-data ratio motivation in Section 3.2) suggest the model is compute-optimally matched to its data volume. This is a reasonable pragmatic choice, but it means the paper cannot speak to the question that its framing raises: does symbolic music modeling exhibit the same "scaling laws" behavior observed in language and vision, where larger models trained on proportionally larger datasets continue to improve predictably? A 650M model on 3–4 billion tokens is the starting point of the scaling curve, not evidence that the curve continues upward.

What evidence exists. The paper's ablation study showing that contrastive training from scratch fails while pretrained initialization succeeds (Table 2) demonstrates the value of pretraining, but does not distinguish between "pretraining is valuable" and "more pretraining is more valuable." The comparison against Music Transformer (Section 4.2, Table 1) shows a large model trained on much more data outperforming a smaller model trained on much less data—consistent with scaling but confounded as described above. The architectural reduction from LLaMA 3.2's 1B to 650M parameters, motivated by "established principles on model-data ratios for language models" (Section 3.2), implies the authors believe the scaling principles transfer, but this is an assumption built into the experimental design rather than a finding demonstrated by it. No experiment varies model size or data quantity independently while measuring downstream performance.

Mitigation status. The paper does not claim to have established scaling laws for symbolic music—the language is aspirational ("promising approach," "foundation for future research"). The modest model size is presented as a feature (computationally accessible, following Chinchilla guidance) rather than a limitation. However, the absence of a controlled scaling experiment means a reader evaluating whether to invest in larger-scale symbolic music pretraining cannot use this paper as direct evidence that the investment will pay off. The paper demonstrates that 60,000 hours works better than 200 hours, but whether 600,000 hours would work proportionally better—or whether a 7B-parameter model would substantially outperform the 650M version on the same data—remains unknown. The paper's contribution is an existence proof for strong piano-specific symbolic music modeling at modest scale, not a scaling analysis that predicts performance at larger scales.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper does not introduce a paradigm shift — it does not propose a fundamentally new architecture, loss function, or theoretical framework. What it does is arguably more consequential for a field that has been starved for data: it provides a methodological existence proof that removes the most persistent excuse for symbolic music modeling's limited progress. Before Aria, researchers could reasonably argue that symbolic music was inherently harder than audio or text for deep learning — that MIDI's sparsity, its complex polyphonic structure, or the semantic gap between note tokens and musical meaning made it resistant to the "pretrain then adapt" workflow that succeeded elsewhere. After Aria, that argument collapses. A ~650M-parameter autoregressive transformer — with simplified attention and normalization relative to its LLaMA 3.2 base, trained by next-token prediction on 60,000 hours of automatically transcribed piano data — adapts to generation, classification, and contrastive embedding without task-specific architectural modifications. This is exactly the behavior the field hoped for but had not previously demonstrated at scale.

The shift is therefore methodological rather than architectural: the paper repositions symbolic music modeling from a field constrained by data curation (designing models around the specific limitations of Lakh, IMSLP, or MAESTRO) to a field where the primary bottleneck is now data acquisition infrastructure — building better AMT pipelines, larger-scale MIDI corpora, and cross-instrument datasets. This reframing matters because it redirects research investment. Prior to this paper, a PhD student entering symbolic music ML faced a landscape where the most natural path to impact was designing clever architectures (MusicBERT's bar-masking, CLaMP's cross-modal alignment, MusicVAE's hierarchical latents) to squeeze signal out of small, curated datasets. After Aria, the natural path shifts toward scaling up data acquisition — transcribing more audio, expanding beyond piano, improving AMT accuracy on polyphonic and multi-instrument recordings — and then applying established SSL recipes. This is not a trivial change; it redraws the boundary between what counts as "systems engineering" and what counts as "research contribution" in the field.

The paper also resolves a tension that has been simmering in the symbolic music literature. On one side, Music Transformer (Huang et al., 2018) and its successors demonstrated that autoregressive transformers could learn long-term musical dependencies on expressive MIDI data, and follow-up work from the same group "provided strong evidence that the musical and creative capabilities of their model scale well with dataset size — reinforcing the value of curating large-scale piano transcription datasets as a future direction" (Section 2). On the other side, the dominant representation learning work — MusicBERT, CLaMP, MidiBERT-piano — pursued architectures specialized for music rather than scaling generic architectures on larger data. This paper demonstrates that the Music Transformer camp's scaling hypothesis was correct and that the field's subsequent pivot to specialized architectures was arguably premature. The finding that a standard LLaMA decoder with minimal modifications (and with the hidden dimension actually reduced following Chinchilla scaling principles) can match or exceed both symbolic and audio baselines across three distinct downstream tasks suggests that the field's investment in music-specific architectural complexity has not been the primary driver of progress. This is a productive simplification: it means future work can focus on data and scale rather than architecture design.

A more subtle repositioning concerns the relationship between symbolic and audio music modeling. Prior to this paper, the implicit narrative was that audio models (MusicLM, Suno, AudioLM) represented the high-water mark for music generation, with symbolic models playing a supporting role for tasks requiring editability or interpretability. The finding in Table 1 — that Aria's continuations are statistically indistinguishable from both Suno 3.5 and human ground truth — challenges this narrative. It suggests that for solo-piano music at least, symbolic models operating in the tokenized-MIDI space can be competitive with proprietary audio models of unknown but presumably much larger scale, while retaining the editability, interpretability, and computational efficiency advantages of the symbolic domain. If this finding generalizes to other instruments and ensembles, it would reposition symbolic modeling as not merely a complementary approach to audio but as a potentially superior one for certain applications — particularly those where fine-grained control, structural editing, or musicological analysis matter.

Finally, the paper's contrastive learning methodology contribution — demonstrating that SimCLR-style contrastive finetuning from pretrained next-token-prediction weights produces general-purpose MIDI embeddings without requiring labels, metadata, or paired text — opens a new avenue for music representation learning that does not depend on natural language descriptions. The CLaMP series has shown the power of text-music alignment, but text descriptions of music are inevitably lossy — they capture what can be described in words (genre, instrumentation, mood) but struggle with the fine-grained performance nuances (timing micro-variations, dynamic shaping, voicing balance) that distinguish an exceptional performance from a competent one. Aria's slice-based SimCLR adaptation captures these features by design, since two slices of the same performance share performance-level attributes but differ in note content. This is not a replacement for text-aligned embeddings but a complementary approach that captures complementary information — and the finding that it works only when initialized from pretrained weights (Table 2, ablation showing scratch contrastive training on 25 pairs per file underperforms single-epoch contrastive finetuning on one pair per file) provides a methodological principle that extends beyond this specific model.

Follow-Up Research This Work Enables

Scaling laws for symbolic music pretraining. The paper demonstrates that 60,000 hours of data with a ~650M-parameter model works well, but provides no evidence about the shape of the scaling curve. The central unanswered question is: for a fixed architecture and tokenization scheme, how does downstream performance (generation coherence, embedding accuracy, finetuning label efficiency) improve as a function of pretraining data volume? A controlled scaling experiment — training the same Aria architecture on 1K, 10K, 100K, and the full 821K-file corpus, then evaluating all checkpoints on a held-out generative quality metric (automated or human) and the Table 2 classification benchmarks — would establish whether symbolic music exhibits the power-law scaling behavior observed in language and vision (where performance improves predictably with data quantity), or whether it saturates at modest scales. The paper's current comparison against Music Transformer (trained on ~200 hours) conflates data quantity with data diversity, tokenization scheme, and model architecture, so the slope of the scaling curve is entirely uncharacterized. This experiment would also test the paper's implicit claim that its architectural simplification (removing GQA and RMSNorm) is neutral — if larger models trained on proportionally more data benefit disproportionately from these efficiency-focused architectural choices, the paper's minimalist design may not be optimal at scale.

Cross-instrument generalization and multi-track scaling. The paper's tokenizer natively supports 13 instrument classes (including percussion) and multi-track MIDI files, but this capability is entirely untested — all training and evaluation is piano-only. The natural extension is to pretrain the same architecture on a multi-instrument version of Aria-MIDI (or a combination of Aria-MIDI with multi-track datasets like Lakh or Slakh) and evaluate whether the SSL scaling paradigm transfers across instrumentation. The key experimental question is not simply "does it work?" — which is likely — but whether piano-only pretraining representations transfer to other instruments in low-data regimes. A concrete experimental design: pretrain Aria on the current 60,000-hour piano corpus, then finetune on progressively smaller subsets of multi-instrument MIDI data (e.g., string quartets, jazz ensembles, orchestral scores) and measure classification or generation performance relative to a model pretrained from scratch on only the multi-instrument data. If piano-pretrained weights accelerate multi-instrument learning (analogous to how ImageNet-pretrained CNNs accelerate medical imaging tasks despite domain mismatch), this would validate that the pretrained representations capture genuinely general musical structure — harmony, voice leading, rhythmic patterns — rather than piano-specific idiomatic patterns. If not, the representations are more domain-bound than the paper's "foundation model" framing implies, and separate pretraining per instrument family may be necessary.

Difficulty-conditioned generation evaluation. The human listening test (Table 1) uses five prompts across different subgenres but reports only aggregate preference rates, masking any systematic variation in Aria's performance as a function of musical complexity. A more diagnostic evaluation would stratify prompts by quantifiable difficulty metrics — polyphonic density (notes per second), harmonic complexity (chord vocabulary size or tonal ambiguity), tempo variability, formal complexity (e.g., sonata form vs. theme-and-variations vs. through-composed) — and measure whether preference rates degrade on harder material. The negative result (Aria performs well on simple material but poorly on complex material) would define the model's capability frontier and guide users toward appropriate applications. This could be extended to automated metrics: train a classifier to predict which prompts will produce high-coherence continuations (using features computable from the prompt alone — note density, pitch entropy, tempo stability) and validate against human judgments. If such a classifier works, it provides a practical prompt-engineering tool; if it fails, it suggests that coherence depends on features the model has learned but that are not trivially measurable from surface statistics, which is itself an informative finding about the nature of musical structure.

Verifier robustness and over-optimization in symbolic music search. The paper's generative finetuning includes a <D> token for explicit termination control, but the model currently generates continuations via standard autoregressive sampling — no beam search, no best-of-N, no verifier-guided decoding. A natural extension is to train a process reward model (PRM) or outcome reward model (ORM) on the same Aria-MIDI data — for instance, by scoring the musical coherence of completions using either human preference labels or automated metrics (harmonic consistency, rhythmic stability, adherence to stylistic conventions) — and then search against this verifier to find optimal continuations from a prompt. The critical experiment would test for over-optimization: does aggressive beam search against a learned verifier produce continuations that score highly under the verifier but are musically degenerate (repetitive, harmonically static, or stylistically inconsistent)? The paper's observations about transformers struggling with arithmetic accumulation (motivating the hybrid timing tokenizer) suggest that verifiers trained on MIDI data may be vulnerable to specific exploitation strategies — for instance, generating sequences with minimal temporal complexity (constant inter-onset intervals) that score well on rhythmic stability metrics but sound mechanical. A controlled experiment sweeping beam width, temperature, and verifier architecture would characterize the safe operating regime and identify whether verifier quality or search strategy is the primary bottleneck.

Cross-modal embedding alignment without paired text. Aria's contrastive embeddings capture performance and compositional features from MIDI alone, but they exist in a separate space from both audio embeddings (MERT) and text-aligned symbolic embeddings (CLaMP 3). An experiment that trains a lightweight projection from Aria's embedding space to CLaMP 3's text-aligned space — using a small set of paired (MIDI, text description) examples as a mapping dataset — would test whether Aria's piano-specific performance features can be retrofitted with semantic interpretability without sacrificing their fine-grained performance sensitivity. The hypothesis is that Aria's slice-based contrastive training has already structured the embedding space such that semantically similar pieces (same composer, same genre) are clustered (as Figure 1 suggests), and a modest number of text labels would be sufficient to anchor these clusters to natural language descriptions. If a few hundred text-MIDI pairs suffice to produce an embedding that matches CLaMP 3 on text-based retrieval while retaining Aria's advantage on performance-level discrimination tasks (distinguishing different pianists playing the same piece), this would demonstrate a practical path to unified embeddings that capture both what a piece is and how it is played.

Failure mode documentation for AMT-derived training data. The paper's data curation pipeline applies heuristic filters for note density, entropy, silence, and repetitive content, plus an explicit filter for synthesized MIDI transcriptions during generative finetuning. These filters removed approximately 40% of the raw Aria-MIDI data (from ~100,000 hours to ~60,473 hours), but the paper provides no systematic characterization of what was removed and how it would have affected the model. A follow-up study that trains small probe models on the filtered-out data and evaluates their outputs qualitatively would serve two purposes: (1) it would characterize the failure modes that the filters prevent (e.g., does unfiltered data produce models that generate Black MIDI-like note floods, or that reproduce synthesized-sounding timing regularities, or that memorize overrepresented pieces?), and (2) it would provide guidance for practitioners building their own AMT-derived datasets by identifying which filtering criteria are most important and what thresholds are appropriate. This is not glamorous research but it is essential for making the paper's methodology replicable beyond the specific Aria-MIDI corpus — if future datasets are curated from different AMT systems with different error profiles, the appropriate filtering thresholds may differ, and understanding why the current filters exist is more valuable than knowing their specific numeric values.

Practical Applications and Downstream Use Cases

Cost-efficient batch classification for large-scale MIDI archives. Organizations managing large collections of MIDI files — music libraries, educational institutions, digital score publishers, streaming platforms with MIDI-based recommendation systems — face the problem of metadata enrichment: millions of files with incomplete or absent composer, genre, period, and form labels. The paper's linear probe results (Table 2) demonstrate that Aria's frozen contrastive embeddings, with a simple linear classifier trained on only 1,000 labeled examples per task, achieve SOTA accuracy on composer (10 classes), genre (2 classes), period (4 classes), and form (6 classes) classification. For an archive with 100,000 untagged MIDI files, the workflow would be: (1) run each file through Aria's frozen encoder to produce a 512-dimensional embedding (computation: one forward pass per file, trivially batchable), (2) manually label 1,000 files (a few days of musicologist time), (3) train linear classifiers on these embeddings (seconds on CPU), and (4) apply to the remaining 99,000 files. The paper's label-efficiency experiments suggest that even 100–500 labeled examples per class may suffice for strong performance, making this practical for archives where expert labeling is the primary cost. This is a direct translation of the paper's findings into a deployable pipeline.

Piano performance search and similarity retrieval without metadata. The t-SNE visualization in Figure 1 — showing natural composer clustering of classical piano works without any metadata during contrastive training — demonstrates that Aria's embedding space organizes MIDI files by compositional and stylistic similarity even when no labels are available. This enables a retrieval application: a user provides a MIDI file (or a snippet of a performance) as a query, and the system returns the most similar files in the embedding space — similar composer, similar performance style, similar period. This is distinct from text-based search (which requires the user to articulate "I want something like Chopin's nocturnes") or audio-based search (which would retrieve based on acoustic similarity). The SimCLR slice-based training design means the embedding captures performance-level features (timing, dynamics, pedaling) that are invisible to score-level representations — a user could search for "recordings with a similar touch to Horowitz" by providing a Horowitz MIDI file as a query, even if the system has no Horowitz metadata. The 512-dimensional embeddings are compact enough for approximate nearest-neighbor indexing (e.g., FAISS), enabling sub-second retrieval over million-file collections on consumer hardware.

Interactive piano performance generation with editing capabilities. Aria's generative finetuning pipeline produces a model that, from a 15-second prompt, generates 45-second continuations that human listeners find statistically indistinguishable from both Suno 3.5 and human ground truth (Table 1). Because the output is MIDI, not audio, a composer or performer can immediately edit the result — adjust timing, change notes, modify dynamics, add or remove ornamentation — in any digital audio workstation (DAW) or notation software. This workflow is fundamentally different from audio generation models (Suno, MusicLM), where the output is a fixed waveform and any editing requires audio processing skills. A composer faced with writer's block could prompt Aria with the opening of a piece they're working on, generate several continuations, select the most promising one, and then refine it — treating the model as an intelligent variation generator rather than a final-output oracle. The paper's finding that performance is strong across five diverse subgenres (Section 4.1) suggests this works for classical, romantic, impressionist, and contemporary solo-piano idioms, though the lack of per-subgenre breakdown means users should verify quality on their specific style.

Data generation for self-improving symbolic music models. The paper notes as future work the possibility of "distilling the outputs of applying additional test-time compute back into the base LLM, enabling an iterative self-improvement loop" (Section 8 of the paper's framing, though in this work this specific direction is not explored). The compute-optimal allocation framework is not part of the Aria paper, so this application is extrapolative, but the paper's architecture makes it natural: use Aria to generate high-quality continuations from prompts drawn from a new composer or style, filter the generations for quality (using the contrastive embedding to verify stylistic consistency, for instance), and finetune the model on the filtered outputs. The paper's supervised finetuning results — where models trained on a few hundred labeled examples achieve strong classification performance — suggest that the pretrained representations are plastic enough to adapt to new distributions with modest data. A self-improvement loop could progressively expand the model's stylistic range without requiring new human-annotated data, bootstrapping from the 60,000-hour pretraining corpus plus the model's own generated high-quality outputs. This is speculative but directly motivated by the paper's demonstration that pretrained representations transfer efficiently.

When to Prefer This Method

The paper does not articulate an explicit tradeoff matrix against named alternatives in a "prefer A when X, prefer B when Y" format. It positions Aria as a foundation model — a single pretrained backbone that adapts to multiple downstream tasks — rather than as a specialized tool competing with specific baselines on a specific axis. The comparisons against Anticipatory Music Transformer, MusicGen, CLaMP 3, M3, and MERT (Tables 1 and 2) demonstrate Aria's performance within its domain (solo piano) but do not characterize the off-domain conditions where these alternatives would be preferable. The choice between Aria and alternatives is therefore governed by the scope limitations the paper acknowledges — piano-only data, symbolic MIDI only — rather than by a performance tradeoff the paper measures.

A practitioner should prefer Aria's approach (pretraining a similar autoregressive transformer on transcribed MIDI data, with optional contrastive finetuning for embeddings) when: the application domain is solo-piano or instrumentally homogeneous symbolic music (the tokenizer supports multi-instrument data but this is untested); fine-grained performance-level features matter (timing, dynamics, pedaling — the SimCLR slice-based training captures these, while CLaMP's text alignment may not); metadata is unavailable or incomplete (the contrastive embeddings require no labels, unlike supervised alternatives); and editability of generated output is required (MIDI is inherently editable; audio model outputs require audio processing). A practitioner should prefer alternatives — CLaMP 3 for text-based retrieval, MERT for audio-based similarity, Suno for quick audio generation without editing needs, Anticipatory Music Transformer for tasks requiring bar-level timing precision — when the application demands cross-modal alignment (text or audio), multi-instrument or multi-track generation, or audio output. The paper provides evidence that within its domain, Aria is competitive with or superior to these alternatives; it does not provide evidence about the reverse (whether these alternatives would outperform Aria within Aria's domain if adapted appropriately).